使用strtok读取多行字符串时无法正常工作的问题排查

我用Flex词法分析器得到以下文本并存在yytext中：

ISEP 1252 "ADDRESS"
"Name1" 1253 "INFORMATICA"
"Name2" 1254 "Boilerplate1"
"Name3" 1255 "Boilerplate2"
"Name4" 1256 "Boilerplate3"
"Name5" 1257 "Boilerplate4"
"Name6" 1258 "Boilerplate5"

之后用strtok分割每行和行内内容，代码如下：

// get first line
char* line = strtok(yytext, "\n");
// get school name
char* schoolName = strtok(line, " \t");
// get school num students
int schoolNumStudents = atoi(strtok(NULL, " \t"));
// get school address inside the quotes
char* schoolAddress = strtok(NULL, "\"");

strcpy(schools[schoolCount].name, schoolName);
schools[schoolCount].numStudents = schoolNumStudents;
strcpy(schools[schoolCount].address, schoolAddress);
//print school[schoolCount]
printf("Escola: %s\n", schools[schoolCount].name);
printf("Num alunos: %d\n", schools[schoolCount].numStudents);
printf("Morada: %s\n", schools[schoolCount].address);

// get teachers
line = strtok(NULL, "\n");
while (line != NULL) {
    char* teacherName = strtok(line, "\"");
    int teacherExt = atoi(strtok(NULL, " \t"));
    char* teacherDepartment = strtok(NULL, "\"");

    schools[schoolCount].numTeachers++;
    if(schools[schoolCount].teachers == NULL) {
        schools[schoolCount].teachers = (struct Teacher*) malloc(sizeof(struct Teacher));
    }
    else {
        schools[schoolCount].teachers = (struct Teacher*) realloc(schools[schoolCount].teachers, sizeof(struct Teacher) * (schools[schoolCount].numTeachers));
    }

    printf("Nome: %s\n", teacherName);
    printf("Ext: %d\n", teacherExt);
    printf("Departamento: %s\n", teacherDepartment);
    line = strtok(NULL, "\n");
}

schoolCount++;

问题是处理完第一行后，第二次调用strtok(NULL, "\n")返回NULL，拿不到第二行。试过把yytext复制到其他变量、清空strtok缓冲区都没用，代码嵌套在Flex规则里，没有其他相关C代码。

核心问题：strtok的全局状态被行内分割操作破坏 #

strtok依赖全局状态维护分割上下文，你第一次用strtok(yytext, "\n")拿到第一行后，紧接着调用strtok(line, " \t")分割行内内容——这会覆盖strtok的内部缓冲区指针，导致后续strtok(NULL, "\n")不再从yytext剩余部分找下一行，而是从第一行的剩余部分继续处理，自然返回NULL。

解决办法：避免嵌套使用strtok，以下两种方案任选其一 #

方案1：先拆分所有行，再逐行处理 #

先把所有行拆分到临时数组，再对每行单独处理，避免上下文干扰：

// 1. 拆分所有行到临时数组（必须复制yytext，Flex会复用其缓冲区）
char* lines[100]; // 按需调整大小
int lineCount = 0;
char* temp = strdup(yytext);
lines[lineCount++] = strtok(temp, "\n");
while ((lines[lineCount] = strtok(NULL, "\n")) != NULL) {
    lineCount++;
}

// 2. 处理学校信息（第一行）
char* schoolLine = lines[0];
char* schoolName = strtok(schoolLine, " \t");
int schoolNumStudents = atoi(strtok(NULL, " \t"));
char* schoolAddress = strtok(NULL, "\"");

strcpy(schools[schoolCount].name, schoolName);
schools[schoolCount].numStudents = schoolNumStudents;
strcpy(schools[schoolCount].address, schoolAddress);
printf("Escola: %s\n", schools[schoolCount].name);
printf("Num alunos: %d\n", schools[schoolCount].numStudents);
printf("Morada: %s\n", schools[schoolCount].address);

// 3. 处理教师信息（剩余行）
for (int i = 1; i < lineCount; i++) {
    char* teacherLine = strdup(lines[i]); // 复制行内容，避免原字符串被strtok修改
    char* teacherName = strtok(teacherLine, "\"");
    int teacherExt = atoi(strtok(NULL, " \t"));
    char* teacherDepartment = strtok(NULL, "\"");

    schools[schoolCount].numTeachers++;
    if (schools[schoolCount].teachers == NULL) {
        schools[schoolCount].teachers = (struct Teacher*)malloc(sizeof(struct Teacher));
    } else {
        schools[schoolCount].teachers = (struct Teacher*)realloc(schools[schoolCount].teachers, sizeof(struct Teacher) * schools[schoolCount].numTeachers);
    }

    printf("Nome: %s\n", teacherName);
    printf("Ext: %d\n", teacherExt);
    printf("Departamento: %s\n", teacherDepartment);
    free(teacherLine); // 释放临时内存
}

schoolCount++;
free(temp); // 释放复制的yytext内存

方案2：使用线程安全的strtok_r #

strtok_r允许手动维护每个分割操作的上下文（通过状态指针），不同分割操作互不干扰：

// 处理第一行，用单独的状态指针
char* lineState;
char* line = strtok_r(yytext, "\n", &lineState);

// 处理学校信息，用另一个状态指针
char* schoolState;
char* schoolName = strtok_r(line, " \t", &schoolState);
int schoolNumStudents = atoi(strtok_r(NULL, " \t", &schoolState));
char* schoolAddress = strtok_r(NULL, "\"", &schoolState);

strcpy(schools[schoolCount].name, schoolName);
schools[schoolCount].numStudents = schoolNumStudents;
strcpy(schools[schoolCount].address, schoolAddress);
printf("Escola: %s\n", schools[schoolCount].name);
printf("Num alunos: %d\n", schools[schoolCount].numStudents);
printf("Morada: %s\n", schools[schoolCount].address);

// 获取下一行，继续使用lineState
while ((line = strtok_r(NULL, "\n", &lineState)) != NULL) {
    // 处理教师行，用单独的teacherState
    char* teacherState;
    char* teacherName = strtok_r(line, "\"", &teacherState);
    int teacherExt = atoi(strtok_r(NULL, " \t", &teacherState));
    char* teacherDepartment = strtok_r(NULL, "\"", &teacherState);

    schools[schoolCount].numTeachers++;
    if (schools[schoolCount].teachers == NULL) {
        schools[schoolCount].teachers = (struct Teacher*)malloc(sizeof(struct Teacher));
    } else {
        schools[schoolCount].teachers = (struct Teacher*)realloc(schools[schoolCount].teachers, sizeof(struct Teacher) * schools[schoolCount].numTeachers);
    }

    printf("Nome: %s\n", teacherName);
    printf("Ext: %d\n", teacherExt);
    printf("Departamento: %s\n", teacherDepartment);
}

schoolCount++;

额外注意点 #

• Flex的yytext是内部缓冲区，会被后续操作覆盖，必须用strdup复制后再处理。
• 使用malloc/realloc时建议检查返回值，避免内存分配失败导致崩溃。

内容的提问来源于stack exchange，提问作者Hi_Im_Frost