NestJS gRPC批量请求卡顿10-15秒问题求助

问题现象 #

• 批量发送请求时，客户端与服务端进入idle状态，需等待10-15秒才会处理待执行调用
• 并行请求发起后1-2秒即出现卡顿；串行请求执行1300-3000次后出现卡顿
• 客户端、服务端均使用默认配置
• 服务器资源充足，剩余大量内存与CPU

已尝试的无效方案 #

• 用Promises替代Observables
• 调整keepAlive相关配置
• 调试确认客户端与服务端均处于idle状态

相关代码与配置 #

Proto文件 #

syntax = "proto3";

package person;

service Greeter {
  rpc GetPerson (PersonSearchByID) returns (PersonResponse);
}

message PersonSearchByID {
  int32 id = 1;
}

message Person {
    string name = 1;
    int32 id = 2;
    string email = 3;
}

message PersonResponse {
  Person person = 1;
}

服务端代码 #

@Controller()
export default class ServerController {
  @GrpcMethod('Greeter', 'GetPerson')
  getPerson(data: aj.PersonSearchByID): aj.PersonResponse {
    console.log(data, Object.getPrototypeOf(data), data.constructor);
    return {
      person: {
        id: data.id,
        name: 'John Doe',
        email: 'found@me.com',
      },
    };
  }
}

客户端代码 #

interface GreeterService {
  GetPerson(data: PersonSearchByID): Observable<PersonResponse>;
}

@Controller('client')
export default class ClientController implements OnModuleInit {
  private personService: GreeterService;

  constructor(@Inject('person_package') private readonly client: ClientGrpc) {}

  onModuleInit() {
    this.personService = this.client.getService<GreeterService>('Greeter');
  }

  @Get()
  public async getClient(
    @Query('id', ParseIntPipe) id: number,
  ): Promise<PersonResponse> {
    console.log('waiting', id);
    const person = await new Promise((resolve) => {
      this.personService.GetPerson({ id }).pipe(first()).subscribe(resolve);
    });
    console.log('done', id, person);

    return person;
  }
}

测试触发命令 #

# 并行请求：发起后1-2秒出现卡顿
seq 0 10000 | xargs -n 1 -P 10 -I {} curl "localhost:3456/client?id={}" -i -w '\n%{time_total}'

# 串行请求：执行约1300-3000次请求后出现卡顿
for i in {0..10000}; do curl "localhost:3456/client/test" -i -w '\n%{time_total}'; done

卡顿现象示例日志 #

done 7241 { person: { name: 'John Doe', id: 7241, email: 'found@me.com' } }
done 7242 { person: { name: 'John Doe', id: 7242, email: 'found@me.com' } }
done 7243 { person: { name: 'John Doe', id: 7243, email: 'found@me.com' } }
done 7244 { person: { name: 'John Doe', id: 7244, email: 'found@me.com' } }
waiting 7245
waiting 7246
done 7245 { person: { name: 'John Doe', id: 7245, email: 'found@me.com' } }
done 7246 { person: { name: 'John Doe', id: 7246, email: 'found@me.com' } }
waiting 7247
waiting 7248
waiting 7249
waiting 7250
waiting 7251
waiting 7252
waiting 7253
waiting 7254
waiting 7255
waiting 7256 # 此时卡顿10-15秒后才会继续输出`done`日志

内容的提问来源于stack exchange，提问作者Enthys