使用OpenAI运行ai-rag-chat-evaluator时遇embedding连接类型错误求助

问题描述 #

我正在开发ai-rag-chat-evaluator项目，执行命令python -m evaltools evaluate --config=example_config.json时触发错误："Not Support connection type None for embedding api. Connection type should be in [AzureOpenAI, OpenAI]"。我使用的是OpenAI实例而非Azure OpenAI，已配置好.env和example_config.json文件，部署Azure OpenAI Search时使用的是OpenAI密钥，怀疑是embedding配置缺失导致问题。

完整堆栈跟踪：

(evalrag) PS C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator> python -m evaltools evaluate --config=example_config.json
[19:52:20] INFO     Running evaluation from config C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator\example_config.json                      evaluate.py:237
           INFO     Replaced results_dir in config with timestamp                                                                             evaluate.py:218
           INFO     Using OpenAI Service with API Key from OPENAICOM_KEY                                                                  service_setup.py:42
           INFO     Running evaluation using data from C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator\example_input\qa.jsonl               evaluate.py:101
           INFO     Sending a test question to the target to ensure it is running...                                                          evaluate.py:107
[19:52:27] INFO     Successfully received response from target for question: "What information is in your knowledge base?"                    evaluate.py:118
                    "answer": "The knowledge base includes information about the ..."                                                                        
                    "context": "Northwind_Standard_Benefits_Details.pdf#page=99:  ..."                                                                        
           INFO     Sending a test chat completion to the GPT deployment to ensure it is running...                                           evaluate.py:128
[19:52:28] INFO     Successfully received response from GPT: "Hello! How can I assist you today?"                                             evaluate.py:134
           INFO     Starting evaluation...                                                                                                    evaluate.py:136
Processing... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Traceback (most recent call last):
  File "C:\Users\Bhabani Mohapatra\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Bhabani Mohapatra\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator\src\evaltools\__main__.py", line 6, in <module>
    app()
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\typer\main.py", line 340, in __call__
    raise e
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\typer\main.py", line 323, in __call__
    return get_command(self)(*args, **kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\click\core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\typer\core.py", line 743, in main
    return _main(
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\typer\core.py", line 198, in _main
    rv = self.invoke(ctx)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\click\core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\click\core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\click\core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\typer\main.py", line 698, in wrapper
    return callback(**use_params)
  File "C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator\src\evaltools\cli.py", line 86, in evaluate
    run_evaluate_from_config(Path.cwd(), config, numquestions, targeturl, resultsdir)
  File "C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator\src\evaltools\eval\evaluate.py", line 245, in run_evaluate_from_config
    evaluation_run_complete = run_evaluation(
  File "C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator\src\evaltools\eval\evaluate.py", line 172, in run_evaluation
    questions_with_ratings.append(evaluate_row(row))
  File "C:\az_ai_rag_chat_evaluator\ai-rag-chat-evaluator\src\evaltools\eval\evaluate.py", line 159, in evaluate_row
    result = metric.evaluator_fn(openai_config=openai_config)(
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\azure\ai\evaluation\_evaluators\_groundedness\_groundedness.py", line 144, in __call__
    return super().__call__(*args, **kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\azure\ai\evaluation\_evaluators\_common\_base_eval.py", line 107, in __call__
    return async_run_allowing_running_loop(self._async_evaluator, **kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\promptflow\_utils\async_utils.py", line 96, in async_run_allowing_running_loop
    return asyncio.run(_invoke_async_with_sigint_handler(async_func, *args, **kwargs))
  File "C:\Users\Bhabani Mohapatra\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\Bhabani Mohapatra\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 646, in run_until_complete
    return future.result()
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\promptflow\_utils\async_utils.py", line 70, in _invoke_async_with_sigint_handler
    return await task
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\azure\ai\evaluation\_evaluators\_common\_base_eval.py", line 414, in __call__
    return await self._real_call(**kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\azure\ai\evaluation\_evaluators\_common\_base_eval.py", line 376, in _real_call
    per_turn_results.append(await self._do_eval(eval_input))
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\azure\ai\evaluation\_evaluators\_common\_base_prompty_eval.py", line 83, in _do_eval
    llm_output = await self._flow(timeout=self._LLM_CALL_TIMEOUT, **eval_input)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\promptflow\tracing\_trace.py", line 476, in wrapped
    return await func(*args, **kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\promptflow\core\_prompty_utils.py", line 1194, in wrapper
    raise e
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\promptflow\core\_prompty_utils.py", line 1191, in wrapper
    return await func(*args, **kwargs)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\promptflow\core\_flow.py", line 549, in __call__
    connection = convert_model_configuration_to_connection(self._model.configuration)
  File "C:\az_ai_rag_chat_evaluator\evalrag\lib\site-packages\promptflow\core\_prompty_utils.py", line 131, in convert_model_configuration_to_connection     
    raise UnknownConnectionType(message=error_message)
promptflow.core._errors.UnknownConnectionType: Not Support connection type None for embedding api. Connection type should be in [AzureOpenAI, OpenAI].

解决步骤 #

• 补充example_config.json中的embedding配置
  打开配置文件，添加或完善embedding节点，明确指定连接类型为OpenAI，并关联对应的模型和API密钥：

  "embedding": {
    "connection_type": "OpenAI",
    "model": "text-embedding-ada-002",
    "api_key": "${OPENAICOM_KEY}"
  }
  

• 验证.env文件的密钥配置
  确认.env文件中OPENAICOM_KEY的值是有效的OpenAI API密钥，且配置文件中的变量引用格式正确。

• 确认评估逻辑的配置传递
  检查项目中evaluate.py等核心文件，确保embedding配置被正确传递给评估器（比如groundedness评估模块），避免出现配置缺失的情况。

• 重新执行评估命令
  修改配置后，重新运行命令：

  python -m evaltools evaluate --config=example_config.json
  

内容的提问来源于stack exchange，提问作者WaterRocket8236