如何在GitHub Actions工作流中用GPT分析Karate测试失败原因？

我开发了一个GitHub Actions的POC工作流，能自动把Karate自动化测试的失败信息提交给GPT分析，最终将汇总后的分析结果展示在GitHub作业的摘要页面里。目前这个工作流在定位测试失败原因方面表现稳定，现在分享给社区，欢迎大家提出反馈或改进建议。

完整工作流代码如下：

- name: Analyze failed Karate tests with GPT and consolidate results
    id: gpt
    if: failure()
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    run: |
      # Read KarateRunner.txt for context
      RUNNER_CONTENT=$(cat target/surefire-reports/KarateRunner.txt)
      
      # Extract unique failed features from KarateRunner.txt (lines starting with "classpath:")
      FAILED_FEATURES=$(grep "^classpath:" target/surefire-reports/KarateRunner.txt | awk -F':' '{print $2}' | sort | uniq)
      echo "Failed features detected: $FAILED_FEATURES"
      
      AGGREGATED_RESPONSES=""
      
      # Loop over each failed feature
      for feature in $FAILED_FEATURES; do
        echo "Processing feature: $feature"
        # Compute report file name:
        #   - Replace "/" with "."
        #   - Remove trailing ".feature" if present
        #   - Append ".karate-json.txt"
        FILE_NAME=$(echo "$feature" | tr '/' '.')
        FILE_NAME=${FILE_NAME%.feature}
        REPORT_FILE="target/karate-reports/${FILE_NAME}.karate-json.txt"
        
        if [ -f "$REPORT_FILE" ]; then
          echo "Adding report: $REPORT_FILE"
          REPORT_CONTENT=$(cat "$REPORT_FILE")
        else
          echo "Report file not found: $REPORT_FILE"
          REPORT_CONTENT="(No report file found)"
        fi
        
        # Compute the corresponding feature file path.
        FOLDER=$(echo "$feature" | cut -d'/' -f1)
        FEATURE_NAME=$(echo "$feature" | cut -d'/' -f2)
        FEATURE_FILE="target/test-classes/${FOLDER}/${FEATURE_NAME}"
        if [ -f "$FEATURE_FILE" ]; then
          echo "Adding feature file: $FEATURE_FILE"
          FEATURE_CONTENT=$(cat "$FEATURE_FILE")
        else
          echo "Feature file not found: $FEATURE_FILE"
          FEATURE_CONTENT="(No feature file found)"
        fi
        
        # Build a local context combining:
        #   - The full KarateRunner.txt content (global context)
        #   - The individual report content
        #   - The feature file content
        LOCAL_CONTEXT="Karate Runner Context:
        ${RUNNER_CONTENT}

        ==== Report: ${REPORT_FILE} ====
        ${REPORT_CONTENT}

        ==== Feature File: ${FEATURE_FILE} ====
        ${FEATURE_CONTENT}"

        # Write the local context to a temporary file to avoid huge argument lists
        echo -e "$LOCAL_CONTEXT" > local_context.txt
        
        # Build a request JSON using jqs --rawfile to load local_context.txt as $ctx
        REQUEST_JSON=$(jq -n --rawfile ctx local_context.txt '{
          model: "chatgpt-4o-latest",
          messages: [
            {
              "role": "system",
              "content": "You are an expert in Karate automation and log file analysis. Analyze the provided context and explain concisely why the test(s) failed (no more than a few sentences). Respond in plain text without any special formatting."
            },
            {
              "role": "user",
              "content": "Analyze the following context:\n\($ctx)"
            }
          ],
          temperature: 0.3
        }')
        rm local_context.txt
        
        # Write the request JSON to a temporary file so curl can read it
        echo "$REQUEST_JSON" > request.json
        
        # Call the OpenAI API for this feature
        RESPONSE=$(curl -s -w "%{http_code}" -X POST "https://api.openai.com/v1/chat/completions" \
          -H "Authorization: Bearer $OPENAI_API_KEY" \
          -H "Content-Type: application/json" \
          --data-binary "@request.json")
        rm request.json
        
        HTTP_STATUS=$(echo "$RESPONSE" | tail -n1)
        RESPONSE_BODY=$(echo "$RESPONSE" | head -n -1)
        
        if [ "$HTTP_STATUS" -ne 200 ]; then
          echo "API request for $feature failed with status $HTTP_STATUS"
          echo "Response: $RESPONSE_BODY"
          exit 1
        fi
        
        FEATURE_RESPONSE=$(echo "$RESPONSE_BODY" | jq -r '.choices[0].message.content')
        
        # Append this feature's analysis to the aggregated responses with a header
        AGGREGATED_RESPONSES="${AGGREGATED_RESPONSES}\n==== Analysis for $feature ====\n${FEATURE_RESPONSE}\n"
      done
      
      # Write the aggregated responses to a temporary file
      echo -e "$AGGREGATED_RESPONSES" > aggregated.txt
      
      # Final pass: Ask GPT to clean up and consolidate the individual analyses
      FINAL_REQUEST=$(jq -n --rawfile agg aggregated.txt '{
        model: "chatgpt-4o-latest",
        messages: [
          {
            "role": "system",
            "content": "You are an expert at consolidating analysis. Clean up and consolidate the following individual failure analyses into one report. Respond in plain text without any special formatting."
          },
          {
            "role": "user",
            "content": "Combine and clean up these analyses:\n\($agg)"
          }
        ],
        temperature: 0.3
      }')
      rm aggregated.txt
      
      # Write final request JSON to a file
      echo "$FINAL_REQUEST" > final_request.json
      
      FINAL_RESPONSE=$(curl -s -w "%{http_code}" -X POST "https://api.openai.com/v1/chat/completions" \
        -H "Authorization: Bearer $OPENAI_API_KEY" \
        -H "Content-Type: application/json" \
        --data-binary "@final_request.json")
      rm final_request.json
      
      FINAL_HTTP_STATUS=$(echo "$FINAL_RESPONSE" | tail -n1)
      FINAL_RESPONSE_BODY=$(echo "$FINAL_RESPONSE" | head -n -1)
      
      if [ "$FINAL_HTTP_STATUS" -ne 200 ]; then
        echo "Final OpenAI API request failed with status $FINAL_HTTP_STATUS"
        echo "Response: $FINAL_RESPONSE_BODY"
        exit 1
      fi
      
      FINAL_CONTENT=$(echo "$FINAL_RESPONSE_BODY" | jq -r '.choices[0].message.content')
      FINAL_SUMMARY="$FINAL_CONTENT"
      echo "$FINAL_SUMMARY" > gpt_summary.txt
      
      # Sanitize the final output for GitHub Actions
      SANITIZED_FINAL=$(echo "$FINAL_SUMMARY" | tr '\n' ' ' | sed 's/"/\\"/g')
      echo "gpt_result=$SANITIZED_FINAL" >> $GITHUB_OUTPUT

  - name: Publish GPT summary to GitHub
    if: always()
    run: |
      if [ -s gpt_summary.txt ]; then
        cat gpt_summary.txt >> $GITHUB_STEP_SUMMARY
      fi

核心工作流程 #

• 触发时机：仅当测试阶段失败时执行该工作流（通过if: failure()控制）
• 失败Feature提取：从target/surefire-reports/KarateRunner.txt中过滤并提取所有失败的Karate Feature文件路径
• 上下文收集：为每个失败的Feature收集三类信息：全局运行日志、该Feature的测试报告、Feature文件本身的内容
• 单Feature分析：将每个Feature的完整上下文发送给GPT-4o，获取针对性的失败原因分析
• 结果汇总：把所有单Feature的分析结果再次提交给GPT，生成一份简洁统一的汇总报告
• 结果展示：将最终的汇总报告写入GitHub作业的STEP_SUMMARY，无需额外点击即可在作业页面查看分析结果

内容的提问来源于stack exchange，提问作者j8d