如何在VSTS中创建KPI及获取手动测试首次75%通过率指标数据

Got it, let's break this down—since the default dashboard widgets and test plan charts only surface current test status, you'll need to use more targeted analytics tools built into Azure DevOps (formerly VSTS) to track that 75% first-pass pass rate for manual tests. Here are three actionable approaches:

1. Use Azure DevOps Analytics Views (Built-In, No Extra Tools) #

This is the easiest way to get custom test metrics without leaving VSTS:

• Navigate to your project, then go to Analytics > Views (you might need to enable the Analytics extension if it's not visible).
• Create a new view, select the Test Runs entity as your base.
• Add filters to narrow down to manual tests:
  • Filter Is Automated equals False
  • Optional: Filter by specific test plans/suites if you only want a subset
• Add a calculated field to measure first-pass rate:
  • First, define first-pass as the first execution of each manual test case (exclude retries of failed tests).
  • Create a calculated column using this logic: Count of Test Runs where Outcome = Pass and Is First Run = True divided by Count of All First Run Test Runs
  • Tip: Use the First Test Run system field to identify initial executions (this flags the first time a test case is run in a given context)
• Once your view is set up, you can visualize the rate as a card or line chart, pin it to your dashboard, or export the data as CSV for further tracking.

2. Integrate with Power BI (For Advanced Trend Tracking) #

If you need to track the KPI over time or combine it with other project metrics, Power BI is ideal:

• Use the Azure DevOps Power BI Connector to import test run data into Power BI Desktop.
• In Power BI, filter the dataset to only include manual tests (IsAutomated = False).
• Create a measure to calculate first-pass rate:
  • First, group test runs by Test Case ID and keep only the earliest run for each case.
  • Then calculate: DIVIDE(COUNTROWS(FILTER(FirstRuns, FirstRuns[Outcome] = "Pass")), COUNTROWS(FirstRuns), 0)
• You can build interactive reports showing weekly/monthly trends of the pass rate, and even set up alerts when it drops below your 75% target.

3. Use the Azure DevOps REST API (Custom Automation) #

If you need to build a custom dashboard or automate KPI reporting, use the Test Runs API:

• Call the Test Runs endpoint to fetch all manual test runs:

  GET https://dev.azure.com/{organization}/{project}/_apis/test/runs?api-version=7.1-preview.7&isAutomated=false
  

• For each test case ID in the response, filter to keep only the earliest run (using the startedDate field).
• Calculate the pass rate by dividing the number of first-run passes by the total number of first runs.
• You can wrap this in a script (Python, PowerShell) to run on a schedule and push the KPI to your internal reporting tools.

Key Notes to Avoid Errors #

• Make sure your test runs are properly tagged as manual (the Is Automated field is set correctly when creating test plans).
• Define "first run" consistently—some teams consider the first run in a sprint, while others use the first run ever for a test case. Adjust your filters/calculations accordingly.

内容的提问来源于stack exchange，提问作者Antony Delaney