Azure DevOps Pipeline中PowerApps测试空引用异常排查求助

按照微软官方文档配置Azure DevOps Pipeline运行PowerApps Test Studio测试用例时，测试执行失败，报错如下：

第1轮测试执行出错：
System.NullReferenceException: 对象引用未设置到对象的实例。
报错位置：
Microsoft.PowerApps.TestAutomation.Api.TestAutomation.ReportResultsToDevOps(JObject jObject, Int32 testRunNumber) 在 D:\a\1\s\PowerAppsTestAutomation\Microsoft.PowerApps.TestAutomation.Api\Pages\TestAutomation.cs:第166行
以及Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation() 在 D:\a\1\s\PowerAppsTestAutomation\Microsoft.PowerApps.TestAutomation.Tests\TestAutomation\RunTestAutomation.cs:第178行

使用的代码仓库为：CesarCalvoCobo/PowerAppsTestAutomation（master分支）

相关代码片段：
TestAutomation.cs

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.

using Microsoft.PowerApps.TestAutomation.Browser;
using Newtonsoft.Json.Linq;
using OpenQA.Selenium;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;

namespace Microsoft.PowerApps.TestAutomation.Api
{
    /// <summary>
    ///   Test Automation methods.
    ///  </summary>
    public class TestAutomation
        : AppPage
    {
        /// <summary>
        /// Initializes a new instance of the <see cref="TestAutomation"/> class.
        /// </summary>
        /// <param name="browser">The browser.</param>
        public TestAutomation(InteractiveBrowser browser)
            : base(browser)
        {
        }

        public BrowserCommandResult<JObject> ExecuteTestAutomation(Uri uri, int testRunNumber, int maxWaitTimeInSeconds)
        {
            return this.Execute(GetOptions("Execute Test Automation"), driver =>
            {
                // Navigate to TestSuite or TestCase URL
                InitiateTest(driver, uri);

                // Check for existence of permissions dialog (1st test load for user)
                CheckForPermissionDialog(driver);

                // Try to report the sessionId. There is a bit of a race condition here,
                // so don't do this too close to fullscreen-app-host visibility or it 
                // will fail to find Core or some other namespace.
                TryReportSessionId(driver, testRunNumber);

                // Wait for test completion and collect results
                JObject testResults = WaitForTestResults(driver, maxWaitTimeInSeconds);

                return testResults;
            });
        }

        public BrowserCommandResult<List<Uri>> GetTestURLs(string filePath)
        {
            return this.Execute(GetOptions("Get List of Test URLs"), driver =>
            {
                //Replace encoded characters (%20) if present
                if (filePath.Contains("%20"))
                {
                    filePath = filePath.Replace("%20", " ");
                }
                // Initialize list of URLs
                List<Uri> testUrlList = new List<Uri>();

                // Read contents of json file
                JObject jObject = Newtonsoft.Json.JsonConvert.DeserializeObject<JObject>(File.ReadAllText(filePath));

                // Retrieve list of Test URLs
                testUrlList = jObject["TestURLs"]?.ToObject<List<Uri>>();

                return testUrlList;
            });
        }

        public class TestSuiteResult
        {
            public string TestSuiteId { get; set; }
            public string TestSuiteName { get; set; }
            public string TestSuiteDescription { get; set; }
            public long StartTime { get; set; }
            public long EndTime { get; set; }
            public int TestsPassed { get; set; }
            public int TestsFailed { get; set; }
        }

        public class TestCaseResult
        {
            public string TestSuiteId { get; set; }
            public string TestSuiteName { get; set; }
            public string TestSuiteDescription { get; set; }
            public string TestCaseId { get; set; }
            public string TestCaseName { get; set; }
            public string TestCaseDescription { get; set; }
            public string TestFailureMessage { get; set; }
            public long StartTime { get; set; }
            public long EndTime { get; set; }
            public bool Success { get; set; }
            public ArraySegment<string> Traces { get; set; }

        }

        internal void CheckForPermissionDialog(IWebDriver driver)
        {
            // Switch to default content
            driver.SwitchTo().DefaultContent();

            var dialogButtons = driver.WaitUntilAvailable(By.XPath(Elements.Xpath[Reference.TestAutomation.PermissionDialogButtons]), new TimeSpan(0, 0, 5));

            if (dialogButtons != null)
            {
                // Should be two buttons (Allow, Don't Allow)
                var buttons = dialogButtons.FindElements(By.TagName("button"));

                foreach (var b in buttons)
                {
                    if (b.Text.Equals("Allow"))
                    {
                        b.Hover(driver, true);
                        b.Click(true);
                        b.SendKeys(Keys.Enter);
                        driver.WaitForPageToLoad();
                    }
                }
            }
        }

        internal JObject WaitForTestResults(IWebDriver driver, int maxWaitTimeInSeconds)
        {
            JObject jsonResultString = new JObject();
            jsonResultString = driver.WaitForTestResults(maxWaitTimeInSeconds);

            return jsonResultString;
        }

        internal void InitiateTest(IWebDriver driver, Uri uri)
        {
            driver.Navigate().GoToUrl(uri);

            // Wait for page to load
            driver.WaitForPageToLoad();

            // Wait for fullscreen-app-host
            driver.WaitUntilVisible(By.Id("fullscreen-app-host"));
            if (driver.IsVisible(By.Id("fullscreen-app-host")))
            {
                Debug.WriteLine("fullscreen-app-host is visible.");
            }
            else
            {
                Debug.WriteLine("fullscreen-app-host is not visible.");
            }
        }

        public Tuple<int, int> ReportResultsToDevOps(JObject jObject, int testRunNumber)
        {
            var testExecutionMode = (int)jObject.GetValue("ExecutionMode");

            int passCount = 0;
            int failCount = 0;

            if (testExecutionMode == 0)
            {
                // TestCase
                var testCaseResults = jObject["TestCaseResult"]?.ToObject<TestCaseResult>();

                if (testCaseResults.Success)
                {
                    passCount = 1;
                    failCount = 0;
                }
                else
                {
                    passCount = 0;
                    failCount = 1;

                }
            }
            // 省略未粘贴的后续代码
            return Tuple.Create(passCount, failCount);
        }
    }
}

RunTestAutomation.cs

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license.

using Microsoft.PowerApps.TestAutomation.Browser;
using Microsoft.PowerApps.TestAutomation.Api;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json.Linq;
using System;
using System.Diagnostics;
using System.Linq;
using OpenQA.Selenium;
using System.Threading;

namespace Microsoft.PowerApps.TestAutomation.Tests
{
    [TestClass]
    public class TestAutomation
    {
        private static string _username = "";
        private static string _password = "";
        private static BrowserType _browserType;
        private static Uri _xrmUri;
        private static Uri _testAutomationUri;
        private static string _loginMethod;
        private static string _resultsDirectory = "";
        private static string _driversPath = "";
        private static string _usePrivateMode;
        private static string _testAutomationURLFilePath = "";
        private static int _globalTestCount = 0;
        private static int _globalPassCount = 0;
        private static int _globalFailCount = 0;
        private static int _testMaxWaitTimeInSeconds = 600;

        public TestContext TestContext { get; set; }

        private static TestContext _testContext;

        [ClassInitialize]
        public static void Initialize(TestContext TestContext)
        {
            _testContext = TestContext;
            
            _username = _testContext.Properties["OnlineUsername"].ToString();
            _password = _testContext.Properties["OnlinePassword"].ToString();
            _xrmUri = new Uri(_testContext.Properties["OnlineUrl"].ToString());
            _loginMethod = _testContext.Properties["LoginMethod"].ToString();
            _resultsDirectory = _testContext.Properties["ResultsDirectory"].ToString();
            _browserType = (BrowserType)Enum.Parse(typeof(BrowserType), _testContext.Properties["BrowserType"].ToString());
            _driversPath = _testContext.Properties["DriversPath"].ToString();
            _usePrivateMode = _testContext.Properties["UsePrivateMode"].ToString();
            _testAutomationURLFilePath = _testContext.Properties["TestAutomationURLFilePath"].ToString();
            _testMaxWaitTimeInSeconds = Convert.ToInt16(_testContext.Properties["TestMaxWaitTimeInSeconds"]);
                        
        }

        [TestCategory("PowerAppsTestAutomation")]
        [Priority(1)]
        [TestMethod]
        public void RunTestAutomation()
        {
            BrowserOptions options = RunTestSettings.Options;
            options.BrowserType = _browserType;
            options.DriversPath = _driversPath;

            using (var appBrowser = new PowerAppBrowser(options))
            {
                // Track current test iteration
                int testRunCounter = 0;
                // Track list of  Test Automation URLs
                var testUrlList = appBrowser.TestAutomation.GetTestURLs(_testAutomationURLFilePath);
                // Track total number of TestURLs
                int testUrlCount = testUrlList.Value.Count();

                foreach (Uri testUrl in testUrlList?.Value)
                {
                    // Test URL
                    _testAutomationUri = testUrl;
                    testRunCounter += 1;

                    try
                    {
                        // if TestCounter > 1, authentication not required
                        if (testRunCounter <= 1)
                        {
                            //Login To PowerApps
                            Debug.WriteLine($"Attempting to authenticate to Maker Portal: {_xrmUri}");

                            for (int retryCount = 0; retryCount < Reference.Login.SignInAttempts; retryCount++)
                            {
                                try
                                {
                                    // See Authentication Types: https://docs.microsoft.com/en-us/office365/servicedescriptions/office-365-platform-service-description/user-account-management#authentication
                                    // CloudIdentity uses standard Office 365 sign-in service
                                    if (_loginMethod == "CloudIdentity")
                                    {
                                        appBrowser.OnlineLogin.Login(_xrmUri, _username.ToSecureString(), _password.ToSecureString());
                                        break;
                                    }
                                    // FederatedIdentity uses AD FS 2.0 or other Security Token Services
                                    else if (_loginMethod == "FederatedIdentity")
                                    {
                                        // Do Federated Login
                                        appBrowser.OnlineLogin.Login(_xrmUri, _username.ToSecureString(), _password.ToSecureString(), FederatedLoginAction);
                                        break;
                                    }
                                    // FederatedIdentity scenario -- but DevOps agent is configured with SSO capability
                                    else if (_loginMethod == "PassThrough")
                                    {
                                        appBrowser.OnlineLogin.Login(_xrmUri);
                                        break;
                                    }
                                    // Fallback to CloudIdentity experience if _loginMethod is not provided
                                    else
                                    {
                                        appBrowser.OnlineLogin.Login(_xrmUri, _username.ToSecureString(), _password.ToSecureString());
                                        break;
                                    }

                                }
                                catch (Exception exc)
                                {
                                    Console.WriteLine($"Exception on Attempt #{retryCount + 1}: {exc}");

                                    if (retryCount + 1 == Reference.Login.SignInAttempts)
                                    {
                                        // Login exception occurred, take screenshot
                                        _resultsDirectory = TestContext.TestResultsDirectory;
                                        string location = $@"{_resultsDirectory}\RunTestAutomation-LoginErrorAttempt{retryCount + 1}.jpeg";

                                        appBrowser.TakeWindowScreenShot(location, OpenQA.Selenium.ScreenshotImageFormat.Jpeg);
                                        _testContext.AddResultFile(location);

                                        // Max Sign-In Attempts reached
                                        Console.WriteLine($"Login failed after {retryCount + 1} attempts.");
                                        throw new InvalidOperationException($"Login failed after {retryCount + 1} attempts. Exception Details: {exc}");
                                    }
                                    else
                                    {
                                        // Login exception occurred, take screenshot
                                        _resultsDirectory = TestContext.TestResultsDirectory;
                                        string location = $@"{_resultsDirectory}\RunTestAutomation-LoginErrorAttempt{retryCount + 1}.jpeg";

                                        appBrowser.TakeWindowScreenShot(location, OpenQA.Selenium.ScreenshotImageFormat.Jpeg);
                                        _testContext.AddResultFile(location);

                                        //Navigate away and retry
                                        appBrowser.Navigate("about:blank");

                                        Console.WriteLine($"Login failed after attempt #{retryCount + 1}.");
                                        continue;
                                    }
                                }
                            }
                        }

                        Console.WriteLine($"Power Apps  Test Automation Execution Starting Test #{testRunCounter} of {testUrlCount}");

                        // Initialize TestFrameworok results JSON object
                        JObject testAutomationResults = new JObject();

                        // Execute TestAutomation and return JSON result object
                        testAutomationResults = appBrowser.TestAutomation.ExecuteTestAutomation(_testAutomationUri, testRunCounter, _testMaxWaitTimeInSeconds);

                        #if DEBUG    
                        // Only output post execution screenshot in debug mode
                        _resultsDirectory = TestContext.TestResultsDirectory;
                        string location1 = $@"{_resultsDirectory}\TestRun{testRunCounter}-PostExecutionScreenshot.jpeg";
                        appBrowser.TakeWindowScreenShot(location1, OpenQA.Selenium.ScreenshotImageFormat.Jpeg);
                        _testContext.AddResultFile(location1);
                        #endif

                        // Report Results to DevOps Pipeline                    
                        var testResultCount = appBrowser.TestAutomation.ReportResultsToDevOps(testAutomationResults, testRunCounter);
                    }
                    // 省略未粘贴的后续代码
                }
            }
        }
    }
}

排查建议 #

• 定位空引用根源：报错在ReportResultsToDevOps第166行，也就是访问testCaseResults.Success时。这里testCaseResults是通过jObject["TestCaseResult"]?.ToObject<TestCaseResult>()获取的，说明返回的测试结果JSON中缺失TestCaseResult字段，或字段值为null，导致testCaseResults为空。
• 打印测试结果JSON：在调用ReportResultsToDevOps前添加日志，输出完整的测试结果JSON结构，确认是否包含ExecutionMode和对应类型的结果字段：

  Console.WriteLine("测试结果JSON：" + testAutomationResults.ToString());
  

• 验证测试执行完整性：检查WaitForTestResults是否真的等到测试执行完成。如果测试超时或中途崩溃，返回的JSON会不完整。可以尝试延长_testMaxWaitTimeInSeconds的值，或手动在浏览器中运行测试用例，确认其能正常执行完毕。
• 检查测试URL类型：如果测试URL指向的是测试套件而非单个用例，ExecutionMode应为1，但当前代码只处理了ExecutionMode == 0（单个用例）的情况，会导致套件结果无法解析。需补充套件模式的处理分支，或确认测试URL均为单个用例地址。
• 添加空值保护：修改ReportResultsToDevOps方法，在访问testCaseResults属性前先判空，避免空引用：

  var testCaseResults = jObject["TestCaseResult"]?.ToObject<TestCaseResult>();
  if (testCaseResults != null)
  {
      passCount = testCaseResults.Success ? 1 : 0;
      failCount = testCaseResults.Success ? 0 : 1;
  }
  else
  {
      failCount = 1;
      Console.WriteLine("警告：测试结果中缺失TestCaseResult字段");
  }
  

• 检查权限与环境：确认Azure DevOps代理账号有足够权限访问Power Apps测试用例，且代理环境的网络、浏览器驱动能正常加载测试页面。可在测试过程中增加更多截图或日志，确认页面加载状态。

内容的提问来源于stack exchange，提问作者dogowar