IIS中C#调用Python脚本访问Scopus失败问题排查

问题核心 #

部署在IIS上的C# MVC应用调用Python脚本访问Scopus时，脚本无法定位页面元素并触发Detected!异常，但脚本在命令行运行完全正常。

错误信息 #

Error in python script: Traceback (most recent call last):
  File "C:\inetpub\site\scopus.py", line 15, in <module>
    verify_success(sb)
  File "C:\inetpub\site\scopus.py", line 9, in verify_success
    sb.assert_element('//span[contains(text(), "Author Search")]', timeout=30)
...
seleniumbase.common.exceptions.NoSuchElementException: Message: 
 Element {//span[contains(text(), "Author Search")]} was not present after 30 seconds!
...
Exception: Detected!

代码片段 #

Python脚本 #

from seleniumbase import SB
from bs4 import BeautifulSoup
import sys
import json

def verify_success(sb):
    sb.assert_element('//span[contains(text(), "Author Search")]', timeout=10)
    sb.sleep(3)

userId = sys.argv[1]

with SB(uc=True) as sb:
    sb.uc_open_with_reconnect(f"https://www.scopus.com/authid/detail.uri?authorId={userId}", 3)
    try:
        verify_success(sb)
    except Exception:
        if sb.is_element_visible('input[value*="Verify"]'):
            sb.uc_click('input[value*="Verify"]')
        else:
            sb.uc_gui_click_captcha()
        try:
            verify_success(sb)
        except Exception:
            raise Exception("Detected!")
    finally:
        page_source = sb.get_page_source()
        document = BeautifulSoup(page_source, 'html.parser')
        citations_node = document.select_one("div[data-testid='metrics-section-citations-count'] span[data-testid='unclickable-count']")
        documents_node = document.select_one("div[data-testid='metrics-section-document-count'] span[data-testid='unclickable-count']")
        hindex_node = document.select_one("div[data-testid='metrics-section-h-index'] span[data-testid='unclickable-count']")
        name_node = document.select_one("h1[data-testid='author-profile-name'] strong")
        institute_node = document.select_one("span[data-testid='authorInstitute']")
        scopus_information = {
            "CitationsNumber": int(citations_node.text.replace(",", "") if citations_node else "0"),
            "Documents": int(documents_node.text if documents_node else "0"),
            "HIndex": int(hindex_node.text if hindex_node else "0"),
            "Name": name_node.text.strip() if name_node else "",
            "Institute": institute_node.text.strip() if institute_node else ""
        }
        print(json.dumps(scopus_information, indent=4))

C#代码 #

public class ScopusInformation
{
    public string? Name { get; set; }
    public int? CitationsNumber { get; set; }
    public int? Documents { get; set; }
    public int? HIndex { get; set; }
    public string? Institute { get; set; }
    public string? ScopusId { get; set; }
}
public interface IScopusService
{
    Task<ScopusInformation?> GetArticlesForUser(string userId);
}
public class ScopusServiceUsingPython(ILogger<ScopusService> logger) : IScopusService
{
  public async Task<ScopusInformation?> GetArticlesForUser(string userId)
    {
        string cmd = "py";
        var result2 = await DoCmdAsync(cmd, $"scopus.py {userId}");
        ScopusInformation? r2 = JsonSerializer.Deserialize<ScopusInformation>(result2);
        return r2;
    }
 private async Task<string> DoCmdAsync(string cmd, string args)
    {
        logger.LogWarning("DoCmd in python script {cmd} {args}", cmd, args);
        try
        {
            await EnsurePythonModulesAsync(cmd, new[] { "seleniumbase", "beautifulsoup4", "pyautogui" });

            var start = new ProcessStartInfo
            {
                FileName = cmd,
                Arguments = args,
                UseShellExecute = false,
                RedirectStandardOutput = true,
                RedirectStandardError = true,
                CreateNoWindow = true
            };

            using var process = new Process { StartInfo = start };
            process.Start();

            var outputTask = process.StandardOutput.ReadToEndAsync();
            var errorTask = process.StandardError.ReadToEndAsync();

            await Task.WhenAll(outputTask, errorTask);

            string result = await outputTask;
            string error = await errorTask;

            if (!process.WaitForExit(30000))
            {
                process.Kill();
                throw new TimeoutException("Python script execution timed out after 30 seconds.");
            }

            if (!string.IsNullOrEmpty(error))
            {
                logger.LogWarning("Error in python script: {error}", error);
            }

            logger.LogWarning("Result in python script: {result}", result);
            return result;
        }
        catch (Exception ex)
        {
            logger.LogWarning("Exception in python script: {ex}", ex.ToString());
            return "";
        }
    }

    private async Task EnsurePythonModulesAsync(string pythonPath, string[] modules)
    {
        foreach (var module in modules)
        {
            logger.LogWarning("Checking Python module: {module}", module);
            var checkStart = new ProcessStartInfo
            {
                FileName = pythonPath,
                Arguments = $"-c \"import {module}\"",
                UseShellExecute = false,
                RedirectStandardOutput = true,
                RedirectStandardError = true,
                CreateNoWindow = true
            };

            using var checkProcess = new Process { StartInfo = checkStart };
            checkProcess.Start();
            if (!checkProcess.WaitForExit(10000))
            {
                checkProcess.Kill();
                throw new TimeoutException($"Checking for Python module {module} timed out.");
            }

            if (checkProcess.ExitCode != 0)
            {
                logger.LogWarning("Installing missing Python module: {module}", module);
                var installStart = new ProcessStartInfo
                {
                    FileName = pythonPath,
                    Arguments = $"-m pip install {module}",
                    UseShellExecute = false,
                    RedirectStandardOutput = true,
                    RedirectStandardError = true,
                    CreateNoWindow = true
                };

                using var installProcess = new Process { StartInfo = installStart };
                installProcess.Start();

                var outputTask = installProcess.StandardOutput.ReadToEndAsync();
                var errorTask = installProcess.StandardError.ReadToEndAsync();

                if (await Task.WhenAny(Task.WhenAll(outputTask, errorTask), Task.Delay(300000)) == Task.Delay(300000))
                {
                    installProcess.Kill();
                    throw new TimeoutException($"Installation of Python module {module} timed out after 5 minutes.");
                }

                string output = await outputTask;
                string error = await errorTask;

                if (installProcess.ExitCode != 0)
                {
                    throw new Exception($"Failed to install Python module {module}: {error}");
                }

                logger.LogWarning("Successfully installed Python module: {module}", module);
                logger.LogWarning("Installation output: {output}", output);
            }
            else
            {
                logger.LogWarning("Python module {module} is already installed.", module);
            }
        }
    }
}

解决方案 #

1. 适配无头服务器环境的浏览器配置 #

IIS运行在无桌面的服务器环境，需明确配置浏览器参数：

# 修改SB初始化代码，添加无头、禁用GPU、固定窗口大小
with SB(uc=True, headless=True, browser_args=["--disable-gpu", "--window-size=1920,1080"]) as sb:
    sb.uc_open_with_reconnect(f"https://www.scopus.com/authid/detail.uri?authorId={userId}", 3)

• headless=True：强制以无头模式运行浏览器，适配无桌面环境
• --disable-gpu：避免服务器无GPU导致的渲染错误
• --window-size：固定窗口大小，确保页面布局与命令行环境一致，防止元素定位偏移

2. 优化元素定位逻辑 #

原脚本依赖文本定位易受页面语言、加载状态影响，改用更稳定的页面标识：

def verify_success(sb):
    # 改用作者名称的data-testid作为验证标识，比文本更可靠
    sb.assert_element('h1[data-testid="author-profile-name"]', timeout=15)
    sb.sleep(2)

3. 移除GUI验证码交互 #

sb.uc_gui_click_captcha()依赖桌面交互，在IIS环境下无法执行，需替换为非GUI方案：

except Exception:
    if sb.is_element_visible('input[value*="Verify"]'):
        sb.uc_click('input[value*="Verify"]')
    # 移除GUI验证码点击，改为日志记录或调用第三方自动识别服务
    else:
        print("Captcha detected, cannot handle in headless environment")
    try:
        verify_success(sb)
    except Exception:
        raise Exception("Detected!")

4. 配置IIS应用池权限 #

• 启用用户配置文件：在IIS应用池高级设置中，将“加载用户配置文件”设为True，确保浏览器能加载必要配置
• 修改应用池身份：将应用池身份改为LocalSystem（或具备桌面交互权限的专用账户），并在服务属性中勾选“允许服务与桌面交互”
• 手动验证权限：使用应用池身份运行脚本，确认环境一致性：

  runas /user:NT AUTHORITY\IUSRS "C:\Python312\python.exe C:\inetpub\site\scopus.py {userId}"
  

5. 优化C#调用逻辑 #

• 指定Python完整路径：避免依赖环境变量，直接使用Python可执行文件绝对路径：

  string cmd = @"C:\Python312\python.exe";
  

• 设置工作目录：确保进程在脚本所在目录运行，防止找不到脚本：

  var start = new ProcessStartInfo
  {
      FileName = cmd,
      Arguments = $"scopus.py {userId}",
      WorkingDirectory = @"C:\inetpub\site",
      UseShellExecute = false,
      RedirectStandardOutput = true,
      RedirectStandardError = true,
      CreateNoWindow = true
  };
  

6. 排查页面实际加载状态 #

在脚本中添加截图功能，确认页面是否被反爬拦截：

try:
    verify_success(sb)
except Exception:
    sb.save_screenshot(@"C:\inetpub\site\scopus_error.png")
    if sb.is_element_visible('input[value*="Verify"]'):
        sb.uc_click('input[value*="Verify"]')
    ...

长期方案：改用Scopus官方API #

直接调用Scopus官方API获取用户信息，比爬虫更稳定合规，彻底避免反爬问题：

• 通过Elsevier开发者平台申请API密钥
• 使用requests库调用API，无需浏览器渲染，性能更高

内容的提问来源于stack exchange，提问作者Abdul Hadi