Jsoup.connect使用有效登录凭证/Cookie仍抛出403错误排查

此前正常运行的Jsoup代码突然报错：

HTTP error fetching URL. Status=403, URL=[https://www.valueresearchonline.com/funds/26123/motilal-oswal-flexi-cap-fund-regular-plan]

我已通过账号（vikram.silk@gmail.com / 0987654321）登录该网站获取了Cookie值，但无法确定代码失效的原因。

失效代码示例1 #

Jsoup.connect("https://www.valueresearchonline.com/funds/26123/motilal-oswal-flexi-cap-fund-regular-plan")
        .timeout(15000)
        .userAgent("Mozilla")
        .header("Cookie", "PHPSESSID=6d9v48p1i5lpgm7pi75ag0okvq; currency=INR; magnitude=LC; ad=ee5ceff9a39de83a4dfcd9cc96efd7aa04912966; ad=ee5ceff9a39de83a4dfcd9cc96efd7aa04912966; wec=296642702; nobtlgn=714443298; ac=68156306%7C526294102%7C424761678; ac=68156306%7C526294102%7C424761678; _gcl_au=1.1.443352322.1663577511; _gid=GA1.2.307799405.1663577512; _fbp=fb.1.1663577512204.684634655; _clck=fgstbv|1|f50|0; __gads=ID=5ed6c786ebad7ee2-2263761e9bd600b6:T=1663577512:S=ALNI_Mb0p_Gif2EChNCORy7JOdTU7x4kjA; __gpi=UID=000009ce9ba54907:T=1663577512:RT=1663577512:S=ALNI_MbPL5xoHxgY76gU9mzDzJltULL80Q; __cf_bm=iIVI9aabT4vAdAmvQQzQTDDs9z4MPaMB1gv602Vn2rI-1663577514-0-AQ9ZKhXneLwVKm6CKEzLoY2EKcrIlNB82wgEPDw7taV6k/fnqTzp0L5zrpAl0fnkF1dn7Ac1DyNdfOnsgCTjBZx5Y6ia4Pvj2ceyIBfyXcIYpR8JkYTYGHfqPlrncv7k6Q==; alp=VROL; aa=364476%7C230264168%7C651860858; aa=364476%7C230264168%7C651860858; arl=604590238; arl=604590238; PERMA-ALERT=0; pgv=6; _ga=GA1.1.1410692956.1663577512; _ga_N9R425YFBJ=GS1.1.1663577511.1.1.1663577540.31.0.0; _clsk=1prm412|1663577540567|4|1|l.clarity.ms/collect")
        .method(Connection.Method.GET)
        .execute();

失效代码示例2 #

Connection.Response initial1 = Jsoup.connect("https://www.valueresearchonline.com/login/?target=%2fmyaccount")
        .timeout(60000)
        .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0")
        .data("username", "vikram.silk@gmail.com")
        .data("password", "0987654321")
        .method(Connection.Method.POST)
        .execute();
System.out.println("Call 1.1 " + initial1.statusCode());
System.out.println(initial1.cookies().values());

Chrome登录调试控制台截图：

更新1：硬编码Cookie可运行，但需动态获取 #

使用Chrome控制台获取的硬编码Cookie可使代码正常运行，但不清楚如何动态获取步骤1中的Cookie值。

步骤1（硬编码Cookie值） #

String apiUrl = "https://www.valueresearchonline.com/api/check-user/";

// Connect to the API URL
Connection.Response response1 = Jsoup.connect(apiUrl)
        .method(Connection.Method.GET)
        .timeout(60000)
        .header("User-Agent", "Mozilla/5.0")
        .header("Accept", "application/json")
        //Hard Coded, taken from Chrome after valid email is entered.
        //Can we get this cookie value dynamically or runtime ??
        .header("Cookie", "HARD_CODED_COOKIE_VALUE_FROM_CHRIME")
        .data("q", "vikram.silk@gmail.com")
        .data("password", "1")
        .ignoreContentType(true)  // Ignore content type to parse non-HTML response
        .execute();

// Get the cookies from the response
Map<String, String> cookies1 = response1.cookies();

步骤2 - 使用步骤1生成的Cookie值 #

Connection.Response response2 = Jsoup.connect("https://www.valueresearchonline.com/login/?target=%2f%3f&utm_source=home&utm_medium=vro&utm_campaign=desktop-profile-menu")
        .timeout(60000)
        .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0")
        .header("Cookie", getCookiesString(cookies1))
        .header("Host", "www.valueresearchonline.com")
        .data("username", "vikram.silk@gmail.com")
        .data("password", "0987654321")
        .method(Connection.Method.POST)
        .ignoreContentType(true)
        .execute();

问题：如何不通过硬编码获取步骤1中的Cookie值？

更新2：cURL正常但代码仍失效 #

正常运行的cURL命令 #

curl 'https://www.valueresearchonline.com/login/?site-code=vrol&target=%2f%3f&utm_source=home&utm_medium=vro&utm_campaign=desktop-profile-menu' \
  -H 'accept: */*' \
  -H 'accept-language: en-US,en;q=0.9,de;q=0.8' \
  -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' \
  -H 'cookie: currency=INR; .......' \
  -H 'login_by: null' \
  -H 'origin: https://www.valueresearchonline.com' \
  -H 'priority: u=1, i' \
  -H 'referer: https://www.valueresearchonline.com/login/?site-code=VROL&target=%2F%3F&utm_source=home&utm_medium=vro&utm_campaign=desktop-profile-menu' \
  -H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"' \
  -H 'sec-ch-ua-arch: "x86"' \
  -H 'sec-ch-ua-bitness: "64"' \
  -H 'sec-ch-ua-full-version: "126.0.6478.183"' \
  -H 'sec-ch-ua-full-version-list: "Not/A)Brand";v="8.0.0.0", "Chromium";v="126.0.6478.183", "Google Chrome";v="126.0.6478.183"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-model: ""' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-ch-ua-platform-version: "11.5.2"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36' \
  -H 'x-requested-with: XMLHttpRequest' \
  --data-raw 'username=vikram....%40gmail.com&password=NUKegc2A%2BJ3qEh6.....'

仍返回403错误的Jsoup代码 #

URL = "https://www.valueresearchonline.com/api/user-data/?url-path=/funds/26123/motilal-oswal-flexi-cap-fund-regular-plan&referrer=";

response2 = Jsoup.connect(URL)
        .timeout(MAX_TIMEOUT + (retry * MAX_TIMEOUT))
        .userAgent(userAgentValue)
        .header("Cookie", hardCodedCookieValue)
        .header("Origin", "https://www.valueresearchonline.com")
        .header("Referer", "https://www.valueresearchonline.com/login/?site-code=VROL&target=%2F%3F&utm_source=home&utm_medium=vro&utm_campaign=desktop-profile-menu")
        //.header("Host", "www.valueresearchonline.com")
        .data("username", "valid_email@gmail.com")
        .data("password", "valid_pwd")
        .method(Connection.Method.POST)
        .ignoreContentType(true)
        .execute();

内容的提问来源于stack exchange，提问作者vikramvi