Java爬取Yahoo Finance时Cookie与Crumb获取失败求助

我尝试用Java从Yahoo Finance爬取Cookie和Crumb，以获取每日股票价格历史数据。修改了Stack Overflow帖子中的代码后无法正常运行，核心问题是页面源码中找不到"CrumbStore"字样；同时获取到三个不同的Cookie，不知道该使用哪一个。

我的代码 #

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class GetYahooQuotes {

    public static void main(String args[]) throws IOException {

        String cookie = null;

    try {
            // Open the URL connection
        URL url = new URL("https://finance.yahoo.com/quote/SPY");
        URLConnection con = url.openConnection();

        for (Map.Entry<String, List<String>> entry : con.getHeaderFields().entrySet()) {
            if (entry.getKey() == null || !entry.getKey().equals("Set-Cookie"))
                    continue;
                for (String s : entry.getValue()) {
                    // store your cookie
                    cookie = s;
                    System.out.println( "Cookie = " + cookie);
                }
            }

        String crumb = null;
        InputStream inStream = con.getInputStream();
        InputStreamReader irdr = new InputStreamReader(inStream);
        BufferedReader rsv = new BufferedReader(irdr);

        Pattern crumbPattern = Pattern.compile(".*\"CrumbStore\":\\{\"crumb\":\"([^\"]+)\"\\}.*");

        String line = null;
        while (crumb == null && (line = rsv.readLine()) != null) {
            Matcher matcher = crumbPattern.matcher(line);
            if (matcher.matches()) {
                crumb = matcher.group(1);
                System.out.println( "Crumb = " + crumb);
            }
        }
        rsv.close();
    }
        catch (java.net.SocketTimeoutException e) {
        // The URL connection timed out.  Try again.
        }
    }
}

Cookie输出结果 #

Cookie = A1S=d=AQABBLeLr2MCEKfVotZCLRRvj9tdxyruiIkFEgEBAQHdsGO5YwAAAAAA_eMAAA&S=AQAAAtgfY5LB0heCm1MP9BNfRAA&j=WORLD; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure
Cookie = A3=d=AQABBLeLr2MCEKfVotZCLRRvj9tdxyruiIkFEgEBAQHdsGO5YwAAAAAA_eMAAA&S=AQAAAtgfY5LB0heCm1MP9BNfRAA; Expires=Sun, 31 Dec 2023 07:09:12 GMT; Max-Age=31557600; Domain=.yahoo.com; Path=/; SameSite=None; Secure; HttpOnly
Cookie = A1=d=AQABBLeLr2MCEKfVotZCLRRvj9tdxyruiIkFEgEBAQHdsGO5YwAAAAAA_eMAAA&S=AQAAAtgfY5LB0heCm1MP9BNfRAA; Expires=Sun, 31 Dec 2023 07:09:12 GMT; Max-Age=31557600; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure; HttpOnly

我知道获取Crumb后需要构造如下链接获取数据，但目前还未成功获取Crumb：

String quoteUrl = "https://query1.finance.yahoo.com/v7/finance/download/IBM?period1=1493425217&period2=1496017217&interval=1d&events=history&crumb="
                           + crumb

一、Crumb获取失败的修复 #

Yahoo Finance页面结构已更新，CrumbStore不再直接出现在页面源码中，当前Crumb嵌套在window.App.main的JSON结构里。修改代码如下：

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class GetYahooQuotes {

    public static void main(String args[]) throws IOException {

        List<String> cookies = new ArrayList<>();

        try {
            // 打开链接并设置浏览器请求头，避免反爬拦截
            URL url = new URL("https://finance.yahoo.com/quote/SPY");
            URLConnection con = url.openConnection();
            con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36");

            // 收集所有Cookie的键值部分（去掉属性字段）
            for (Map.Entry<String, List<String>> entry : con.getHeaderFields().entrySet()) {
                if (entry.getKey() == null || !entry.getKey().equals("Set-Cookie"))
                    continue;
                for (String s : entry.getValue()) {
                    String cookieValue = s.split(";")[0];
                    cookies.add(cookieValue);
                    System.out.println( "Cookie = " + cookieValue);
                }
            }

            String crumb = null;
            InputStream inStream = con.getInputStream();
            InputStreamReader irdr = new InputStreamReader(inStream);
            BufferedReader rsv = new BufferedReader(irdr);

            // 先提取window.App.main中的JSON块
            Pattern appPattern = Pattern.compile("window\\.App\\.main = (.*?);\\n");
            String line;
            String appJson = null;
            while ((line = rsv.readLine()) != null) {
                Matcher appMatcher = appPattern.matcher(line);
                if (appMatcher.find()) {
                    appJson = appMatcher.group(1);
                    break;
                }
            }

            // 从JSON块中提取Crumb
            if (appJson != null) {
                Pattern crumbPattern = Pattern.compile("\"crumb\":\"([^\"]+)\"");
                Matcher crumbMatcher = crumbPattern.matcher(appJson);
                if (crumbMatcher.find()) {
                    crumb = crumbMatcher.group(1);
                    System.out.println( "Crumb = " + crumb);
                }
            }

            rsv.close();

            // 构造下载链接并携带所有Cookie
            String cookieStr = String.join("; ", cookies);
            String quoteUrl = "https://query1.finance.yahoo.com/v7/finance/download/IBM?period1=1493425217&period2=1496017217&interval=1d&events=history&crumb=" + crumb;
            System.out.println("下载链接：" + quoteUrl);

            // 发起下载请求（示例）
            URL downloadUrl = new URL(quoteUrl);
            URLConnection downloadCon = downloadUrl.openConnection();
            downloadCon.setRequestProperty("Cookie", cookieStr);
            downloadCon.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36");
            // 后续可处理下载流，读取股票数据...

        } catch (java.net.SocketTimeoutException e) {
            e.printStackTrace();
        }
    }
}

二、Cookie选择问题 #

不需要单独选择某个Cookie，所有返回的Cookie键值对都需要携带。Yahoo的验证机制需要多个Cookie协同验证，只需将每个Cookie的键值部分（;之前的内容）拼接成key1=value1; key2=value2; key3=value3的格式，在请求下载接口时放入Cookie请求头即可。

三、额外注意事项 #

• 必须设置User-Agent请求头模拟浏览器，否则Yahoo会拒绝请求。
• Yahoo的页面结构和接口可能随时更新，需定期检查正则表达式是否有效。
• 控制请求频率，避免触发反爬机制导致IP被封禁。

内容的提问来源于stack exchange，提问作者sawong