已设置浏览器标识与Cookie仍出现401 Unauthorized错误求助

问题描述 #

我通过Java代码从印度国家证券交易所（NSE）的REST API获取期权链数据，流程为：先请求官网首页提取Cookie，再使用该Cookie发起期权链数据请求，循环定时重复这两步操作。但流程仅能成功1-2次，之后持续返回401 Unauthorized错误，已在两次请求的请求头中设置浏览器标识，寻求技术解决办法。

现有代码 #

import java.net.HttpURLConnection;
import java.net.URL;
import java.util.List;
import java.io.InputStream;

public class PollNSEIndia {
    public static void main(String args[]) throws Exception {
        while (true) {
            HttpURLConnection baseUrlConnection = (HttpURLConnection) new URL("https://www.nseindia.com/").openConnection();
            baseUrlConnection.setRequestProperty("Connection", "keep-alive");
            baseUrlConnection.setRequestProperty("Cache-Control", "max-age=0");
            baseUrlConnection.setRequestProperty("Upgrade-Insecure-Requests", "1");
            baseUrlConnection.setRequestProperty(
                    "User-Agent",
                    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)"
                            + " Chrome/89.0.4389.114 Safari/537.36");
            baseUrlConnection.setRequestProperty(
                    "Accept",
                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
            baseUrlConnection.setRequestProperty("Accept-Language", "en-US,en;q=0.9");
            List<String> cookies = baseUrlConnection.getHeaderFields().get("Set-Cookie");

            URL url = new URL("https://www.nseindia.com/api/option-chain-indices?symbol=MIDCPNIFTY");
            HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
            httpURLConnection.setRequestMethod("GET");
            for (String cookie : cookies) {
                httpURLConnection.addRequestProperty("Cookie", cookie.split(";", 2)[0]);
            }
            httpURLConnection.setRequestProperty("Connection", "keep-alive");
            httpURLConnection.setRequestProperty("Cache-Control", "max-age=0");
            httpURLConnection.setRequestProperty("Upgrade-Insecure-Requests", "1");
            httpURLConnection.setRequestProperty(
                    "User-Agent",
                    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)"
                            + " Chrome/89.0.4389.114 Safari/537.36");
            httpURLConnection.setRequestProperty(
                    "Accept",
                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
            httpURLConnection.setRequestProperty("Accept-Language", "en-US,en;q=0.9");
            InputStream inputStream = httpURLConnection.getInputStream();
            System.out.println("Got inputstream.");
            Thread.sleep(1000);
        }
    }
}

问题分析与解决方案 #

1. Cookie复用策略错误 #

每次循环都重新请求首页获取新Cookie，会触发NSE的Session风控机制。正确做法是复用同一个Session Cookie，仅当请求返回401时再重新获取新Cookie，避免频繁更换Session。

2. 请求头缺失关键字段 #

NSE的API会校验请求来源和请求类型，当前代码缺少两个关键请求头：

• Referer: https://www.nseindia.com/：告诉服务器请求来自NSE官网，模拟正常用户操作路径
• X-Requested-With: XMLHttpRequest：模拟浏览器AJAX请求，匹配NSE API的调用场景

3. 请求频率过高 #

当前循环间隔仅1秒，远高于正常用户操作频率，直接触发反爬机制。建议将间隔调整为15-30秒，贴近真实用户行为。

4. Cookie处理需完善 #

需添加Cookie列表为空的判断，避免空指针异常；同时统一拼接Cookie字符串，避免多次调用addRequestProperty导致的头信息混乱。

修改后的代码示例 #

import java.net.HttpURLConnection;
import java.net.URL;
import java.util.List;
import java.io.InputStream;

public class PollNSEIndia {
    private static String currentCookies = "";

    public static void main(String args[]) throws Exception {
        // 初始化获取一次Cookie
        refreshCookies();

        while (true) {
            try {
                HttpURLConnection apiConn = (HttpURLConnection) new URL("https://www.nseindia.com/api/option-chain-indices?symbol=MIDCPNIFTY").openConnection();
                apiConn.setRequestMethod("GET");
                // 复用已有的Cookie
                apiConn.setRequestProperty("Cookie", currentCookies);
                // 添加缺失的关键请求头
                apiConn.setRequestProperty("Referer", "https://www.nseindia.com/");
                apiConn.setRequestProperty("X-Requested-With", "XMLHttpRequest");
                // 精简必要的请求头
                apiConn.setRequestProperty("Connection", "keep-alive");
                apiConn.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36");
                apiConn.setRequestProperty("Accept", "application/json, text/javascript, */*; q=0.01");
                apiConn.setRequestProperty("Accept-Language", "en-US,en;q=0.9");

                int responseCode = apiConn.getResponseCode();
                if (responseCode == 401) {
                    // 401时刷新Cookie并重试
                    refreshCookies();
                    continue;
                } else if (responseCode == 200) {
                    InputStream inputStream = apiConn.getInputStream();
                    System.out.println("数据获取成功");
                    // 此处可添加数据处理逻辑
                } else {
                    System.out.println("意外响应码: " + responseCode);
                }
            } catch (Exception e) {
                e.printStackTrace();
                // 异常时尝试刷新Cookie
                refreshCookies();
            }
            // 调整请求间隔为30秒
            Thread.sleep(30000);
        }
    }

    private static void refreshCookies() throws Exception {
        HttpURLConnection homeConn = (HttpURLConnection) new URL("https://www.nseindia.com/").openConnection();
        homeConn.setRequestProperty("Connection", "keep-alive");
        homeConn.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36");
        homeConn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
        homeConn.setRequestProperty("Accept-Language", "en-US,en;q=0.9");

        List<String> cookies = homeConn.getHeaderFields().get("Set-Cookie");
        if (cookies != null) {
            StringBuilder cookieBuilder = new StringBuilder();
            for (String cookie : cookies) {
                cookieBuilder.append(cookie.split(";", 2)[0]).append("; ");
            }
            if (cookieBuilder.length() > 0) {
                currentCookies = cookieBuilder.substring(0, cookieBuilder.length() - 2);
            }
            System.out.println("已刷新Cookie");
        }
        homeConn.disconnect();
    }
}

额外注意事项 #

• NSE的API反爬策略可能随时调整，若后续仍出现401，需进一步校验请求头或模拟更完整的浏览器行为
• 严格遵守NSE的使用条款，避免过度爬虫，防止触发法律风险

内容的提问来源于stack exchange，提问作者Kannan J