为何GCP虚拟机向NSE网站发送GET请求无响应，本地机器可正常访问？

Certificate chain
0 s:C = IN, ST = Maharashtra, L = Mumbai, O = National Stock Exchange of India Ltd., OU = IT, CN = www.nseindia.com
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = GeoTrust RSA CA 2018
1 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = GeoTrust RSA CA 2018
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
2 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA #

Server certificate
-----BEGIN CERTIFICATE-----
CERTIFICATE REMOVED
-----END CERTIFICATE-----
subject=C = IN, ST = Maharashtra, L = Mumbai, O = National Stock Exchange of India Ltd., OU = IT, CN = www.nseindia.com
issuer=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = GeoTrust RSA CA 2018 #

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-256, 256 bits #

SSL handshake has read 6232 bytes and written 444 bytes
Verification: OK #

New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: xxxxxxxxxxxxxxxxxxxxxxx
Session-ID-ctx:
Master-Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - 00 00 26 30 a1 f5 2f bf-a1 57 5e 66 57 e6 b6 f6 ..&0../..W^fW...
.... ....
00a0 - b6 84 52 56 e5 fb f1 4f-03 c5 8e 1a 04 c1 b8 6b ..RV...O.......k
Start Time: 1583754185
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no #

GET / HTTP/1.1
Host: www.nseindia.com
accept-encoding: gzip, deflate, br
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-language: en-GB,en-US;q=0.9,en;q=0.8
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36
cache-control: no-cache
pragma: no-cache
upgrade-insecure-requests: 1

---

### 问题分析
This is a super common issue with financial websites like NSE—they actively block cloud provider IP ranges (including GCP) because these IPs are frequently used for large-scale scraping that violates their terms of service. 

The key clue here is that your SSL handshake succeeds but the server ignores your GET request, leading to a timeout. This is a silent ban tactic: NSE allows the initial connection to avoid obvious blocking, but drops all subsequent traffic from flagged IPs.

Other possible factors:
- **Environment fingerprinting**: Cloud VMs have distinct network traits (like TCP stack signatures or DNS behavior) that NSE's anti-scraping systems detect, even if your IP isn't explicitly blocked.
- **Regional restrictions**: NSE might prioritize traffic from Indian-based IPs, and GCP's non-Indian regions are more likely to be blocked.

---

### 解决方案

#### 1. Use Residential Proxies (Most Effective)
The quickest fix is to route your requests through a residential proxy (preferably based in India). Residential IPs are tied to real user devices, so they're much less likely to be flagged by NSE's anti-scraping tools.

Update your Python code to use a proxy:
```python
from requests import Session

headers = {
    "Host": "www1.nseindia.com",
    "Referer": "https://www1.nseindia.com/products/content/equities/equities/eq_security.htm",
    "X-Requested-With": "XMLHttpRequest",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
    "Cache-Control": "no-cache",
    "Connection": "keep-alive",
}

s = Session()
s.headers.update(headers)

# Replace with your proxy details
proxies = {
    "http": "http://your-residential-proxy:port",
    "https": "https://your-residential-proxy:port"
}
s.proxies.update(proxies)

url = "https://www.nseindia.com"
r = s.get(url)
print(r.text)

2. Switch GCP VM to Mumbai Region

Try deploying your VM in GCP's asia-south1 (Mumbai) region. NSE might have looser restrictions on Indian-based cloud IPs since it's a local service. Create a new VM instance in this region and test your code again.

3. Mimic Real Browser Behavior More Closely

NSE's anti-scraping systems check for realistic request patterns. You can improve your success rate by:

• Adding valid cookies from a real browser session (visit NSE in Chrome/Firefox, copy the cookies into your request headers)
• Adding random delays between requests (import time; time.sleep(2) before each GET)
• Using tools like undetected-chromedriver to simulate a real browser, which bypasses basic anti-scraping checks

4. Use NSE's Official Data Interfaces

If you're accessing public data legally, consider using the NSEpy library—it wraps NSE's official public data APIs, so you won't run into blocking issues:

from nsepy import get_history
from datetime import date

# Fetch historical data for INFOSYS
data = get_history(symbol='INFY', start=date(2024, 1, 1), end=date(2024, 1, 31))
print(data.head())

Verification Step #

Test your connection from GCP using curl with a proxy first to confirm it works:

curl -x http://your-proxy-ip:port -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" https://www.nseindia.com

If this returns the website HTML, your proxy is working, and you can safely update your Python code.

内容的提问来源于stack exchange，提问作者Casual Coder