使用Python Requests发送POST请求无法获取日本海关HTML表格数据的技术求助

Let's work through why your POST request isn't returning the table data you're targeting. Government sites often have strict session and form validation rules, so small oversights can break your scraper. Here are actionable fixes to try:

1. Use a Persistent Session Instead of Manual Cookies #

Manual cookies are often short-lived or incomplete. Most government sites require you to first load the original search page to establish a valid session before submitting the form. Try using requests.Session() to handle this automatically:

import requests

# Initialize a session to persist cookies and headers across requests
session = requests.Session()
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36',
    'Referer': 'https://www.customs.go.jp/toukei/srch/indexe.htm?M=05&P=1,2,,,,,,,,1,0,2020,0,12,0,2,230660,,,,,,,,,,1,,,,,,,,,,,,,,,,,,,,,,200'
}

# First, load the original search page to set up session cookies
original_page = session.get(
    'https://www.customs.go.jp/toukei/srch/indexe.htm?M=05&P=1,2,,,,,,,,1,0,2020,0,12,0,2,230660,,,,,,,,,,1,,,,,,,,,,,,,,,,,,,,,,200',
    headers=headers
)

# Now submit the form with the established session
data = {
    'CW_SEARCHID': 'JCCHT05S',
    'CW_JAPANKBN': '2',
    'CW_IMPKBN': '2',
    'CW_CARGOKBN': '',
    'CW_SUMKBN': '',
    'CW_SPCODE': '',
    'CW_SPNAME': '',
    'CW_YMSORTKBN': '',
    'CW_SISUKBN': '',
    'CW_SENKIKBN': '',
    'CW_HKKBN': '',
    'CW_YMKBN': '1',
    'CW_KI': '',
    'CW_SYY': '2020',
    'CW_EYY': '',
    'CW_SMM': '12',
    'CW_EMM': '',
    'CW_HSKBN': '2',
    'CW_HSCODE': '230660',
    'CW_HSNAME': '',
    'CW_KUNIKBN': '1',
    'CW_KUNICODE': '',
    'CW_KUNINAME': '',
    'CW_ZMKBN': '1',
    'CW_ZMCODE': '',
    'CW_ZMNAME': '',
    'CW_MEISAICNT': '200'
    # Include ALL empty form fields from the browser's request—even blank ones matter!
}

target_url = "https://www.customs.go.jp/JCWSV02/servlet/JCWSV02"
response = session.post(target_url, headers=headers, data=data)

# Debug check: print status code and first 500 characters of response
print(f"Status Code: {response.status_code}")
print(response.text[:500])

2. Include Every Form Parameter (Even Empty Ones) #

Your initial data dictionary was missing several fields that the browser sends in a valid request (like CW_CARGOKBN or CW_SUMKBN). Government forms often require every field to be present, even if the value is empty. To get the full list:

• Open Chrome DevTools → Network tab
• Submit the form manually in your browser
• Find the POST request to JCWSV02 → Go to the "Form Data" section
• Copy every key-value pair into your data dictionary (including empty strings)

3. Check for Hidden CSRF Tokens #

Many sites inject hidden CSRF tokens into forms to block automated requests. Look for hidden inputs in the original search page's HTML (using original_page.text or DevTools) like:

<input type="hidden" name="CSRF_TOKEN" value="abc123xyz">

If you find one, extract its value and add it to your data dictionary.

4. Match Browser Headers Exactly #

Government sites often validate headers like Referer, Accept, and Origin. Copy all request headers from the browser's successful POST into your headers dictionary. For example:

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36',
    'Referer': 'https://www.customs.go.jp/toukei/srch/indexe.htm?M=05&P=1,2,,,,,,,,1,0,2020,0,12,0,2,230660,,,,,,,,,,1,,,,,,,,,,,,,,,,,,,,,,200',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Origin': 'https://www.customs.go.jp',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

5. Debug the Response for Clues #

If you're still stuck, check these details:

• response.status_code: A 403 means the site is blocking you (try adding more browser-like headers or a small time.sleep() to avoid rate limiting)
• response.headers: Look for Location headers that point to an error page
• response.text: Search for keywords like "error" or "invalid" to see if the site is returning a rejection message instead of data

内容的提问来源于stack exchange，提问作者Hanalia