Python爬虫代码遇400 Bad Request错误：URL访问与元素点击异常求助

编写代码时遇到问题：尝试打开指定URL并点击页面元素时，出现400 Bad Request错误。以下是我的代码：

def had_val(url, ip, num):
    driver = webdriver.Chrome('chromedriver')
    driver.header_overrides = {
        'USER-AGENT':'Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/103.0.5060.63 Mobile/15E148 Safari/604.1',
        'X-FORWARDED-FOR':str(ip),
        'msisdn':str(num),
        'Access-Control-Allow-Credentials':'True',
        'Access-Control-Allow-Headers':'Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With',
        'Access-Control-Allow-Methods':'POST, GET, OPTIONS, DELETE, PUT',
        'Access-Control-Max-Age':'3600',
        'Cache-Control':'no-cache, must-revalidate, private',
        'X-FRAME-OPTIONS':'SAMEORIGIN'
    }
    time.sleep(3)
    driver.get(url)
    try :
        driver.find_element(by=By.XPATH,value='//*[@id="submitbtn"]').click()
        driver.implicitly_wait(10)
    except NoSuchElementException as exception:
        print("Button not found and test failed")

    time.sleep(10)
    driver.quit()


my_file = open('http_proxies.txt')
all_the_lines = my_file.readlines()
ip_range = []
for i in all_the_lines:
    ip_range.append(str(i))
    

def pass_data():
    urls= 'https://affnet.gotrackier.com/t/MjM1XzI4Mg/?p1={tid}'
    wb = openpyxl.load_workbook("Book2.xlsx")
    sheet = wb.active
    m_row = sheet.max_row
    for row in sheet.iter_rows(min_row=1, min_col=1, max_row=m_row, max_col=1):
        for cell in row:
            secure_random = random.SystemRandom()
            random_ip = secure_random.choice(ip_range)
            print(random_ip,"and",cell.value)
        had_val(urls,random_ip,cell.value)
pass_data() 

可能的原因及修复方案 #

1. 请求头包含无效字段 #

你添加了大量服务器响应头（比如Access-Control-Allow-Credentials、Access-Control-Allow-Methods），这些是服务器返回给浏览器的头，不是浏览器发送请求时需要携带的。服务器收到非法请求头会直接返回400错误。

修复：只保留必要的请求头：

# 只保留客户端需要发送的请求头
driver.header_overrides = {
    'USER-AGENT':'Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/103.0.5060.63 Mobile/15E148 Safari/604.1',
    'X-FORWARDED-FOR':str(ip),
    'msisdn':str(num)
}

2. URL参数未替换 #

urls变量中的{tid}占位符没有被替换成实际值，导致URL格式无效，服务器无法解析。

修复：调用had_val时替换参数：

had_val(urls.format(tid=cell.value), random_ip, cell.value)

3. IP地址包含换行符 #

从文件读取IP时，readlines()会保留每行末尾的\n换行符，导致X-FORWARDED-FOR头的IP格式错误。

修复：读取时去除换行符：

for i in all_the_lines:
    ip_range.append(str(i).strip())

4. ChromeDriver设置请求头的方式错误 #

原生webdriver.Chrome没有header_overrides属性，直接设置不会生效，需要通过ChromeOptions配置。

修复：用ChromeOptions添加请求头：

from selenium.webdriver.chrome.options import Options

def had_val(url, ip, num):
    chrome_options = Options()
    headers = {
        'USER-AGENT':'Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/103.0.5060.63 Mobile/15E148 Safari/604.1',
        'X-FORWARDED-FOR':str(ip),
        'msisdn':str(num)
    }
    for key, value in headers.items():
        chrome_options.add_argument(f'--header={key}:{value}')
    
    driver = webdriver.Chrome('chromedriver', options=chrome_options)
    driver.implicitly_wait(10)  # 提前设置隐式等待
    driver.get(url)
    # 后续代码不变

5. 隐式等待位置错误 #

driver.implicitly_wait(10)应该放在初始化driver之后，这样才能对后续所有元素查找生效，而不是点击元素之后才设置。

内容的提问来源于stack exchange，提问作者rohan