Python网络爬虫POST登录网络设备时遇会话过期问题求助

问题描述 #

尝试通过Python requests登录NETGEAR M4300-52G设备以访问下载端点，将curl请求转换为代码后执行POST请求，返回会话过期页面。已尝试使用requests.session和爬取CSRF Token，但问题仍未解决。

当前代码 #

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'de,en-US;q=0.7,en;q=0.3',
    # 'Accept-Encoding': 'gzip, deflate',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Origin': 'http://192.168.18.96',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Referer': 'http://192.168.18.96/v1/base/cheetah_login.html',
    'Cookie': 'testcookie; SID=a243eb355e16a079226a443d4a0c5f0ff453f032a99e8752084796c681aff36a66de4e88bb0aad7f272b3be8fe011c9364f4659efcc0d08f2ee98f7180944760; testcookie; Language_In_Use=',
    'Upgrade-Insecure-Requests': '1',
}

data = {
    'CSRFToken': '1679065361ce165a07724a0553789eee63938e08ab7be0148728a5f79878469cefbb3bfb6938ed526996fd9cee8e8c307ad4177e4abb257f4ae8be878e88a8f40a2ad97e92',
    'uname': 'admin',
    'pwd': 'password',
    'err_flag': '',
    'err_msg': '',
    'submt': '',
}

response = requests.post('http://192.168.18.96/v1/base/cheetah_login.html', headers=headers, data=data)
print(response.text)

返回的会话过期页面 #

<!--
/*********************************************************************
* Copyright 2016-2021 Broadcom.
**********************************************************************
*
* @filename 
*
* @purpose Login page
*
* @comments 
*
* @create 
*
* @author 
* @end
*
**********************************************************************/
-->
<html>

<head>
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
<LINK REL=stylesheet HREF="/v1/base/style.css" TYPE="text/css">
<link rel=stylesheet href="/v1/base/nanoscroller.css" TYPE="text/css">
<META http-equiv="Pragma" content="no-cache">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">   
  <TITLE>NETGEAR M4300-52G</TITLE>       <!-- Cheetah Page Title  -->
  <link rel="shortcut icon" href="/v1/base/favicon.ico">             <!-- fav icon -->

  <script type='text/javascript' src='/v1/base/js/jquery-3.6.0.min.js'></script>

  <script type='text/javascript' src='/v1/base/js/jquery.cookie.js?v="12.0.17.7"'></script>

  <script type='text/javascript' src='/v1/base/js/ng_help.js?v="12.0.17.7"'></script>

  <script type='text/javascript' src='/v1/base/js/nav_nls.js?v="12.0.17.7"'></script>


  <style>

  .loginPage_inlineEr_padding
  {
    padding-left: 25px;
  }
  h1 
  {
    font: 14pt Arial;
    color: #672E8F;
    white-space: nowrap;
    padding-left: 25px;
    text-align: center;
  }

  </style>
</head>

<script>
var timeOut = 0;
var timerNum = 0;
var presentHost;
function  expiredLoad()
{
  timerNum = setInterval(reloginTimeout, 1000);
  var head = document.getElementById("expiredmessage");
  var timeRemaining = 10-timeOut;
  presentHost = top.location.protocol+ "//" +top.location.host;
  var tag = "<a href=\""+presentHost+"\">"+js_NLS_here+"</a>";
  var msg = js_NLS_valMapSessionExpiredMsg1 + tag + js_NLS_valMapSessionExpiredMsg2 + String(timeRemaining) + js_NLS_seconds;
  head.innerHTML = msg; 
}

function  reloginTimeout()
{
  timeOut = timeOut + 1;
  if(timeOut == 10)
  {
    clearInterval(timerNum);
    var presentHost;
    presentHost = top.location.protocol+ "//" + top.location.host;
    top.location.replace(presentHost);
  }  
  var head = document.getElementById("expiredmessage");
  var timeRemaining = 10-timeOut;
  presentHost = top.location.protocol+ "//" +top.location.host;
  var tag = "<a href=\""+presentHost+"\">"+js_NLS_here+"</a>";
  var msg = js_NLS_valMapSessionExpiredMsg1 + tag + js_NLS_valMapSessionExpiredMsg2 + String(timeRemaining) + js_NLS_seconds;
  head.innerHTML = msg; 
}

</script>
<body onLoad="expiredLoad()">

  <div class="spacer100Percent">
      <div class="loginPage_mainPadding top">
        <div class="netgearLogoPadding">
          <a href="http://www.netgear.com/">
            <img width="138" alt="Netgear" src="/v1/base/images/Netgear-logo.png"/>
          </a>
        </div>
        <div class="separator13"></div>
          <div>
            <label class="productName">
              M4300-52G ProSAFE 48-port 1G and 2-port 10GBASE-T and 2-port 10G SFP+
            </label>
          </div>
        </div>
        <h1 id = "expiredmessage"> The login page has expired. Will be redirected to login page Automatically </h1>
      </div>
  </div>
  <div id="footer_image" class="bottom_image" style="height: 408px; padding-top: 40px;"></div>
  <div id="branding_text" class="footer paddingLeft25 separator25" style="position: absolute; top: 745px; line-height: 25px; vertical-align: middle;">
        &copy; 2022 NETGEAR, Inc. All rights reserved.
  </div>
</body>
</html>

可能的解决方案 #

1. 移除硬编码Cookie值 #

当前代码中硬编码的SID等Cookie属于旧会话，服务器会直接判定会话过期。必须使用requests.Session()自动管理Cookie，禁止手动设置Cookie头。

2. 动态获取CSRF Token #

NETGEAR设备的CSRF Token是每次访问登录页面时动态生成的，硬编码的Token无效。需要先GET登录页面，从HTML中提取最新Token后再发起POST请求。

3. 模拟完整登录流程 #

完整的登录流程应该是：

• 创建Session对象自动维护会话状态
• GET登录页面，获取初始Cookie和CSRF Token
• 使用新Token和登录凭证发送POST请求
• 后续下载请求复用同一个Session

修正后的示例代码：

import requests
from bs4 import BeautifulSoup

# 创建会话对象，自动管理Cookie
session = requests.Session()

# 1. 访问登录页面，获取CSRF Token和初始Cookie
login_page_url = "http://192.168.18.96/v1/base/cheetah_login.html"
response = session.get(login_page_url)
soup = BeautifulSoup(response.text, 'html.parser')

# 提取页面中的CSRF Token（根据实际HTML结构调整选择器）
csrf_token = soup.find('input', {'name': 'CSRFToken'}).get('value')

# 2. 构造登录请求数据
login_data = {
    'CSRFToken': csrf_token,
    'uname': 'admin',
    'pwd': 'password',
    'err_flag': '',
    'err_msg': '',
    'submt': ''
}

# 3. 发送登录POST请求
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'de,en-US;q=0.7,en;q=0.3',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Origin': 'http://192.168.18.96',
    'Referer': login_page_url,
    'Upgrade-Insecure-Requests': '1'
}

login_response = session.post(login_page_url, headers=headers, data=login_data)

# 验证登录结果
if "session expired" not in login_response.text.lower():
    print("登录成功，可继续访问下载端点")
    # 示例：访问下载接口
    # download_response = session.get("http://192.168.18.96/下载路径")
    # with open("文件名称", "wb") as f:
    #     f.write(download_response.content)
else:
    print("登录失败，返回内容：")
    print(login_response.text)

4. 额外注意事项 #

• 需提前安装BeautifulSoup：pip install beautifulsoup4
• 若CSRF Token不在隐藏input中，需根据页面实际HTML结构调整提取逻辑
• 若仍失败，可开启requests调试日志排查请求细节：

import logging
logging.basicConfig(level=logging.DEBUG)

内容的提问来源于stack exchange，提问作者LiiVion