如何用Python提取动态加载iframe中#document内的iframe src链接

问题描述 #

问题1：目标iframe需点击按钮触发AJAX后出现 #

点击按钮前，目标区域HTML：

<div class="video-content" id="video">
 <div class="autosize-container">
    <a href="#stream" id="stream" title="xyz"><img src="/img/tem/image.png"></a>
  </div>
</div>

触发按钮代码：

<a href="#video" data-mirror="1" data-host="1" title="xyz">xyz</a>

按钮点击后触发AJAX请求，返回的HTML包含表单和无src的外层iframe，自动提交表单后填充iframe内容：

<form id="moodleform" target="iframe" method="post" action="">
      <input type="hidden" name="req" value="2" />
      <input type="hidden" name="pid" value="57316" />
      <input type="hidden" name="mirror" value="1" />
      <input type="hidden" name="host" value="1" />
      <input type="hidden" name="rel" value="71584,0" />
    </form>
    <iframe name="iframe" scrolling="no" frameborder="0" width="100%" height="480px" allowfullscreen="true" webkitallowfullscreen="true" mozallowfullscreen="true" style="background-color:#eee"></iframe>
    <script>
      document.getElementById('moodleform').submit();
    </script>

已通过Python模拟AJAX请求获取到上述HTML内容。

问题2：外层iframe无src，内部#document嵌套带目标src的子iframe #

浏览器开发者工具中可见外层iframe的#document内有带目标src的子iframe：

<iframe src="https://example.com" scrolling="no" frameborder="0" width="100%" height="480px" allowfullscreen="true" webkitallowfullscreen="true" mozallowfullscreen="true"></iframe>

但直接用requests无法获取该内容，需进一步处理。

解决方案 #

方法1：模拟表单提交（轻量无浏览器） #

利用返回的表单信息，手动提交请求获取iframe内容：

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

# 1. 发送初始AJAX请求获取表单
ajax_url = "https://website.com"
ajax_data = {
    "req": "1",
    "pid": "57316",
    "mirror": "1",
    "host": "1",
    "rel": "71584,0"
}
response = requests.post(ajax_url, data=ajax_data, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

# 2. 提取表单的action和数据
form = soup.find('form', id='moodleform')
form_action = form.get('action') or ajax_url  # 若action为空，复用初始AJAX地址
form_data = {input.get('name'): input.get('value') for input in form.find_all('input', type='hidden')}

# 3. 提交表单获取iframe内容
iframe_response = requests.post(form_action, data=form_data, headers=headers)
iframe_soup = BeautifulSoup(iframe_response.text, 'html.parser')

# 4. 提取目标src
target_iframe = iframe_soup.find('iframe')
print(target_iframe.get('src') if target_iframe else "未找到目标iframe")

方法2：无头浏览器模拟真实行为（应对复杂JS场景） #

使用Selenium模拟浏览器执行JS提交表单、加载iframe：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import time

# 配置无头Chrome
chrome_options = Options()
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36')

driver = webdriver.Chrome(options=chrome_options)

try:
    # 加载AJAX返回的HTML内容
    ajax_html = response.text  # 这里用第一步AJAX请求的返回结果
    driver.get(f"data:text/html;charset=utf-8,{ajax_html}")
    
    # 等待表单提交和iframe加载完成
    time.sleep(2)
    
    # 切换到外层iframe
    outer_iframe = driver.find_element(By.NAME, 'iframe')
    driver.switch_to.frame(outer_iframe)
    
    # 提取目标src
    target_iframe = driver.find_element(By.TAG_NAME, 'iframe')
    print(target_iframe.get_attribute('src'))
    
finally:
    driver.quit()

内容的提问来源于stack exchange，提问作者Ammy