Requests+BeautifulSoup爬取预订网站时文本与值字段为空

问题描述 #

我尝试从阿尔卑斯山小屋预订网站爬取可用房间信息，运行代码后获取的所有标签文本和值字段均为空。

我的实现代码 #

import requests
from bs4 import BeautifulSoup

#FORMAT: TT_tt_MMM_jjjj
#At max 2 weeks in advance
date = "Sa 23 Sep 2023"
min_space = 3
only_big_rooms = False
hut_id = 92


URL = "https://www.alpsonline.org/reservation/calendar?hut_id=" + str(hut_id)


response = requests.get(URL)
#print(response.text)
soup = BeautifulSoup(response.text, 'html.parser')

for i in range(0,13):
    bookingID = "bookingDate" + str(i)
    block = soup.find('div', {'id':bookingID})
    if (block.text == date):
        print("found it once!")
    print(block.text)
    print(block)

#also failed: 
for i in range(0,13):
    bookingID = "bookingDateHidden" + str(i)
    block = soup.find('input', {'id':bookingID})
    if (block.text == date):
        print("found it once!")
    print(block.value)
    print(block)

控制台输出 #

moritz@cupid:~$ python3 Alpines_scraper.py 
<div class="main-label" id="bookingDate0"></div>
<div class="main-label" id="bookingDate1"></div>
<div class="main-label" id="bookingDate2"></div>
<div class="main-label" id="bookingDate3"></div>
<div class="main-label" id="bookingDate4"></div>
<div class="main-label" id="bookingDate5"></div>
<div class="main-label" id="bookingDate6"></div>
<div class="main-label" id="bookingDate7"></div>
<div class="main-label" id="bookingDate8"></div>
<div class="main-label" id="bookingDate9"></div>
<div class="main-label" id="bookingDate10"></div>
<div class="main-label" id="bookingDate11"></div>
<div class="main-label" id="bookingDate12"></div>
None
<input id="bookingDateHidden0" type="hidden"/>
None
<input id="bookingDateHidden1" type="hidden"/>
None
<input id="bookingDateHidden2" type="hidden"/>
...

初步推测与需求 #

我推测页面内容未在请求阶段完成加载，找到相关Stack Overflow帖子但无法理解。这是我首次尝试网页爬取，恳请提供帮助或相关教程指引。

内容的提问来源于stack exchange，提问作者Moritz