如何用Selenium和Python单独提取div内的每个元素？

核心思路 #

目标数据都包裹在div.message-userExtras容器内，每个dl.pairs.pairs--justified是一组键值对（<dt>为键，<dd>为值），无需唯一ID，通过遍历分组标签即可提取所有信息。

实现方式1：使用Python BeautifulSoup #

from bs4 import BeautifulSoup

html = '''
<div class="message-userExtras">
            
    <dl class="pairs pairs--justified">
    <dt>Joined</dt>
    <dd>Feb 23, 2023</dd>
    </dl>


    <dl class="pairs pairs--justified">
    <dt>Messages</dt>
    <dd>22</dd>
    </dl>

    <dl class="pairs pairs--justified">
    <dt>Reaction score</dt>
    <dd>
    <a href="/members/297754/reactions-info" class="link">18</a>
    </dd>
    </dl>
                    
                                        
    <dl class="pairs pairs--justified">
    <dt>Escrow deals</dt>
    <dd>3</dd>
    </dl>
                    

    <dl class="pairs pairs--justified">
    <dt>Deposit</dt>
    <dd><a class="deposit-button" href="/members/297754/deposit" data-xf-click="overlay">0.1 <i class="fa--xf fab fa-btc deposit-fa-icon" aria-hidden="true"></i></a></dd>
    </dl>

</div>
'''

soup = BeautifulSoup(html, 'html.parser')
user_extras = soup.find('div', class_='message-userExtras')
result = {}

for dl in user_extras.find_all('dl', class_='pairs pairs--justified'):
    # 提取键并转为小写
    key = dl.find('dt').get_text(strip=True).lower()
    # 提取值，处理Deposit中的图标文本，只保留有效内容
    value = dl.find('dd').get_text(strip=True).split()[0]
    result[key] = [value]

print(result)

执行后输出：

{'joined': ['Feb 23, 2023'], 'messages': ['22'], 'reaction score': ['18'], 'escrow deals': ['3'], 'deposit': ['0.1']}

实现方式2：使用XPath（lxml/Scrapy） #

from lxml import etree

html = '''
<div class="message-userExtras">
            
    <dl class="pairs pairs--justified">
    <dt>Joined</dt>
    <dd>Feb 23, 2023</dd>
    </dl>


    <dl class="pairs pairs--justified">
    <dt>Messages</dt>
    <dd>22</dd>
    </dl>

    <dl class="pairs pairs--justified">
    <dt>Reaction score</dt>
    <dd>
    <a href="/members/297754/reactions-info" class="link">18</a>
    </dd>
    </dl>
                    
                                        
    <dl class="pairs pairs--justified">
    <dt>Escrow deals</dt>
    <dd>3</dd>
    </dl>
                    

    <dl class="pairs pairs--justified">
    <dt>Deposit</dt>
    <dd><a class="deposit-button" href="/members/297754/deposit" data-xf-click="overlay">0.1 <i class="fa--xf fab fa-btc deposit-fa-icon" aria-hidden="true"></i></a></dd>
    </dl>

</div>
'''

tree = etree.HTML(html)
dl_list = tree.xpath('//div[@class="message-userExtras"]/dl[@class="pairs pairs--justified"]')
result = {}

for dl in dl_list:
    key = dl.xpath('./dt/text()')[0].strip().lower()
    # 提取dd下所有文本并处理
    value_text = ' '.join(dl.xpath('./dd//text()')).strip()
    value = value_text.split()[0]
    result[key] = [value]

print(result)

执行后输出与上面一致。

内容的提问来源于stack exchange，提问作者BGreeneGiant