使用Python 3和Scrapy提取href中what-i-want参数的值

嘿，针对你用Python 3和Scrapy提取what-i-want参数值的需求，我给你整理了几个实用的方案：

方法1：用urllib.parse解析查询参数（最稳妥） #

Scrapy可以结合Python标准库的urllib.parse来解析URL的查询字符串，这种方法不容易出错，还能自动处理URL编码：

import urllib.parse
from scrapy import Spider

class MySpider(Spider):
    name = 'my_spider'
    start_urls = ['your-target-url-here']

    def parse(self, response):
        # 先获取a标签的href属性，注意转义class里的=号
        href = response.css('div[class="class=a1"] span.a-small a.a-nm::attr(href)').get()
        if href:
            # 把相对URL转成绝对URL（可选，但推荐处理完整URL场景）
            absolute_url = response.urljoin(href)
            # 解析URL的查询参数部分
            query_part = urllib.parse.urlparse(absolute_url).query
            query_params = urllib.parse.parse_qs(query_part)
            # 获取what-i-want的值，parse_qs返回列表，取第一个元素
            what_i_want = query_params.get('what-i-want', [''])[0]
            # 把URL编码的+转成空格，也可以用unquote处理更复杂的编码
            what_i_want_decoded = urllib.parse.unquote(what_i_want)
            print(what_i_want_decoded)  # 输出: Nice Home

方法2：用XPath结合正则表达式直接提取 #

如果想一步到位，也可以在XPath选择器里用正则匹配目标内容：

from scrapy import Spider

class MySpider(Spider):
    name = 'my_spider'
    start_urls = ['your-target-url-here']

    def parse(self, response):
        # 用XPath的regex函数精准匹配what-i-want后的内容
        what_i_want = response.xpath('//div[@class="class=a1"]/span[@class="a-small"]/a[@class="a-nm"]/@href'
                                     '/regex("what-i-want=([^&]+)")').get()
        if what_i_want:
            # 解码URL编码字符
            what_i_want_decoded = urllib.parse.unquote(what_i_want)
            print(what_i_want_decoded)  # 输出: Nice Home

方法3：用CSS选择器结合Python正则处理 #

先通过CSS获取href属性，再用Python正则提取目标片段：

import re
import urllib.parse
from scrapy import Spider

class MySpider(Spider):
    name = 'my_spider'
    start_urls = ['your-target-url-here']

    def parse(self, response):
        href = response.css('div[class="class=a1"] span.a-small a.a-nm::attr(href)').get()
        if href:
            # 用正则匹配what-i-want=和&之间的内容
            match = re.search(r'what-i-want=([^&]+)', href)
            if match:
                what_i_want = match.group(1)
                what_i_want_decoded = urllib.parse.unquote(what_i_want)
                print(what_i_want_decoded)  # 输出: Nice Home

小提示 #

• 你的HTML中div的class是class=a1，用div[class="class=a1"]比转义=的写法更直观，避免选择器出错
• 如果遇到%20这类URL编码，urllib.parse.unquote()能自动转成空格，比手动替换+更通用

内容的提问来源于stack exchange，提问作者TJ1