Scrapy多类名div元素爬取失败问题求助

Hey there! As a fellow Scrapy user, I spot this super common selector mistake right away—let's get your parse method working properly and fix that empty output issue.

What Went Wrong #

Your CSS selector div.col-xs-12 available-columns inner-available trans-fade-in uses spaces between class names, which tells Scrapy to look for a nested structure:

• A div with class col-xs-12
• Inside that div, an element with class available-columns
• Inside that element, another with class inner-available, and so on

But what you actually need is a single div that has all those classes at the same time. For this case, you have to connect each class name with a dot (.), no spaces allowed.

Corrected Parse Method #

Here's the fixed code for your parse function:

def parse(self, response):
    # Use dots to chain multiple classes on the same div element
    for flight in response.css('div.col-xs-12.available-columns.inner-available.trans-fade-in'):
        yield {
            'price': flight.css('span.w-bold::text').extract_first(),
        }

Quick Bonus Tips #

• If some of those classes are dynamic (like trans-fade-in might be a temporary animation class that doesn't always load), you can simplify the selector to use only the stable, required classes. For example:

  response.css('div.col-xs-12.available-columns.inner-available')
  

• If you prefer using XPath instead, the equivalent selector would be:

  response.xpath('//div[contains(@class, "col-xs-12") and contains(@class, "available-columns") and contains(@class, "inner-available") and contains(@class, "trans-fade-in")]')
  

  But CSS selectors are usually cleaner for this scenario.

内容的提问来源于stack exchange，提问作者niloofar