Scrapy新手求助：从指定剧集脚本网站提取字幕的代码问题

Let's complete and refine your spider to properly scrape episode scripts from the target site. Here's a working, newbie-friendly example that follows Scrapy best practices:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from scrapy.item import Item, Field

# Define a container to store our scraped data
class ScriptItem(Item):
    show_name = Field()
    episode_title = Field()
    script_content = Field()
    episode_url = Field()

class BojackSpider(CrawlSpider):
    name = "bojack_scripts"
    allowed_domains = ["springfieldspringfield.co.uk"]
    start_urls = ["https://www.springfieldspringfield.co.uk/episode_scripts.php?tv-show=bojack-horseman-2014"]
    
    # Rule to follow links to individual episode pages
    rules = (
        Rule(
            LinkExtractor(
                allow=r'/episode_script.php\?tv-show=bojack-horseman-2014&episode=.+'
            ),
            callback='parse_episode',
            follow=False
        ),
    )

    def parse_episode(self, response):
        # Initialize our data container
        item = ScriptItem()
        
        # Set show name (can also scrape this from the page if you prefer)
        item['show_name'] = "Bojack Horseman (2014)"
        
        # Extract episode title from the page header
        item['episode_title'] = response.xpath('//h3[@class="episode-title"]/text()').get().strip()
        
        # Extract full script content from the scrolling container
        script_lines = response.xpath('//div[@class="scrolling-script-container"]//p/text()').getall()
        item['script_content'] = "\n".join(script_lines)
        
        # Store the episode URL for reference
        item['episode_url'] = response.url
        
        return item

Key Breakdown for Newbies: #

• Item Class: Think of this as a labeled folder for your data—we define exactly what pieces of info we want to collect (show name, episode title, etc.).

• Spider Basics:

  • name: A unique ID for your spider (you'll use this when running the command scrapy crawl bojack_scripts).
  • allowed_domains: Prevents the spider from wandering to unrelated websites, keeping it focused on the target site.
  • start_urls: The starting page where your crawl begins (the main Bojack Horseman episode list).

• Rule & Link Extraction:

  • The LinkExtractor uses a regex pattern to find all links pointing to individual episode scripts. The pattern matches any URL that follows the site's episode script format.
  • callback='parse_episode': Tells Scrapy to run our custom function on every episode page it finds.
  • follow=False: We don't need to crawl more links from episode pages, so we turn this off to save resources.

• Parsing Episode Pages:

  • response.xpath(): A tool to target specific elements on the page using their HTML structure. Use your browser's DevTools (right-click → Inspect) to find these selectors if the site ever updates its layout.
  • For the script: We grab all text from <p> tags inside the script container and join them with newlines to keep the script formatting intact.

Quick Troubleshooting Tips: #

• Selector Issues: If the script isn't pulling through, the site might have changed its HTML classes. Recheck the element with DevTools and update the XPath selectors accordingly.
• Avoid Getting Blocked: Add a delay between requests in your settings.py to be polite:

  DOWNLOAD_DELAY = 2
  

• Save Your Results: Run the spider and export data to a JSON file with this command:

  scrapy crawl bojack_scripts -o bojack_scripts.json
  

Note on Deprecated Code: #

You imported HtmlXPathSelector, which is outdated in newer Scrapy versions. The example uses response.xpath() directly—this is the modern, supported way to extract data.

内容的提问来源于stack exchange，提问作者MrRobot9