求助：wget无法访问kenpom.com，曾可爬取现公共内容也无法获取

Hey there! It’s super common for websites to tweak their anti-scraping measures over time, so let’s walk through the most likely fixes for your wget problem:

1. Update Your Outdated wget Version #

Your wget 1.15 is from 2014—way too old to handle modern web security and anti-scraping checks. On OS X 10.13.3, use Homebrew to grab a newer build:

brew update
brew install wget

Once updated, try a basic crawl first:

wget https://kenpom.com/

2. Spoof a Browser User-Agent #

Wget’s default user-agent string is a dead giveaway that you’re using a scraper. Most sites block it outright. Mimic a real browser instead—here’s one tailored to your OS X version:

wget --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" https://kenpom.com/

This tricks the site into thinking you’re browsing with Chrome on your Mac.

3. Load Valid Session Cookies #

Even public content might require a valid session cookie to access (sites use this to block bots that don’t "behave" like real users). Here’s how to get and use them:

• Open kenpom.com in your browser (Chrome/Safari), let it load fully.
• In Chrome, hit F12 → go to the Application tab → expand Cookies → select https://kenpom.com.
• Copy all the cookie entries into a text file (name it cookies.txt) in this format:

  .kenpom.com	TRUE	/	FALSE	1719859200	__cfduid	abc123...
  

• Then run wget with the cookie file:

  wget --load-cookies cookies.txt --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" https://kenpom.com/
  

  Pro tip: Use a browser extension to export cookies directly in wget-compatible format to save time.

4. Check for IP Blocking #

If you scraped the site heavily before, your IP might have been flagged and blocked. Test this by switching to a different network (like your phone’s hotspot). If it works on the new network, you’ll need to wait for the block to lift, or use a proxy with wget:

wget --proxy=http://your-proxy-address:port --user-agent="your-UA-string" https://kenpom.com/

5. Rule Out JavaScript-Rendered Content #

While kenpom’s public content used to be static, they might have switched to dynamic JS loading. Wget can’t execute JavaScript, so if this is the case, you’ll need a tool like Puppeteer (Node.js) or Playwright that can render pages like a real browser. But start with the steps above first—this is less likely to be the issue if you could crawl it before.

Start with updating wget and spoofing the user-agent—those two fixes resolve 90% of these kinds of problems. Good luck!

内容的提问来源于stack exchange，提问作者dwm8