为何requests.get()添加headers参数后谷歌搜索HTML返回结果不同？

Absolutely, using the headers parameter does change the HTML content Google sends back—that’s exactly why you’re seeing such a big difference in the linkedElems count. Let’s break this down:

• When you skip the User-Agent header, the requests library uses its default one (something like python-requests/2.31.0). Google’s servers spot this as an automated, non-browser request. To discourage scrapers, they’ll return a stripped-down or restricted version of the search page—one that doesn’t include the .r a elements you’re targeting. That’s why you end up with 0 results.

• When you add the Firefox User-Agent header, Google treats your request like it’s coming from a real human using a browser. It then serves the full, standard search results page—the exact same one you see when you use Firefox to search manually. This page has all the elements matched by .r a, hence the 44 results you get.

Your check using Firefox’s dev tools confirms this: the page you inspect in the browser is only served when Google recognizes a valid browser User-Agent. Without it, you’re getting an entirely different page structure.

A quick heads-up: Google’s page layout can change over time, so the .r a selector might stop working eventually. But the core rule stays the same: many websites adjust their responses based on the User-Agent header to differentiate between browsers and automated tools.

内容的提问来源于stack exchange，提问作者Redwan Hossain Arnob