使用requests获取谷歌搜索页源码与浏览器不一致的问题及解决
2026-8-5
问题描述
作为网页爬虫新手，刚接触Python的requests模块与selenium工具，我用requests尝试获取谷歌搜索关键词「Cats」的页面源码，代码如下：
import requests #The keyword I want to search Google For search_keyword = "Cats" #Opening the URL using requests page = requests.get(r"https://www.google.com/search?q"+search_keyword) #Break program if there is an error in opening previous link page.raise_for_status() #obtain the source code for the Google Webpage as a String source_code = page.text
但发现source_code中的源码和Chrome浏览器Ctrl+U查看的源码差异极大，以下是两边源码的最后7行：
Chrome浏览器中查看的源码最后7行
}catch(e){_._DumpException(e)} try{ var uj=function(a,b,c){_.Kd.log(46,{att:a,max:b,url:c})},wj=function(a,b,c){_.Kd.log(47,{att:a,max:b,url:c});a<b?vj(a+1,b):_.J.log(Error("aa`"+a+"`"+b),{url:c})},vj=function(a,b){if(xj){const d=_.ke("SCRIPT");d.async=!0;d.type="text/javascript";d.charset="UTF-8";var c=d;c.src=_.Ac(xj);_.tj(c);d.onload=_.be(uj,a,b,d.src);d.onerror=_.be(wj,a,b,d.src);_.Kd.log(45,{att:a,max:b,url:d.src});_.ee("HEAD")[0].appendChild(d)}},yj=class extends _.H{constructor(a){super(a)}},zj=_.F(_.Fd,yj,17)||new yj,Aj,xj= (Aj=_.F(zj,_.ic,1))?_.qj(_.C(Aj,4)||""):null,Bj,Cj=(Bj=_.F(zj,_.ic,2))?_.qj(_.C(Bj,4)||""):null,Dj=function(){vj(1,2);if(Cj){const a=_.ke("LINK");a.setAttribute("type","text/css");_.pj(a,Cj,"stylesheet");let b=_.$c();b&&a.setAttribute("nonce",b);_.ee("HEAD")[0].appendChild(a)}};(function(){const a=_.Gd();if(_.E(a,18))Dj();else{const b=_.C(a,19)||0;window.addEventListener("load",()=>{window.setTimeout(Dj,b)})}})(); }catch(e){_._DumpException(e)} })(this.gbar_); // Google Inc. ;});})();(function(){google.drty&&google.drty(undefined,true);})();});</script><div></div><div jscontroller="MTV2Lb" style="display:none" src="/uviewer?q=cats&origin=https%3A%2F%2Fwww.google.com" id="Rvx4kc" jsaction="rcuQ6b:npT2md;u0pjoe:Hq0NGf"></div><div jscontroller="W0N1pf" id="DDeXhf" jsaction="u0pjoe:Hq0NGf"></div><div id="lfootercc"><script nonce="aZ5SspHCLpIp5NbwBB_sog">(function(){for(var i in google.iir||{}){_setImagesSrc([i],google.iir[i]);}google.iir={};})();(function(){var _jnu='https://www.google.com/ads/measurement/t';var _jnuid=2500;google.x({'id':'jnu'},function(){var bt = google.sx && google.sx.setTimeout || window.setTimeout;bt(function(){google.log('','',_jnu);},_jnuid);});})();google.jslm=3;</script><div id="reviewDialog" data-async-context="async_id_prefix:" data-jiis="up" data-async-type="reviewDialog" data-async-context-required="async_id_prefix" class="yp"></div><div id="dbg_"></div></div></body></html>
requests获取的源码最后7行
else top.location='/doodles/';};})();</script><input value="AK50M_UAAAAAY717njRHtGFQobKw12k0KwtaX02a7bb3" name="iflsig" type="hidden"></span></span></td><td class="fl sblc" align="left" nowrap="" width="25%"><a href="/advanced_search?hl=en-PK&authuser=0">Advanced search</a></td></tr></table><input id="gbv" name="gbv" type="hidden" value="1"><script nonce="t1EKgWOJQagENoMoyIq8zw">(function(){var a,b="1";if(document&&document.getElementById)if("undefined"!=typeof XMLHttpRequest)b="2";else if("undefined"!=typeof ActiveXObject){var c,d,e=["MSXML2.XMLHTTP.6.0","MSXML2.XMLHTTP.3.0","MSXML2.XMLHTTP","Microsoft.XMLHTTP"];for(c=0;d=e[c++];)try{new ActiveXObject(d),b="2"}catch(h){}}a=b;if("2"==a&&-1==location.search.indexOf("&gbv=2")){var f=google.gbvu,g=document.getElementById("gbv");g&&(g.value=a);f&&window.setTimeout(function(){location.href=f},0)};}).call(this);</script></form><div id="gac_scont"></div><div style="font-size:83%;min-height:3.5em"><br><div id="gws-output-pages-elements-homepage_additional_languages__als"><style>#gws-output-pages-elements-homepage_additional_languages__als{font-size:small;margin-bottom:24px}#SIvCob{color:#3c4043;display:inline-block;line-height:28px;}#SIvCob a{padding:0 3px;}.H6sW5{display:inline-block;margin:0 2px;white-space:nowrap}.z4hgWe{display:inline-block;margin:0 2px}</style><div id="SIvCob">Google offered in: <a dir="rtl" href="https://www.google.com/setprefs?sig=0_r1jPgArqJ4B4gDPx-dpt_P_fF1c%3D&hl=ur&source=homepage&sa=X&ved=0ahUKEwi7w86Hk738AhVwkZUCHQZ8Bm4Q2ZgBCAU">اردو</a> <a dir="rtl" href="https://www.google.com/setprefs?sig=0_r1jPgArqJ4B4gDPx-dpt_P_fF1c%3D&hl=ps&source=homepage&sa=X&ved=0ahUKEwi7w86Hk738AhVwkZUCHQZ8Bm4Q2ZgBCAY">پښتو</a> <a dir="rtl" href="https://www.google.com/setprefs?sig=0_r1jPgArqJ4B4gDPx-dpt_P_fF1c%3D&hl=sd&source=homepage&sa=X&ved=0ahUKEwi7w86Hk738AhVwkZUCHQZ8Bm4Q2ZgBCAc">سنڌي</a> </div></div></div><span id="footer"><div style="font-size:10pt"><div style="margin:19px auto;text-align:center" id="WqQANb"><a href="/intl/en/ads/">Advertising�Programs</a><a href="/intl/en/about.html">About Google</a><a href="https://www.google.com/setprefdomain?prefdom=PK&prev=https://www.google.com.pk/&sig=K_MVkcQYGVALpK2CEzShaNNS889C4%3D">Google.com.pk</a></div></div><p style="font-size:8pt;color:#70757a">© 2023 - <a href="/intl/en/policies/privacy/">Privacy</a> - <a href="/intl/en/policies/terms/">Terms</a></p></span></center><script nonce="t1EKgWOJQagENoMoyIq8zw">(function(){window.google.cdo={height:757,width:1440};(function(){var a=window.innerWidth,b=window.innerHeight;if(!a||!b){var c=window.document,d="CSS1Compat"==c.compatMode?c.documentElement:c.body;a=d.clientWidth;b=d.clientHeight}a&&b&&(a!=google.cdo.width||b!=google.cdo.height)&&google.log("","","/client_204?&atyp=i&biw="+a+"&bih="+b+"&ei="+google.kEI);}).call(this);})();</script> <script nonce="t1EKgWOJQagENoMoyIq8zw">(function(){google.xjs={ck:'xjs.hp.L0TU2uVtv08.L.X.O',cs:'ACT90oHT93DEQlP6_b1H10Ysv5zFk8Lfjw',excm:[]};})();</script> <script nonce="t1EKgWOJQagENoMoyIq8zw">(function(){var u='/xjs/_/js/k\x3dxjs.hp.en.O9Ix1H-6nJI.O/am\x3dAADoBABQAGAB/d\x3d1/ed\x3d1/rs\x3dACT90oE7kMfbiyrYlGzQlW0b_TuKFV-B6g/m\x3dsb_he,d';var amd=0; var d=this||self,e=function(a){return a};var g;var l=function(a,b){this.g=b===h?a:""};l.prototype.toString=function(){return this.g+""};var h={}; function m(){var a=u;google.lx=function(){p(a);google.lx=function(){}};google.bx||google.lx()} function p(a){google.timers&&google.timers.load&&google.tick&&google.tick("load","xjsls");var b=document;var c="SCRIPT";"application/xhtml+xml"===b.contentType&&(c=c.toLowerCase());c=b.createElement(c);a=null===a?"null":void 0===a?"undefined":a;if(void 0===g){b=null;var k=d.trustedTypes;if(k&&k.createPolicy){try{b=k.createPolicy("goog#html",{createHTML:e,createScript:e,createScriptURL:e})}catch(q){d.console&&d.console.error(q.message)}g=b}else g=b}a=(b=g)?b.createScriptURL(a):a;a=new l(a,h);c.src= a instanceof l&&a.constructor===l?a.g:"type_error:TrustedResourceUrl";var f,n;(f=(a=null==(n=(f=(c.ownerDocument&&c.ownerDocument.defaultView||window).document).querySelector)?void 0:n.call(f,"script[nonce]"))?a.nonce||a.getAttribute("nonce")||"":"")&&c.setAttribute("nonce",f);document.body.appendChild(c);google.psa=!0};google.xjsu=u;setTimeout(function(){0<amd?google.caft(function(){return m()},amd):m()},0);})();function _DumpException(e){throw e;} function _F_installCss(c){} (function(){google.jl={blt:'none',chnk:0,dw:false,dwu:true,emtn:0,end:0,ico:false,ikb:0,ine:false,injs:'none',injt:0,injth:0,injv2:false,lls:'default',pdt:0,rep:0,snet:true,strt:0,ubm:false,uwp:true};})();(function(){var pmc='{\x22d\x22:{},\x22sb_he\x22:{\x22agen\x22:true,\x22cgen\x22:true,\x22client\x22:\x22heirloom-hp\x22,\x22dh\x22:true,\x22ds\x22:\x22\x22,\x22fl\x22:true,\x22host\x22:\x22google.com\x22,\x22jsonp\x22:true,\x22msgs\x22:{\x22cibl\x22:\x22Clear Search\x22,\x22dym\x22:\x22Did you mean:\x22,\x22lcky\x22:\x22I\\u0026#39;m Feeling Lucky\x22,\x22lml\x22:\x22Learn more\x22,\x22psrc\x22:\x22This search was removed from your \\u003Ca href\x3d\\\x22/history\\\x22\\u003EWeb History\\u003C/a\\u003E\x22,\x22psrl\x22:\x22Remove\x22,\x22sbit\x22:\x22Search by image\x22,\x22srch\x22:\x22Google Search\x22},\x22ovr\x22:{},\x22pq\x22:\x22\x22,\x22rfs\x22:[],\x22sbas\x22:\x220 3px 8px 0 rgba(0,0,0,0.2),0 0 0 1px rgba(0,0,0,0.08)\x22,\x22stok\x22:\x22xGs-7Ks6Q8q3RMls1jXZ4MoFStc\x22}}';google.pmc=JSON.parse(pmc);})();</script> </body></html>
问题：
- 出现该差异的原因是什么？
- 谷歌是否采取了反爬措施？
- 如何在Python中获取该页面的真实源码？
解答
一、差异原因与反爬确认
是的，谷歌明确采取了反爬措施，同时结合页面渲染机制，导致你拿到的源码不一致：
- 反爬识别：你的requests请求未携带浏览器的
User-Agent等关键请求头，谷歌服务器判定这是自动化程序发起的请求，返回的是简化版防爬页面，而非真实搜索结果页。 - 动态渲染差异：谷歌搜索页面大量内容通过JavaScript动态加载、渲染，requests只能获取服务器返回的初始HTML；而浏览器会执行页面JS，加载后续资源并生成完整页面，所以Chrome中看到的是渲染后的最终源码，和requests拿到的初始HTML完全不同。
- URL拼接错误：你的代码里URL拼接存在问题，
"https://www.google.com/search?q"+search_keyword应改为"https://www.google.com/search?q=" + search_keyword（q后缺少等号），这也导致请求的不是正确的搜索结果页。
二、获取真实源码的方法
方法1：给requests添加请求头，模拟浏览器请求
通过添加User-Agent等请求头，让谷歌服务器认为这是浏览器发起的请求，同时修正URL拼接错误：
import requests search_keyword = "Cats" # 模拟Chrome浏览器请求头，可替换为你自己浏览器的User-Agent headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" } # 修正URL拼接，q后添加等号 url = f"https://www.google.com/search?q={search_keyword}" page = requests.get(url, headers=headers) page.raise_for_status() source_code = page.text
注意：这种方法仅能获取服务器返回的初始HTML，无法拿到JS渲染后的内容，若需完整页面源码，建议使用Selenium。
方法2：使用Selenium模拟真实浏览器
Selenium会启动真实浏览器，执行页面JS，完全模拟用户操作，能获取到和Chrome Ctrl+U一致的完整源码：
from selenium import webdriver from selenium.webdriver.chrome.options import Options search_keyword = "Cats" url = f"https://www.google.com/search?q={search_keyword}" # 配置Chrome选项，可按需调整 chrome_options = Options() # 可选：无头模式，不显示浏览器窗口 # chrome_options.add_argument("--headless=new") # 添加User-Agent（Selenium默认也会携带，可省略） chrome_options.add_argument("User-Agent=M
相关产品推荐
相关产品推荐
