使用Python Selenium爬取Godaddy网站失败的问题求助

GoDaddy actively blocks automated tools like Selenium by detecting browser attributes that reveal automation. Here are actionable fixes to bypass this:

Basic Fixes with Regular Selenium #

Update your ChromeOptions to mimic a real user's browser more closely:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

chrome_options = webdriver.ChromeOptions()

# Disable core automation detection flags
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)

# Set a valid user-agent (match your actual browser's UA for best results)
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36")

# Disable extensions and infobars to reduce detection cues
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-infobars")

# Start browser maximized to avoid mobile site triggers
chrome_options.add_argument("--start-maximized")

# Optional fixes for Linux environments
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")

driver_path = 'drive_path'
service = Service(driver_path)
driver = webdriver.Chrome(service=service, options=chrome_options)

# Override navigator.webdriver property to hide automation
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

url = 'https://in.godaddy.com/domainsearch/find?checkAvail=1&domainToCheck=bjmtuc.club'
driver.get(url)

Key Notes: #

• The --disable-blink-features=AutomationControlled flag is critical—it prevents Chrome from exposing that it’s controlled by automation.
• Your original javascript.enabled argument was unnecessary; JavaScript is enabled by default in Chrome, so remove that line.
• The navigator.webdriver script further hides automation by overriding a property sites commonly check.

Advanced Fix: Use Undetected Chromedriver #

If regular Selenium still gets blocked, use undetected-chromedriver—a modified ChromeDriver built to bypass anti-bot systems:

1. Install the package first:

pip install undetected-chromedriver

2. Implement it in your code:

import undetected_chromedriver as uc

# The driver handles most anti-bot checks automatically
driver = uc.Chrome()

url = 'https://in.godaddy.com/domainsearch/find?checkAvail=1&domainToCheck=bjmtuc.club'
driver.get(url)

Additional Tips: #

• Add small delays (time.sleep(2) between actions) to mimic human browsing speed.
• Use residential proxies instead of datacenter proxies if you need to scale requests.
• Save and reuse cookies from a manually logged-in session to reduce login-related detection.

内容的提问来源于stack exchange，提问作者Khush