Hey there! Let's dig into why you're getting 403/401 errors when trying to access DNS-shop.ru with Selenium. I took a look at your code, and there are a few key anti-bot checks that sites like this enforce—your current setup is missing some critical pieces to pass them.

Common Issues & Fixes #

Here are the most likely reasons you're hitting these errors, along with actionable fixes:

• Incorrect header configuration
  Your code uses AddArgument() to set the Accept header, but Firefox doesn't recognize that format for modifying HTTP headers. Instead, use Firefox's profile preferences to set headers properly. Additionally, sites often require other headers like Accept-Language or Upgrade-Insecure-Requests to validate requests as coming from a real user.

• WebDriver detection
  Selenium-driven browsers leave a clear fingerprint: the window.navigator.webdriver property is set to true, which most anti-bot systems flag immediately. You need to disable this and enable fingerprint resistance to blend in with regular users.

• Lack of human-like behavior
  Automated tools often make requests too quickly, which triggers rate limits or bot detection. Adding small delays and avoiding headless mode (at least initially) can help.

Modified Code Example #

Here's an updated version of your code that addresses these issues:

using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI;
using System;
using SeleniumExtras.WaitHelpers;

class Program
{
    static void Main(string[] args)
    {
        string userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/117.0";
        
        FirefoxOptions options = new FirefoxOptions();
        
        // Set user-agent via profile preference (more reliable than command line)
        options.SetPreference("general.useragent.override", userAgent);
        
        // Disable WebDriver detection flags
        options.SetPreference("dom.webdriver.enabled", false);
        options.SetPreference("privacy.resistFingerprinting", true);
        
        // Configure essential request headers
        FirefoxProfile profile = new FirefoxProfile();
        profile.SetPreference("network.http.accept.default", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8");
        profile.SetPreference("network.http.accept-language", "en-US,en;q=0.5");
        profile.SetPreference("network.http.upgrade-insecure-requests", 1);
        options.Profile = profile;
        
        // Skip headless mode for initial testing (many sites block headless browsers)
        // options.AddArgument("--headless");
        
        IWebDriver driver = new FirefoxDriver(options);
        
        try
        {
            // Add a short delay to mimic human behavior before navigating
            System.Threading.Thread.Sleep(2000);
            
            driver.Navigate().GoToUrl("https://www.dns-shop.ru");
            
            WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(15));
            wait.Until(ExpectedConditions.TitleContains("DNS"));
            
            Console.WriteLine("Page title: " + driver.Title);
        }
        finally
        {
            driver.Quit();
        }
    }
}

Additional Tips #

• Handle anti-bot challenges: If you encounter CAPTCHAs or JS-based checks, you might need tools to bypass these (though be aware of the site's terms of service regarding scraping).
• Mimic user interactions: After loading the page, add small delays, simulate scrolling, or random clicks to make your activity look less automated.
• Check IP reputation: If you're still blocked, your IP might have been flagged. Try using a residential proxy or waiting some time before retrying.

备注：内容来源于stack exchange，提问作者Mark