使用UrlFetchApp实现带登录验证的网站数据抓取时遭遇Cannot POST /错误求助

It sounds like you're hitting a common snag when trying to authenticate a website with Google Apps Script's UrlFetchApp: the server is rejecting your POST request to the root path (/). Let’s break down the most likely causes and fixes step by step.

1. Use the Correct Login Endpoint URL #

The biggest reason for the Cannot POST / error is that you’re sending your login request to the website’s homepage (/), but actual login forms always submit to a specific endpoint (not the root).

To find the right URL:

• Open the target website’s login page in your browser
• Press F12 to open Developer Tools, switch to the Network tab
• Submit the login form (you can use fake credentials for this test)
• Look for the first POST request in the network log—its Request URL is the correct adresse value you should use (e.g., https://example.com/login, https://example.com/api/auth/signin)

2. Add Required Request Headers #

Many websites validate request headers to ensure the request comes from a legitimate client. Two key headers to include are:

• Content-Type: For form data, this is almost always application/x-www-form-urlencoded
• User-Agent: Mimics a browser to avoid being flagged as a script

Update your options object like this:

var options = {
  "method" : "post",
  "muteHttpExceptions" : true,
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
  },
  "payload":{"login[email]" : email, "login[password]" : password }
};

3. Include Hidden Form Fields (Like CSRF Tokens) #

Most modern login forms include hidden security fields (e.g., CSRF tokens) that must be included in your payload to pass validation. To get these values:

1. First fetch the login page HTML
2. Parse the HTML to extract the hidden field values
3. Add them to your payload

Example code to extract a CSRF token:

// First, fetch the login page to get the CSRF token
var loginPageUrl = "https://example.com/login"; // Replace with actual login page URL
var loginPageResponse = UrlFetchApp.fetch(loginPageUrl, {"muteHttpExceptions": true});
var loginHtml = loginPageResponse.getContentText();

// Extract CSRF token using regex (adjust the pattern to match your target site's field)
var csrfMatch = loginHtml.match(/<input type="hidden" name="authenticity_token" value="([^"]+)"/);
var csrfToken = csrfMatch ? csrfMatch[1] : "";

// Update payload with the CSRF token
var payload = {
  "login[email]": email,
  "login[password]": password,
  "authenticity_token": csrfToken // Match the field name from the login form
};

// Now send the login request with the updated payload
var options = {
  "method" : "post",
  "muteHttpExceptions" : true,
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
  },
  "payload": payload
};
var response = UrlFetchApp.fetch(loginEndpointUrl, options); // Use the correct login endpoint here

4. Check Redirect Handling #

Some websites redirect after a successful login. UrlFetchApp follows redirects by default, but if you need to capture session cookies from the initial response, you can disable redirects with followRedirects: false and handle them manually.

Start with fixing the login endpoint first—that’s the most likely fix for your Cannot POST / error. If that doesn’t work, add the headers and check for hidden fields.

内容的提问来源于stack exchange，提问作者Patpower