Awk脚本与Shell命令执行结果不一致问题求助

Hey there! Let's work through fixing that discrepancy between your Awk script and direct shell command execution. First, let's break down the common pitfalls that cause this kind of issue, then walk through solutions and a refined script example.

Common Issues & Fixes #

1. Shell Command Execution in Awk

When you run shell commands inside Awk, it doesn't inherit the exact same environment (like PATH) as your interactive shell. This can lead to commands like chrome-cli not being found, or behaving differently.

• Use absolute paths: Replace chrome-cli with its full path (e.g., /usr/local/bin/chrome-cli) to avoid PATH mismatches.
• Capture command output correctly: system() only returns exit codes, not command output. To grab page source, use getline with a pipe, and always close the pipe to avoid resource leaks:

  cmd = "chrome-cli open 'https://www.google.com/search?q=" keyword_plus "+'"
  cmd | getline
  close(cmd)
  

2. Page Loading Delay

When you run commands manually, you wait for the page to load—but your Awk script might try to extract data immediately after opening the tab, before Google's results finish rendering.

• Add a sleep timer: Insert a delay after opening the tab to let the page load:

  system("chrome-cli open 'https://www.google.com/search?q=" keyword_plus "+'")
  system("sleep 2") # Adjust time based on your network speed
  

3. Fragile HTML Extraction Logic

Google's search page structure can vary by region, language, or even user session. A simple "About" to "results" string match might fail if the HTML changes slightly.

• Use a more robust regex: Target the numeric count directly with a pattern that accounts for commas in large numbers:

  if (html ~ /About ([0-9,]+) results/) {
      count = gensub(/.*About ([0-9,]+) results.*/, "\\1", "g", html)
      gsub(/,/, "", count) # Remove thousands separators
      print count >> "freq.log"
      close("freq.log")
  }
  

• Consider using curl instead of Chrome: Browsers introduce variables like login states and dynamic loading. Using curl with a desktop user-agent is faster and more consistent:

  cmd = "curl -s -A \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36\" \"https://www.google.com/search?q=" keyword_encoded "\""
  

4. Incomplete Keyword Encoding

Replacing spaces with + isn't enough for special characters (like &, %, or =). These need proper URL encoding to avoid broken search URLs.

• Add a URL encoding function to Awk:

  function urlencode(str, c, len, i) {
      len = length(str)
      result = ""
      for (i = 1; i <= len; i++) {
          c = substr(str, i, 1)
          if (c ~ /[a-zA-Z0-9\/._-]/) {
              result = result c
          } else {
              result = result sprintf("%%%02X", ord(c))
          }
      }
      return result
  }
  function ord(c) { return sprintf("%d", c) }
  

  Use this to encode your keywords before building the search URL.

5. Debugging Tips

To pinpoint where things go wrong, add debug logs to your script:

print "Processing keyword:", keyword >> "debug.log"
print "Executing command:", cmd >> "debug.log"

Copy the logged commands and run them directly in your shell—if they work there but not in Awk, you've found your environment or execution gap.

Refined Example Script #

Here's a full script that incorporates all these fixes, using curl instead of Chrome for reliability:

#!/usr/bin/awk -f

function urlencode(str, c, len, i) {
    len = length(str)
    result = ""
    for (i = 1; i <= len; i++) {
        c = substr(str, i, 1)
        if (c ~ /[a-zA-Z0-9\/._-]/) {
            result = result c
        } else {
            result = result sprintf("%%%02X", ord(c))
        }
    }
    return result
}

function ord(c) { return sprintf("%d", c) }

{
    keyword = $0
    gsub(/\n/, "", keyword)
    keyword_encoded = urlencode(keyword)
    search_url = "https://www.google.com/search?q=" keyword_encoded

    # Fetch page source with curl
    cmd = "curl -s -A \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36\" \"" search_url "\""
    while ((cmd | getline line) > 0) {
        html = html line
    }
    close(cmd)

    # Extract and log result count
    if (html ~ /About ([0-9,]+) results/) {
        count = gensub(/.*About ([0-9,]+) results.*/, "\\1", "g", html)
        gsub(/,/, "", count)
        print keyword "\t" count >> "freq.log"
        close("freq.log")
        print "Processed:", keyword, "| Count:", count
    } else {
        print "Failed to find count for:", keyword >> "error.log"
        close("error.log")
        print "Warning: No results found for", keyword
    }
    html = "" # Reset for next keyword
}

Use it by saving your keywords to a file (e.g., keywords.txt, one per line) and running:

awk -f search_count.awk keywords.txt

内容的提问来源于stack exchange，提问作者vanomart