PHP定时访问网页需执行JavaScript，求替代CURL的可行方案

Alright, let's tackle this—since cURL can't run JavaScript and you need your bot to act just like a real human browsing a site (executing all JS, rendering the page properly), here are the top PHP-based solutions you can use right now:

1. PHP Puppeteer #

This is a PHP wrapper for Google's Puppeteer, which controls a headless Chrome/Chromium browser. It's perfect for simulating real user behavior, executing all JavaScript on the page, and even interacting with elements (clicks, form fills, etc.).

Setup #

First, install the package via Composer:

composer require nesk/puppeteer

You'll also need to have Chrome or Chromium installed on your server.

Example Code #

<?php
require 'vendor/autoload.php';

use Nesk\Puphpeteer\Puppeteer;

$puppeteer = new Puppeteer;
$browser = $puppeteer->launch([
    'headless' => true, // Set to false if you want to see the browser window
    'args' => ['--no-sandbox', '--disable-setuid-sandbox'], // Required for some server environments
]);

$page = $browser->newPage();
// Set a realistic user agent to avoid being blocked
$page->setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36');

// Navigate to your target site
$page->goto('https://your-target-site.com', [
    'waitUntil' => 'networkidle2', // Wait until most JS has finished loading
]);

// Optional: Wait for a specific element to ensure JS has rendered content
$page->waitForSelector('.some-js-rendered-element');

// Get the fully rendered HTML (including JS-generated content)
$html = $page->content();

// Do something with the HTML (parse, extract data, etc.)
echo $html;

$browser->close();
?>

2. Symfony Panther #

Built on top of ChromeDriver and Symfony's components, Panther is designed for web scraping and testing, but works great for bots too. It supports JavaScript execution, and makes it easy to simulate user interactions.

Setup #

Install via Composer:

composer require symfony/panther

The package will automatically install ChromeDriver if needed.

Example Code #

<?php
require 'vendor/autoload.php';

use Symfony\Component\Panther\Client;

// Launch a headless Chrome browser
$client = Client::createChromeClient();

// Navigate to the target site, wait for JS to load
$crawler = $client->request('GET', 'https://your-target-site.com');

// Wait for a JS-rendered element to appear
$client->waitFor('.some-js-element');

// Extract data from the rendered page
$text = $crawler->filter('.some-js-element')->text();
echo $text;

// You can also simulate clicks, form submissions, etc.
// $client->click($crawler->selectLink('Click Me')->link());

$client->quit();
?>

3. Goutte + PhantomJS (Legacy Option) #

If you need a lighter-weight option (though PhantomJS is no longer actively maintained), you can use Goutte (a PHP scraping library) with PhantomJS as a headless browser to execute JavaScript.

Setup #

Install Goutte via Composer:

composer require fabpot/goutte

You'll need to download PhantomJS and make it executable on your server.

Example Code #

<?php
require 'vendor/autoload.php';

use Goutte\Client;

$client = new Client();
// Configure Goutte to use PhantomJS
$client->setHeader('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36');
$client->setClient(new \GuzzleHttp\Client([
    'timeout' => 60,
    'headers' => ['User-Agent' => 'Mozilla/5.0...'],
    'handler' => \GuzzleHttp\HandlerStack::create(new \GuzzleHttp\Handler\CurlHandler()),
]));

// Use PhantomJS to fetch the rendered page
$phantomJsPath = '/path/to/phantomjs';
$url = 'https://your-target-site.com';
$command = "$phantomJsPath --ssl-protocol=any --ignore-ssl-errors=true evaluate.js $url";
$html = shell_exec($command);

// Now you can parse $html with Goutte or another parser
$crawler = new \Symfony\Component\DomCrawler\Crawler($html);
echo $crawler->filter('body')->text();
?>

Note: You'll need a simple evaluate.js script to tell PhantomJS to render the page and output the HTML.

To run your bot at regular intervals, use your system's task scheduler:

• Linux/macOS: Use crontab. Edit your crontab with crontab -e, then add a line like:

  # Run every hour
  0 * * * * php /path/to/your/bot-script.php >> /var/log/bot.log 2>&1
  

• Windows: Use Task Scheduler to create a basic task that runs php.exe with your script's path as an argument.

• Always use a realistic user-agent string to avoid being flagged as a bot.
• Add wait conditions (like waitForSelector in Puppeteer/Panther) to ensure JavaScript has finished rendering content before you extract data.
• Respect the target site's robots.txt and terms of service to avoid legal issues or getting blocked.

内容的提问来源于stack exchange，提问作者Bruno Andrade