为何部分网站能检测到我的Perl脚本请求非浏览器发起？

作为编程练习，我尝试用Perl编写一个简易网页浏览器，使用LWP::UserAgent获取页面内容时，遇到了被Cloudflare防护的网站返回403 Forbidden的问题。

最初的代码实现（返回403） #

my $host   = "www.somehost.com";
my $scheme = "https";
my $pageContent = "";

my $browserObj = LWP::UserAgent->new();
$browserObj->cookie_jar( {} );
$browserObj->timeout(600);
push @{ $browserObj->requests_redirectable }, 'POST';

$browserObj->add_handler("request_send",  sub { shift->dump; return });

$response = $browserObj->get( "$scheme://$host" );
if( $response->is_success ) {
  $pageContent = $response->decoded_content();
} else {
  print "Unable to retrieve $host.\nError: " . $response->status_line;
}

尝试复制Firefox请求头（仍返回403） #

对比后发现请求头和Firefox发送的差异很大，于是我复制了Firefox的请求头修改代码，但依旧得到403 Forbidden：

my $host = "www.somehost.com";
my $scheme = "https";
my $pageContent = "";

my $browserObj = LWP::UserAgent->new();
$browserObj->cookie_jar( {} );
$browserObj->timeout(600);
push @{ $browserObj->requests_redirectable }, 'POST';

$browserObj->add_handler("request_send",  sub { shift->dump; return });

# 发送和Firefox一致的请求头
my @header = (
           'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0',
           'Host' => $host,
           'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
           'Accept-Encoding' => 'gzip, deflate, br',
           'Accept-Language' => 'en-US,en;q=0.5',
           'Connection' => 'keep-alive',
           'DNT' => '1',
           'Sec-Fetch-Dest' => 'document',
           'Sec-Fetch-Mode' => 'navigate',
           'Sec-Fetch-Site' => 'none',
           'Sec-Fetch-User' => '?1',
           'Sec-GPC' => '1',
           'Upgrade-Insecure-Requests' => '1',
         );

$response = $browserObj->get( "$scheme://$host", @header );
if( $response->is_success ) {
  $pageContent = $response->decoded_content();
} else {
  print "Unable to retrieve $host.\nError: " . $response->status_line;
}

手动构造请求头（仍返回403） #

此时我发现唯一差异是请求头顺序，而LWP::UserAgent无法设置头顺序，于是改用IO::Socket::SSL手动构造请求，确保请求头的内容和顺序完全匹配Firefox的原始请求，但还是返回403 Forbidden。

我确认这不是Cookie的问题——用Firefox隐私模式（无Cookie）访问该网站可以正常加载页面。

use strict;
use IO::Socket::SSL;

my $host   = "www.somedomain.com";
my $port   = 443;
my $sock = IO::Socket::SSL->new("$host:$port") || die $!;

print $sock "GET / HTTP/1.1\r\n";
print $sock "Host: $host\r\n";
print $sock "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0\r\n";
print $sock "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\r\n";
print $sock "Accept-Language: en-US,en;q=0.5\r\n";
print $sock "Accept-Encoding: gzip, deflate, br\r\n";
print $sock "DNT: 1\r\n";
print $sock "Sec-GPC: 1\r\n";
print $sock "Connection: keep-alive\r\n";
print $sock "Upgrade-Insecure-Requests: 1\r\n";
print $sock "Sec-Fetch-Dest: document\r\n";
print $sock "Sec-Fetch-Mode: navigate\r\n";
print $sock "Sec-Fetch-Site: none\r\n";
print $sock "Sec-Fetch-User: ?1\r\n\r\n";

print while <$sock>;

close $sock;

请问这类网站是如何检测出我的请求来自脚本而非浏览器的？

内容的提问来源于stack exchange，提问作者Zippy1970