curl请求返回403 Forbidden错误但wget可正常下载的求助

问题背景 #

我有一个第三方提供的大型应用安装构建脚本，最近执行到以下命令时失败：

curl -f http://www.mjr19.org.uk/sw/inode64.c

该命令返回403 Forbidden错误，但使用wget执行相同文件的下载却能成功：

wget http://www.mjr19.org.uk/sw/inode64.c

我使用的是Ubuntu 22.04系统，curl版本试过7.81和8.1.2，两台不同电脑上都出现相同现象。我不想修改这个不属于我的脚本（它之前能正常运行），想请教有没有人遇到过类似情况，以及其他可尝试的解决方法？另外网上部分建议提到使用不同凭证，请问如何在不修改脚本的情况下为curl提供凭证？

调试信息 #

curl调试输出（curl -f -v https://www.mjr19.org.uk/sw/inode64.c） #

*   Trying 93.93.131.127:443...
* Connected to www.mjr19.org.uk (93.93.131.127) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=www.mjr19.org.uk
*  start date: Aug 12 09:35:52 2023 GMT
*  expire date: Nov 10 09:35:51 2023 GMT
*  subjectAltName: host "www.mjr19.org.uk" matched cert's "www.mjr19.org.uk"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/1.1
> GET /sw/inode64.c HTTP/1.1
> Host: www.mjr19.org.uk
> User-Agent: curl/8.1.2
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 403 Forbidden
< Date: Fri, 29 Sep 2023 09:32:36 GMT
< Server: Apache/2.4.38 (Debian)
< Content-Length: 282
< Content-Type: text/html; charset=iso-8859-1
* The requested URL returned error: 403
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (22) The requested URL returned error: 403

wget调试输出（wget -d https://www.mjr19.org.uk/sw/inode64.c） #

DEBUG output created by Wget 1.21.2 on linux-gnu.

Reading HSTS entries from /home/md1/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'inode64.c' (UTF-8) -> 'inode64.c' (UTF-8)
--2023-09-29 10:33:32--  https://www.mjr19.org.uk/sw/inode64.c
Resolving www.mjr19.org.uk (www.mjr19.org.uk)... 93.93.131.127, 2a00:1098:0:86:1000::10
Caching www.mjr19.org.uk => 93.93.131.127 2a00:1098:0:86:1000::10
Connecting to www.mjr19.org.uk (www.mjr19.org.uk)|93.93.131.127|:443... connected.
Created socket 3.
Releasing 0x000055a3132759c0 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x000055a313277110
certificate:
  subject: CN=www.mjr19.org.uk
  issuer:  CN=R3,O=Let's Encrypt,C=US
X509 certificate successfully verified and matches host www.mjr19.org.uk

---request begin---
GET /sw/inode64.c HTTP/1.1
Host: www.mjr19.org.uk
User-Agent: Wget/1.21.2
Accept: */*
Accept-Encoding: identity
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Fri, 29 Sep 2023 09:33:32 GMT
Server: Apache/2.4.38 (Debian)
Vary: User-Agent
Last-Modified: Tue, 03 Dec 2019 12:17:59 GMT
ETag: "1d03-598cbafd9f0a9"
Accept-Ranges: bytes
Content-Length: 7427
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/x-csrc

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 7427 (7.3K) [text/x-csrc]
Saving to: ‘inode64.c’

inode64.c                            100%[=====================================================================>]   7.25K  --.-KB/s    in 0s

2023-09-29 10:33:32 (119 MB/s) - ‘inode64.c’ saved [7427/7427]

问题分析与解决方法 #

对比两者的请求头和服务器响应，能看到关键差异：

1. User-Agent不同：curl使用curl/8.1.2，wget使用Wget/1.21.2，且服务器响应头包含Vary: User-Agent，说明服务器大概率根据User-Agent做了拦截。
2. wget额外发送了Accept-Encoding: identity和Connection: Keep-Alive请求头。

解决办法 #

1. 不修改脚本，让curl模拟wget请求

利用curl的环境变量CURL_OPTIONS，在执行脚本前设置该变量，自动为curl添加指定参数：

export CURL_OPTIONS="-A 'Wget/1.21.2' -H 'Accept-Encoding: identity' -H 'Connection: Keep-Alive'"

设置完成后再运行脚本，curl会自动带上这些请求头，模拟wget的请求方式，大概率能绕过403拦截。

2. 单独验证User-Agent的影响

先单独测试修改User-Agent是否能成功下载：

curl -f -A 'Wget/1.21.2' http://www.mjr19.org.uk/sw/inode64.c

如果成功，说明核心问题就是User-Agent被拦截，直接用上面的环境变量方法即可。

3. 不修改脚本为curl添加凭证

如果确实需要提供凭证（当前场景大概率不需要，但可以参考），同样可以通过环境变量实现：

• 方式一：设置CURL_USERPWD变量

  export CURL_USERPWD="用户名:密码"
  

• 方式二：通过CURL_OPTIONS添加参数

  export CURL_OPTIONS="-u 用户名:密码"
  

内容的提问来源于stack exchange，提问作者Icy