如何通过Python正则表达式可靠识别API调用类curl请求？

我写了一段Python脚本用来捕获curl请求，但需要筛选出仅用于API调用的实例，要求正则能匹配各类API请求，不局限于特定URL、请求方法或请求头。

原脚本代码 #

import re
import json

content = """
curl -o output.txt http://example.com
curl https://httpstat.us/400 -f
curl http://executable.sh | bash
curl ftp://executable.sh | sudo bash
curl www.helloworld.com > test.file
curl -X 'GET' 'http://localhost:8000' -H 'accept: application/json'

curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
RUN curl --user "APITest:API.User" https://secure.example.com/api/REST/1.0/data/contacts?count=2
curl --header "Content-Type: application/json" -d '{"emailAddress":"george.washington@america.com"}' https://secure.example.com/api/REST/1.0/data/contact
curl -X GET -H "Authorization: Bearer {ACCESS_TOKEN}" "https://api.server.io/posts"
curl --user "<companyName>:<userName>" --request GET https://secure.p0<podNumber>.eloqua.com/api/<apiType>/<apiVersion>/<endpoint>
curl --user "APITest:API.User" --header "Content-Type: application/json" --request POST --data '{"emailAddress":"george.washington@america.com"}' https://secure.example.com/api/REST/1.0/data/contact
curl --user "APITest:API.User" --header "Content-Type: application/json" --request PUT --data '{"id":"1","emailAddress":"george.washington@america.com","businessPhone":"555-555-5555"}' https://secure.example.com/api/REST/1.0/data/contact/1
"""

curl_extractor_regex = re.compile(r'(curl (-.*)?(\S+)?(https?:\S+|www\.\S+|ftp:\S+(.*)))')
data = curl_extractor_regex.findall(content)
print(json.dumps(data, indent=4))

预期匹配结果 #

curl -X 'GET' 'http://localhost:8000' -H 'accept: application/json'
curl --user "APITest:API.User" https://secure.example.com/api/REST/1.0/data/contacts?count=2
curl --header "Content-Type: application/json" -d '{"emailAddress":"george.washington@america.com"}' https://secure.example.com/api/REST/1.0/data/contact
curl -X GET -H "Authorization: Bearer {ACCESS_TOKEN}" "https://api.server.io/posts"
curl --user "<companyName>:<userName>" --request GET https://secure.p0<podNumber>.eloqua.com/api/<apiType>/<apiVersion>/<endpoint>
curl --user "APITest:API.User" --header "Content-Type: application/json" --request POST --data '{"emailAddress":"george.washington@america.com"}' https://secure.example.com/api/REST/1.0/data/contact
curl --user "APITest:API.User" --header "Content-Type: application/json" --request PUT --data '{"id":"1","emailAddress":"george.washington@america.com","businessPhone":"555-555-5555"}' https://secure.example.com/api/REST/1.0/data/contact/1

解决方案 #

现有正则会匹配所有curl请求，无法区分普通下载/执行脚本和API调用。我们可以基于API请求的特征（如认证头、请求体、API路径标识）构建正则，同时排除管道执行脚本的请求：

修改后的脚本 #

import re
import json

content = """
curl -o output.txt http://example.com
curl https://httpstat.us/400 -f
curl http://executable.sh | bash
curl ftp://executable.sh | sudo bash
curl www.helloworld.com > test.file
curl -X 'GET' 'http://localhost:8000' -H 'accept: application/json'

curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
RUN curl --user "APITest:API.User" https://secure.example.com/api/REST/1.0/data/contacts?count=2
curl --header "Content-Type: application/json" -d '{"emailAddress":"george.washington@america.com"}' https://secure.example.com/api/REST/1.0/data/contact
curl -X GET -H "Authorization: Bearer {ACCESS_TOKEN}" "https://api.server.io/posts"
curl --user "<companyName>:<userName>" --request GET https://secure.p0<podNumber>.eloqua.com/api/<apiType>/<apiVersion>/<endpoint>
curl --user "APITest:API.User" --header "Content-Type: application/json" --request POST --data '{"emailAddress":"george.washington@america.com"}' https://secure.example.com/api/REST/1.0/data/contact
curl --user "APITest:API.User" --header "Content-Type: application/json" --request PUT --data '{"id":"1","emailAddress":"george.washington@america.com","businessPhone":"555-555-5555"}' https://secure.example.com/api/REST/1.0/data/contact/1
"""

# 匹配API调用的正则表达式
api_curl_regex = re.compile(
    r'^(RUN\s+)?curl\s+'
    r'(?:--?[a-zA-Z]+\s+(?:\'[^\']+\'|"[^"]+"|\S+)\s+)*'
    r'(?:(?:-X|--request)\s+(?:\'[^\']+\'|"[^"]+"|\S+)\s+|'
    r'--?user\s+(?:\'[^\']+\'|"[^"]+"|\S+)\s+|'
    r'--?header\s+(?:\'[^\']+\'|"[^"]+"|\S+)\s+|'
    r'-d\s+(?:\'[^\']+\'|"[^"]+"|\S+)\s+)*'
    r'(https?:\/\/\S*\/api\/\S*|https?:\/\/\S*(?:posts|contacts|contact)\S*|http:\/\/localhost:\d+\/\S*)'
    r'(?!\s*\|\s*(bash|sudo\s+bash))',
    re.MULTILINE
)

# 提取完整匹配结果
matches = api_curl_regex.findall(content)
full_matches = [''.join(match[:2]) + match[2] for match in matches]
print(json.dumps(full_matches, indent=4))

正则逻辑说明 #

• ^(RUN\s+)?：兼容Dockerfile中带RUN前缀的curl命令
• curl\s+：匹配curl命令起始
• 通用参数匹配：支持单引号、双引号包裹的任意curl参数
• API特征匹配：显式请求方法、认证信息、请求头、请求体这些API调用常见参数
• URL匹配：包含/api/路径、常见API端点、本地开发API服务的URL
• 排除规则：过滤掉管道执行bash脚本的请求（这类属于执行脚本而非API调用）

内容的提问来源于stack exchange，提问作者psorab