执行Datadog RUM事件提取脚本时遇无效游标错误求协助

我编写了一段用于提取Datadog RUM事件日志的Shell脚本，代码如下：

cursor=$(echo $response | jq -r .meta.page.after)

data='{
"filter": {
"from": "now",
"to": "now-2d",
"query": "*"
},
"page": {
"cursor": "{{cursor}}",
"limit": 1000
}
}'
data=$(echo $data | sed "s/{{cursor}}/$cursor/")

response =$(curl -k POST "https://api.datadoghq.com/api/v2/rum/events/search" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: $DATADOG_API_KEY" \
-H "DD-APPLICATION-KEY: $DATADOG_APP_KEY" \
--data-raw "$data")

echo $response >> output

执行脚本时遇到错误："input invalidation error(Field 'page' is invalid: invalid cursor)"

1. 修复变量赋值语法错误 #

脚本中response =$(curl ...)存在空格，这会导致response变量无法正确接收curl的返回结果，后续提取的cursor会是空值。修正为：

response=$(curl -k POST "https://api.datadoghq.com/api/v2/rum/events/search" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: $DATADOG_API_KEY" \
-H "DD-APPLICATION-KEY: $DATADOG_APP_KEY" \
--data-raw "$data")

2. 处理初始请求的cursor为空场景 #

第一次请求Datadog API时不需要传入cursor字段，只有当第一次请求返回了有效的cursor后，后续分页请求才需要携带该参数。可以通过判断cursor是否为空来动态生成请求体：

# 初始请求时cursor为空，构建不带page.cursor的请求体
if [ -z "$cursor" ]; then
  data='{
  "filter": {
  "from": "now",
  "to": "now-2d",
  "query": "*"
  },
  "page": {
  "limit": 1000
  }
  }'
else
  # 有有效cursor时，携带cursor参数
  data='{
  "filter": {
  "from": "now",
  "to": "now-2d",
  "query": "*"
  },
  "page": {
  "cursor": "'"$cursor"'",
  "limit": 1000
  }
  }'
fi

3. 验证cursor的有效性 #

提取cursor后，先检查是否为非空且有效的值（比如jq返回null时要处理）：

# 提取cursor，若不存在则设为空字符串
cursor=$(echo "$response" | jq -r '.meta.page.after // empty')

完整修正后的脚本示例 #

# 初始化cursor为空
cursor=""

# 循环获取所有分页数据，直到cursor为空
while true; do
  # 动态构建请求体
  if [ -z "$cursor" ]; then
    data='{
    "filter": {
    "from": "now",
    "to": "now-2d",
    "query": "*"
    },
    "page": {
    "limit": 1000
    }
    }'
  else
    data='{
    "filter": {
    "from": "now",
    "to": "now-2d",
    "query": "*"
    },
    "page": {
    "cursor": "'"$cursor"'",
    "limit": 1000
    }
    }'
  fi

  # 发送请求
  response=$(curl -k POST "https://api.datadoghq.com/api/v2/rum/events/search" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: $DATADOG_API_KEY" \
  -H "DD-APPLICATION-KEY: $DATADOG_APP_KEY" \
  --data-raw "$data")

  # 将结果写入文件
  echo "$response" >> output

  # 提取下一页cursor，若不存在则退出循环
  cursor=$(echo "$response" | jq -r '.meta.page.after // empty')
  if [ -z "$cursor" ]; then
    break
  fi
done

内容的提问来源于stack exchange，提问作者rose1110