Scala调用CURL重建Elasticsearch索引时映射配置失效问题

问题场景 #

本地使用Scala的sys.process调用CURL重置Elasticsearch索引，删除索引的操作成功，但重建索引时传递包含settings和mappings的JSON请求体遇到问题：

1. 删除索引的成功代码：

import scala.sys.process._

val delete_cmd = Seq("curl", "-H", "Content-Type: application/json", "-X", "DELETE", "http://127.0.0.1:9222/my_index")
delete_cmd.!

2. 重建索引的尝试代码：

val body =
      """
        |{
        |    "settings": {
        |        "index": {
        |            "number_of_shards" : 40,
        |            "number_of_replicas": 0
        |        }
        |    },
        |    "mappings": {
        |        "properties": {
        |            "pk": {
        |                "type": "text",
        |                "analyzer": "keyword"
        |            },
        |            "field1": {
        |                "type": "text"
        |            },
        |            "field2": {
        |                "type": "text"
        |            },
        |            "field3": {
        |                "type": "text"
        |            }
        |        }
        |    }
        |}
        """

val recreate_cmd = Seq("curl", "-H", "Content-Type: application/json", "-X", "PUT", "-d", s"@$body","http://127.0.0.1:9222/my_index")
recreate_cmd.!

执行后索引虽创建成功，但自定义配置被忽略，收到警告：

Warning: Couldn't read data from file "
        |{
        |    settings: {

Warning:         |        index: {
        |            number_of_shards : 
Warning: 40,
        |            number_of_replicas: 0
        |        }
 
Warning:        |    },
        |    mappings: {
        |        
Warning: properties: {
        |            pk: {
        |                
Warning: type: text,
        |                analyzer: keyword
        |    
Warning:         },
        |            field1: {
        |                
Warning: type: text
        |            },
        |            field2: {
 
Warning:        |                type: text
        |            },
        
Warning: |            field3: {
        |                type: text
        
Warning: |            }
        |        }
        |    }
        |}
      
Warning:   ", this makes an empty POST.

去掉-d后的@符号后，又出现错误：

{"error":{"root_cause":[{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}],"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"},"status":500}

错误原因 #

• 带@的参数：CURL中-d @文件名表示从文件读取请求体，这里把JSON字符串直接拼接在@后，CURL会将整个带格式的JSON当作文件名尝试读取，找不到文件导致请求体为空，ES使用默认配置创建索引。
• 去掉@后直接传字符串：多行字符串中的|符号和多余换行未被清理，传递给ES的JSON格式不符合要求，触发解析错误。

正确解法 #

步骤1：清理JSON字符串格式 #

使用stripMargin去掉多行字符串中的|符号，再用trim清理首尾空格，得到标准的JSON字符串：

val body =
      """
        |{
        |    "settings": {
        |        "index": {
        |            "number_of_shards" : 40,
        |            "number_of_replicas": 0
        |        }
        |    },
        |    "mappings": {
        |        "properties": {
        |            "pk": {
        |                "type": "text",
        |                "analyzer": "keyword"
        |            },
        |            "field1": {
        |                "type": "text"
        |            },
        |            "field2": {
        |                "type": "text"
        |            },
        |            "field3": {
        |                "type": "text"
        |            }
        |        }
        |    }
        |}
        """.stripMargin.trim

步骤2：正确传递请求体 #

直接将清理后的JSON字符串作为-d的参数，不需要添加@符号：

val recreate_cmd = Seq(
  "curl", 
  "-H", "Content-Type: application/json", 
  "-X", "PUT", 
  "-d", body,
  "http://127.0.0.1:9222/my_index"
)
recreate_cmd.!

替代方案（管道传递） #

也可以通过bash管道将JSON传递给CURL，避免字符串转义问题：

val recreate_cmd = Seq("bash", "-c", s"""echo '$body' | curl -H "Content-Type: application/json" -X PUT -d @- http://127.0.0.1:9222/my_index""")
recreate_cmd.!

执行修正后的代码，Elasticsearch会正确读取settings和mappings配置，创建符合要求的索引。

内容的提问来源于stack exchange，提问作者Ignacio Alorre