Spark连接AWS OpenSearch报"Cannot detect ES version"错误的技术求助

Let's figure out why you're running into this error when connecting to AWS OpenSearch—especially since your setup works perfectly with Elasticsearch 7.10. Here's a breakdown of the root cause and actionable fixes:

Root Cause #

The main issue here is version compatibility between your Elasticsearch Spark Connector and AWS OpenSearch:

• AWS OpenSearch is a fork of Elasticsearch 7.10, with its own versioning scheme (e.g., 1.x, 2.x, 3.x) that differs from Elasticsearch's 8.x+ releases.
• You're using Elasticsearch's official Spark Connector 8.2.3, built specifically for Elasticsearch 8.x clusters. It expects the Elasticsearch 8.x version response format, which doesn't match what OpenSearch returns—leading to the "Cannot detect ES version" error.

Solutions #

1. Switch to the Official OpenSearch Spark Connector (Recommended) #

AWS maintains its own Spark connector for OpenSearch, designed to work seamlessly with OpenSearch clusters. This is the most reliable long-term fix.

Step 1: Update Your Dependency

Replace your existing Elasticsearch connector dependency with the OpenSearch equivalent. Ensure the connector version matches your OpenSearch cluster's major/minor version (e.g., use 2.11.0 for OpenSearch 2.11.x):

libraryDependencies += "org.opensearch" % "opensearch-spark-30_2.12" % "2.11.0"

Step 2: Adjust Configuration Parameters

Most configuration keys are similar to the Elasticsearch connector, but it’s best practice to use the opensearch. prefix (though the es. prefix still works for backward compatibility). Here’s your updated config:

val conf = new SparkConf().setAppName(appName)
  .setMaster(master)
  .set("opensearch.nodes", "https://*************************.us-east-1.es.amazonaws.com")
  .set("opensearch.port", "443") // AWS OpenSearch uses 443 for HTTPS by default
  .set("opensearch.http.timeout", "5m")
  .set("opensearch.nodes.wan.only", "true")
  .set("opensearch.net.ssl", "true")
  .set("opensearch.net.http.auth.user", "********")
  .set("opensearch.net.http.auth.pass", "********")

2. Force Version Detection (Workaround for Elasticsearch 8.x Connector) #

If you need to stick with the Elasticsearch 8.x connector temporarily, you can manually specify the compatible Elasticsearch version your OpenSearch cluster is based on (7.10.x). This bypasses automatic version detection:

val conf = new SparkConf().setAppName(appName)
  // ... your existing configs ...
  .set("es.version", "7.10.2") // Match the ES version your OpenSearch is forked from

⚠️ Note: This is a workaround—you may encounter API compatibility issues later, as OpenSearch diverges from Elasticsearch 8.x in some features.

Additional Checks #

• Verify Access Permissions: Double-check that your Spark environment's IP is allowed in the AWS OpenSearch domain's access policy (your working ES 7.10 connection suggests this is set, but it’s worth confirming).
• Confirm Port and Endpoint: Ensure you’re using port 443 for HTTPS (AWS OpenSearch’s default) and that your endpoint URL is correct.

内容的提问来源于stack exchange，提问作者Ilia Torua