Spark Structured Streaming对接Kinesis出现ResourceNotFoundException求助

Let's break down why you're hitting this ResourceNotFoundException even though the AWS SDK can successfully reach your Kinesis stream, and how to fix it:

1. Fix the Spark Kinesis Connector Parameter Name (Most Likely Root Cause) #

Looking at your Spark code, you're using option("streams", streamName) to specify the Kinesis stream name—but this is incorrect. The official Spark Kinesis Connector expects the parameter name to be streamName (singular), not streams.

This mismatch means Spark is actually trying to connect to a stream with an invalid/empty name (or misinterpreting the parameter), hence the "stream not found" error. Your SDK code works because you're using the correct streamName parameter in the describeStream call.

Corrected Spark Code Snippet: #

val kinesis = spark 
 .readStream 
 .format("kinesis") 
 .option("streamName", streamName)  // Changed from "streams" to "streamName"
 .option("region", "us-east-1") 
 .option("initialPosition", "TRIM_HORIZON") 
 .option("endpointUrl", "kinesis.us-east-1.amazonaws.com") 
 .option("awsAccessKey", accessKey) 
 .option("awsSecretKey", secretKey) 
 .option("format", "json") 
 .option("inferSchema", "true") 
 .schema(schema) 
 .load

2. Verify Connector and AWS SDK Compatibility #

If fixing the parameter name doesn't resolve the issue, check that your Spark Kinesis Connector version is compatible with the AWS SDK version you're using. Mismatched versions can lead to unexpected API behavior or authentication issues.

For example:

• If you're using Spark 3.x, ensure you're using a connector version designed for Spark 3 (like org.apache.spark:spark-streaming-kinesis-asl_2.12:3.3.0 or similar)
• Avoid mixing old and new AWS SDK dependencies in your project build (e.g., aws-java-sdk-kinesis versions should align with the connector's transitive dependencies)

3. Double-Check Credential and Region Consistency #

Even though your SDK code works, confirm that the credentials and region used in your Spark code are identical:

• Ensure the accessKey and secretKey variables in Spark are pulling the correct values from your environment variables (no typos in variable names)
• Verify the Kinesis stream is definitely provisioned in the us-east-1 region (cross-region calls will fail with this error)

4. Validate Kinesis Stream ARN/Name Spelling #

A simple typo in the stream name can cause this error. Cross-check the streamName variable in both your Spark and SDK code to ensure they're identical—even a single character difference (like uppercase/lowercase, hyphens, or numbers) will cause the stream to be "not found".

内容的提问来源于stack exchange，提问作者mc29