咨询spark-redshift连接器的替代方案及相关维护情况

Great question! The original spark-redshift connector has indeed been unmaintained for a while now, even though it used to come pre-installed in older Databricks clusters (like Spark 2.1.0-db2 and beyond). Luckily, there are several robust, actively maintained alternatives that work seamlessly with modern Databricks Runtime versions. Let’s walk through the best options:

1. Databricks Official Redshift Connector #

This is the most direct, optimized replacement since it’s built and maintained by Databricks specifically for their platform. It’s fully compatible with all recent Databricks Runtime versions and includes performance tweaks that the old connector lacked:

• Automatic parallel data reads (splits queries based on your Redshift cluster’s capacity)
• Efficient bulk writes to reduce commit overhead
• Full support for complex data types (like arrays, structs) and Redshift-specific features
• Built-in handling of S3 temporary data transfers (a requirement for Redshift bulk operations)

Usage Example (Scala): #

// Read data from Redshift
val redshiftDF = spark.read
  .format("com.databricks.spark.redshift")
  .option("url", "jdbc:redshift://your-redshift-cluster:5439/your-db?user=your-username&password=your-password")
  .option("dbtable", "your_schema.target_table")
  .option("tempdir", "s3://your-bucket/temp-redshift/") // S3 bucket with proper IAM permissions
  .load()

// Write data to Redshift
redshiftDF.write
  .format("com.databricks.spark.redshift")
  .option("url", "jdbc:redshift://your-redshift-cluster:5439/your-db?user=your-username&password=your-password")
  .option("dbtable", "your_schema.target_table")
  .option("tempdir", "s3://your-bucket/temp-redshift/")
  .mode("append")
  .save()

2. Amazon Redshift JDBC Driver #

If you prefer a lightweight, dependency-free approach, the official Redshift JDBC driver is a reliable choice. It works with all Spark versions and doesn’t require any Spark-specific connectors—just the JDBC driver JAR (which Databricks can automatically install for you).

Pros & Cons: #

• ✅ Simple setup for basic read/write operations
• ✅ Compatible with any Spark-based platform, not just Databricks
• ❌ Less optimized for large datasets (you’ll need to manually configure parallelism with numPartitions)

Usage Example (Scala): #

// Read data via JDBC
val jdbcDF = spark.read
  .format("jdbc")
  .option("url", "jdbc:redshift://your-redshift-cluster:5439/your-db")
  .option("dbtable", "your_schema.target_table")
  .option("user", "your-username")
  .option("password", "your-password")
  .option("driver", "com.amazon.redshift.jdbc.Driver")
  .option("numPartitions", "8") // Adjust based on your Redshift cluster size
  .load()

// Write data via JDBC
jdbcDF.write
  .format("jdbc")
  .option("url", "jdbc:redshift://your-redshift-cluster:5439/your-db")
  .option("dbtable", "your_schema.target_table")
  .option("user", "your-username")
  .option("password", "your-password")
  .option("driver", "com.amazon.redshift.jdbc.Driver")
  .mode("overwrite")
  .save()

3. Delta Lake + Redshift COPY/UNLOAD Commands #

For large-scale data processing (TB+ datasets), combining Delta Lake (Databricks’ default table format) with Redshift’s native COPY and UNLOAD commands delivers the best performance. This approach leverages Redshift and S3’s native capabilities instead of relying on Spark connectors.

How it works: #

• To read from Redshift: Use UNLOAD to export data directly to S3 in Parquet format, then load the Parquet files into a Delta table.
• To write to Redshift: Write your Delta table data to S3 as Parquet, then use COPY to bulk import into Redshift.

Usage Example (Scala/SQL): #

// Unload data from Redshift to S3
spark.sql("""
  UNLOAD ('SELECT * FROM your_schema.source_table')
  TO 's3://your-bucket/redshift-unload/'
  IAM_ROLE 'arn:aws:iam::123456789012:role/your-redshift-iam-role'
  FORMAT PARQUET
  PARTITION BY (load_date)
""")

// Load unloaded data into Delta Lake
val deltaDF = spark.read.parquet("s3://your-bucket/redshift-unload/")
deltaDF.write.format("delta").mode("overwrite").save("/delta/your-table")

// Write Delta data to S3 for Redshift COPY
deltaDF.write.parquet("s3://your-bucket/redshift-copy/")

// COPY data from S3 to Redshift
spark.sql("""
  COPY your_schema.target_table
  FROM 's3://your-bucket/redshift-copy/'
  IAM_ROLE 'arn:aws:iam::123456789012:role/your-redshift-iam-role'
  FORMAT PARQUET
  UPDATE
""")

Quick Recommendation #

• Drop-in replacement: Go with the Databricks Official Redshift Connector (minimal code changes, optimized for Databricks)
• Simple workflows: Use the Redshift JDBC Driver
• Large-scale data: Use Delta Lake + Redshift COPY/UNLOAD for maximum performance

内容的提问来源于stack exchange，提问作者sylvinho81