如何为Spark JDBC连接指定trustStore及trustStoreType

Hey there! I know how frustrating it can be when SSL configurations don't stick in Spark—let's work through this together. You mentioned you've already placed the SSO (Oracle Wallet) file on all executor nodes, so let's dive into the common misconfigurations that might still be blocking you.

1. Make Sure You're Configuring Both Driver and Executor Nodes #

Spark runs tasks across executor nodes, so setting SSL properties only on the driver won't work. You need to pass these settings to both driver and executors via SparkConf or your spark-submit command.

Example with SparkConf (Java Code) #

import org.apache.spark.SparkConf;
import org.apache.spark.sql.SparkSession;

public class OracleToORC {
    public static void main(String[] args) {
        SparkConf conf = new SparkConf()
                .setAppName("OracleToORCExport")
                // Configure SSL properties for driver
                .set("spark.driver.extraJavaOptions", 
                     "-Djavax.net.ssl.trustStore=/absolute/path/to/your/truststore -Djavax.net.ssl.trustStorePassword=your-truststore-pass -Djavax.net.ssl.trustStoreType=JKS")
                // Mirror the same config for executors
                .set("spark.executor.extraJavaOptions", 
                     "-Djavax.net.ssl.trustStore=/absolute/path/to/your/truststore -Djavax.net.ssl.trustStorePassword=your-truststore-pass -Djavax.net.ssl.trustStoreType=JKS");

        SparkSession spark = SparkSession.builder()
                .config(conf)
                .getOrCreate();

        // Your Oracle read and ORC write logic here
        spark.read()
                .format("jdbc")
                .option("url", "jdbc:oracle:thin:@your-oracle-host:port/your-service-name")
                .option("dbtable", "your_table")
                .option("user", "your_username")
                .option("password", "your_password")
                .load()
                .write()
                .format("orc")
                .save("path/to/orc/output");

        spark.stop();
    }
}

2. Use Oracle Wallet-Specific Configuration (If Using SSO Files) #

Since you're working with an Oracle SSO file (part of an Oracle Wallet), you might need to use Oracle's native Wallet parameters instead of generic SSL trustStore settings. Update your JDBC URL to include the wallet location:

Updated JDBC URL with Wallet #

String jdbcUrl = "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=your-oracle-host)(PORT=your-ssl-port))(CONNECT_DATA=(SERVICE_NAME=your-service-name))(SECURITY=(SSL_SERVER_CERT_DN=\"CN=your-oracle-cert-dn\")))?oracle.net.wallet_location=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/absolute/path/to/wallet-folder)))";

• Ensure the wallet folder contains all required files (usually cwallet.sso, ewallet.p12, sqlnet.ora, etc.) on every executor node.
• Double-check the absolute path is identical across all nodes—relative paths can cause issues.

3. Verify Spark Submit Command (If Using Cluster Mode) #

If you're submitting your app via spark-submit, explicitly pass the SSL/Wallet properties in the command to ensure they're applied to all nodes:

spark-submit \
  --class com.your.package.OracleToORC \
  --master yarn \
  --driver-java-options "-Djavax.net.ssl.trustStore=/path/to/truststore -Djavax.net.ssl.trustStorePassword=your-pass" \
  --executor-java-options "-Djavax.net.ssl.trustStore=/path/to/truststore -Djavax.net.ssl.trustStorePassword=your-pass" \
  --jars ojdbc8.jar \
  your-spark-app.jar

4. Quick Checks to Rule Out Simple Issues #

• Permissions: Ensure the Spark user has read access to the trustStore/wallet files on all nodes.
• JDBC Driver Version: Use a compatible driver (ojdbc8 or ojdbc11) that matches your Oracle database version—older drivers might not support modern SSL configurations.
• Validate Paths: SSH into an executor node and confirm the trustStore/wallet path exists and is accessible.

内容的提问来源于stack exchange，提问作者Sai Kumar