如何在本地AWS Glue交互式Notebook中添加Hudi包及魔法命令使用咨询

Hey there! Let's break down exactly which magic commands to use and how to implement them for adding Hudi to your local Glue Interactive Sessions.

First, let's clarify what each of the magic commands you mentioned does, so you understand why we'll use specific ones for Hudi:

• %additional_python_modules: Installs Python packages (like Hudi's Python client) directly into your session's environment.
• %extra_jars: Adds Java/Scala JAR files that Spark (and Glue) needs for integrating with tools like Hudi—this is critical because Hudi's core Spark integration relies on these JARs.
• %extra_py_files: Used for loading custom Python scripts, ZIP files, or modules you've written yourself. You won't typically need this for standard Hudi setup.

Step-by-Step Setup for Hudi #

Since Hudi requires both Python API access and Spark JAR dependencies, you'll need to use both %extra_jars and %additional_python_modules together. Here's how to do it correctly:

1. Match Versions First
   Make sure you pick Hudi versions compatible with your Glue session's Spark version. For example:

   • Glue 4.0 uses Spark 3.3 → Use Hudi 0.13.x (e.g., 0.13.1)
   • Glue 3.0 uses Spark 3.1 → Use Hudi 0.12.x
     Always align the Python package version with the JAR version to avoid compatibility issues.

2. Add Hudi Spark JARs
   Use %extra_jars to pull the pre-built Hudi Spark bundle from Maven (Glue can fetch directly from Maven repositories):

   %extra_jars org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1
   

   If you have a local JAR file instead, you can specify its absolute path:

   %extra_jars /your/local/path/hudi-spark3.3-bundle_2.12:0.13.1.jar
   

3. Install Hudi Python Package
   Run %additional_python_modules to install the matching Hudi Python package:

   %additional_python_modules hudi==0.13.1
   

   For a local Python package file (like a .tar.gz), use:

   %additional_python_modules /your/local/path/hudi-0.13.1.tar.gz
   

4. Verify the Setup
   After running the commands, test if Hudi is working by importing the modules and running a basic operation. For example:

   from hudi import HoodieSparkSession
   from pyspark.sql import SparkSession
   
   # Initialize a Spark session with Hudi configs
   spark = SparkSession.builder \
       .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
       .getOrCreate()
   
   # Check if Hudi is accessible
   spark.sql("CREATE TABLE IF NOT EXISTS hudi_test (id STRING, name STRING) USING hudi")
   

Key Notes #

• Always run these magic commands before importing Hudi or running any Hudi-related code in your session.
• If you run into version conflicts, double-check that your Hudi versions match your Glue/Spark version.
• If you need to add multiple JARs or Python packages, separate them with commas:

  %extra_jars org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1,com.amazonaws:aws-java-sdk-bundle:1.12.262
  %additional_python_modules hudi==0.13.1,pandas==2.0.3
  

内容的提问来源于stack exchange，提问作者NarenS