Sqoop导入语法：从100张表的数据库中仅导5张，禁用--exclude命令

Since you can't use the --exclude flag, the simplest and most reliable method is to import each of your 5 tables individually with Sqoop. If you want to avoid repeating the same command multiple times, you can wrap the imports in a shell script to automate the process.

1. Individual Sqoop Import Command (Per Table) #

For each table you want to import, run a dedicated sqoop import command. Here's a template you can adapt to your setup:

sqoop import \
  --connect jdbc:mysql://your-db-host:3306/your-database-name \
  --username your-db-username \
  --password your-db-password \
  --table your-target-table \
  --target-dir /hdfs/storage/path/for/table \
  --fields-terminated-by ',' \
  --lines-terminated-by '\n' \
  --delete-target-dir  # Optional: removes existing target directory to avoid conflicts

Quick Tips: #

• Swap out placeholders like your-db-host and your-target-table with your actual database details.
• Adjust the JDBC connection string if you’re using a non-MySQL database:
  • Oracle: jdbc:oracle:thin:@//host:port/service
  • PostgreSQL: jdbc:postgresql://host:port/dbname
• The --delete-target-dir flag is optional but handy if you want to overwrite existing data in HDFS without manual cleanup.

2. Automated Shell Script for Multiple Tables #

To import all 5 tables in one go, use a shell script to loop through your target tables. This saves you from typing the same command repeatedly:

#!/bin/bash

# Configure your database and HDFS settings
DB_HOST="your-db-host"
DB_NAME="your-database-name"
DB_USER="your-db-username"
DB_PASS="your-db-password"
HDFS_ROOT="/hdfs/base/directory"

# List of the 5 tables you want to import
TARGET_TABLES=("table1" "table2" "table3" "table4" "table5")

# Loop through each table and run the import
for TABLE in "${TARGET_TABLES[@]}"
do
  echo "Starting import for table: $TABLE"
  sqoop import \
    --connect jdbc:mysql://$DB_HOST:3306/$DB_NAME \
    --username $DB_USER \
    --password $DB_PASS \
    --table $TABLE \
    --target-dir "$HDFS_ROOT/$TABLE" \
    --fields-terminated-by ',' \
    --lines-terminated-by '\n' \
    --delete-target-dir
  echo "Completed import for table: $TABLE"
done

How to Use the Script: #

1. Save the code as import_tables.sh.
2. Make it executable with chmod +x import_tables.sh.
3. Run it with ./import_tables.sh.

This approach gives you full control over exactly which tables get imported, and works seamlessly when --exclude isn’t available.

内容的提问来源于stack exchange，提问作者K Rakesh patra