Sqoop增量导入（append模式）未知last-value时如何编写命令

Got it, let's tackle this problem step by step. When you're doing an incremental append import with Sqoop and don't know the last-value from the previous run, you've got a few solid options depending on your scenario:

1. 手动查询数据库获取当前最大的check-column值 #

This is the most precise approach if you want to avoid importing duplicate data. First, run a SQL query on your MySQL database to get the maximum value of your check-column (which is rank in your case):

SELECT MAX(rank) FROM ydb.yloc;

Let's say the result is 100—you can then plug this value directly into your Sqoop command using the --last-value parameter:

sqoop import \
--connect jdbc:mysql://localhost:3306/ydb \
--table yloc \
--username root \
-P \
--check-column rank \
--incremental append \
--last-value 100

This ensures you only import records where rank is greater than 100—no duplicates, no missed data.

2. 使用极小值作为last-value（适合首次增量导入） #

If this is your first time running an incremental import (or you're okay with doing a full import first), you can set --last-value to the smallest possible value for your rank column. For example, if rank is a positive integer, use 0:

sqoop import \
--connect jdbc:mysql://localhost:3306/ydb \
--table yloc \
--username root \
-P \
--check-column rank \
--incremental append \
--last-value 0

This will import all records where rank is greater than 0—essentially a full import. After this run, you can note the max rank from this import for future incremental runs, or use the metastore option to automate tracking.

3. 使用Sqoop Metastore自动跟踪last-value #

If you want to avoid manually specifying last-value altogether, set up Sqoop's metastore to automatically store and retrieve the last imported value.

First, start the Sqoop metastore service (if it's not already running):

sqoop metastore

Then, add the --metastore-server parameter to your import command (default port is 16000):

sqoop import \
--connect jdbc:mysql://localhost:3306/ydb \
--table yloc \
--username root \
-P \
--check-column rank \
--incremental append \
--metastore-server localhost:16000

The metastore will save the maximum rank value from this import. The next time you run the same command, Sqoop will automatically pull the stored last-value and only import new records—no manual input needed.

关键注意事项 #

• Make sure your check-column (rank here) is a sortable data type (integer, timestamp, etc.)—this is required for incremental imports to work correctly.
• The append mode works best when your check-column is strictly incrementing (like an auto-increment ID or a timestamp that only goes forward).

内容的提问来源于stack exchange，提问作者Mr R Rao