受限权限下AWS Glue Java SDK 1.X获取数据库列表返回空

问题背景 #

当前AWS Glue环境存在多个数据库，IAM资源策略仅允许查看名称以glue开头的数据库。使用AWS CLI执行aws glue get-databases --query "DatabaseList[*].Name"可正常返回目标数据库列表，但使用AWS Glue SDK for Java 1.X编写的代码调用GetDatabases接口却返回空列表。

相关策略配置 #

IAM资源策略

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetTables",
                "glue:GetTableVersions",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetDatabase",
                "glue:GetJob",
                "glue:StartJobRun",
                "glue:BatchStopJobRun",
                "glue:GetCrawler"
            ],
            "Resource": [
                "arn:aws:glue:*:248135293344:connection/*",
                "arn:aws:glue:*:248135293344:job/*",
                "arn:aws:glue:*:248135293344:database/glue*",
                "arn:aws:glue:*:248135293344:catalog",
                "arn:aws:glue:*:248135293344:table/glue*"
            ]
        }
    ]
}

S3策略

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketAcl"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        }
    ]
}

问题代码示例 #

package alation.awsglue.mde.extractor;

import alation.aws.auth.AWSAuthentication;
import alation.awsglue.Main;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.glue.AWSGlue;
import com.amazonaws.services.glue.AWSGlueClientBuilder;
import com.amazonaws.services.glue.model.AWSGlueException;
import com.amazonaws.services.glue.model.Database;
import com.amazonaws.services.glue.model.GetDatabasesRequest;
import com.amazonaws.services.glue.model.GetDatabasesResult;
import org.apache.log4j.PropertyConfigurator;

import java.util.List;

public class sampleMDE {
    public static void main(String[] args) {
        System.setProperty("com.amazonaws.sdk.enableDefaultMetrics", "true");
        PropertyConfigurator.configure(Main.class.getClassLoader().getResource("log4j.properties"));
        AWSGlue amazonGlueClient = null;
        try{
        // Initialize the Glue client
        AWSCredentialsProvider awsCredProvider =
            new AWSAuthentication(
                "<your_key>",
                "<your_id>")
                .getCredential();
        amazonGlueClient = AWSGlueClientBuilder.standard()
            .withCredentials(awsCredProvider)
            .withRegion(Regions.US_EAST_1).build();
    } catch (Exception e) {
        System.err.println("Failed to create Glue client: " + e.getMessage());
        e.printStackTrace();
        return;
    }
        try{

        // Create the request
        GetDatabasesRequest request = new GetDatabasesRequest();

        // Get the list of databases
        GetDatabasesResult response = amazonGlueClient.getDatabases(request);

        // Process the response
        List<Database> databases = response.getDatabaseList();
        if (databases.isEmpty()) {
            System.out.println("No databases found.");
        }
        for (Database database : databases) {
            System.out.println("Database name: " + database.getName());
        }
    } catch (AWSGlueException e) {
        System.err.println("Error occurred while retrieving databases: " + e.getErrorMessage());
        e.printStackTrace();
    }

    }
}

排查分析与解决方案 #

核心原因 #

1. SDK版本兼容性：旧版本的AWS Glue Java SDK 1.X对资源级权限的过滤支持不完善，无法正确返回符合权限要求的数据库列表。
2. 分页处理缺失：当数据库数量超过API默认分页大小（100条）时，Java SDK默认不会自动遍历所有分页结果，而CLI会自动处理分页。
3. 请求参数默认行为差异：CLI调用get-databases时会隐式设置部分参数（如MaxResults），而Java SDK的GetDatabasesRequest默认配置可能未匹配该行为。

排查步骤 #

1. 验证SDK版本：检查当前使用的AWS Glue SDK版本，建议升级到最新的1.X稳定版（如1.12.500+）。
2. 开启调试日志：添加以下代码开启SDK调试日志，查看实际API请求与响应内容，确认是否是API返回空列表还是SDK解析问题：

   System.setProperty("com.amazonaws.sdk.debug", "true");
   

3. 检查凭证与区域：确认Java代码使用的凭证、区域与CLI完全一致，避免因区域不匹配导致资源不可见。

解决方案 #

方案1：升级AWS Glue SDK版本

更新项目依赖中的AWS Glue SDK版本，例如Maven依赖：

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-glue</artifactId>
    <version>1.12.600</version>
</dependency>

方案2：显式处理分页逻辑

修改代码，添加分页遍历逻辑，确保获取所有有权限的数据库：

try {
    GetDatabasesRequest request = new GetDatabasesRequest();
    String nextToken = null;
    do {
        request.setNextToken(nextToken);
        GetDatabasesResult response = amazonGlueClient.getDatabases(request);
        List<Database> databases = response.getDatabaseList();
        if (!databases.isEmpty()) {
            for (Database database : databases) {
                System.out.println("Database name: " + database.getName());
            }
        }
        nextToken = response.getNextToken();
    } while (nextToken != null);
} catch (AWSGlueException e) {
    System.err.println("Error occurred while retrieving databases: " + e.getErrorMessage());
    e.printStackTrace();
}

方案3：验证IAM策略权限

确认IAM策略中glue:GetDatabases的资源包含catalog和database/glue*，确保权限覆盖正确。当前用户提供的策略已包含这两个资源，此步骤可作为验证。

内容的提问来源于stack exchange，提问作者Shubhankar Bhadra