NiFi ListS3处理器无法访问Scality S3存储桶的URI解析问题

环境部署配置 #

通过Docker Compose部署以下服务：

• Scality S3Server
• Apache NiFi 1.24.0
• Apache NiFi Registry 1.24.0

对应的Docker Compose配置：

version: '3'

services:
  s3:
    image: scality/s3server
    ports:
      - "8000:8000"
    env_file:
      - .env_s3
    networks:
      - network
  

  nifi-registry:
    image: apache/nifi-registry:1.24.0
    ports:
      - "18080:18080"
    environment:
      - NIFI_REGISTRY_WEB_HTTP_PORT=18080
    networks:
      - network


  nifi:
    image: apache/nifi:1.24.0
    ports:
      - "8080:8080"
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_REGISTRY_CLIENT_SERVICE="nifi-registry:18080"
      - NIFI_REGISTRY_CLIENT_ID="nifi-registry"
    networks:
      - network
networks:
  network:

问题现象 #

配置NiFi的ListS3处理器（已设置端点覆盖URL端口为8000）后，出现如下错误：

ListS3[id=id_number] Failed to list contents of bucket due to could not parse the specified URI. Check your restEndpoints configuration.(Service: Amazon S3; Status Code: 400; Error Code: Invalid URI; Request ID: some_id; S3 Extended Request ID: null; Proxy: null):
com.amazonaws.services.s3.model.AmazonS3Exception: Could not parse the specified URI. Check your restEndpoints configuration.(Service: Amazon S3; Status Code: 400; Error Code: Invalid URI; Request ID: some_id; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null

验证情况 #

该存储桶已通过Python Boto3成功创建并可正常列出内容，相关代码如下：

import os
import boto3
from botocore.exceptions import NoCredentialsError

s3_server_host = 'localhost'
s3_server_port = 8000
s3_access_key = os.getenv('SCALITY_ACCESS_KEY_ID')
s3_secret_key = os.getenv('SCALITY_SECRET_ACCESS_KEY')

s3_bucket_name = 'loadcsvbucket'
s3_object_key = 'load_data.csv'
local_file_path = 'load_data.csv'


def create_s3_bucket():
    s3_client = boto3.client(
        "s3",
        endpoint_url=f"http://{s3_server_host}:{s3_server_port}",
        aws_access_key_id=s3_access_key,
        aws_secret_access_key=s3_secret_key,
    )

    try:
        s3_client.create_bucket(Bucket=s3_bucket_name)
        print(f"S3 bucket '{s3_bucket_name}' created successfully.")
    except NoCredentialsError as e:
        print(f"Failed to create S3 bucket. No valid credentials. Error: {e}")
    except Exception as e:
        print(f"Failed to create S3 bucket. Error: {e}")


def list_objects_in_s3_bucket():
    s3_client = boto3.client(
        "s3",
        endpoint_url=f"http://{s3_server_host}:{s3_server_port}",
        aws_access_key_id=s3_access_key,
        aws_secret_access_key=s3_secret_key,
    )

    try:
        response = s3_client.list_objects(Bucket=s3_bucket_name)
        objects = response.get("Contents", [])

        if objects:
            print(f"Objects in S3 bucket '{s3_bucket_name}':")
            for obj in objects:
                print(obj["Key"])
        else:
            print(f"S3 bucket '{s3_bucket_name}' is empty.")
    except NoCredentialsError as e:
        print(f"Failed to list objects. No valid credentials. Error: {e}")
    except Exception as e:
        print(f"Failed to list objects. Error: {e}")

替换MinIO后的结果 #

将S3服务替换为MinIO后，错误消失，MinIO的Docker配置如下：

s3:
    image: minio/minio:RELEASE.2023-12-13T23-28-55Z-cpuv1
    ports:
      - "9000:9000"
    env_file:
      - .env_s3
    volumes:
      - ./data_minio:/data
    command: server /data
    networks:
      - network

疑问 #

为何使用Scality S3Server时NiFi的ListS3处理器会出现URI解析错误，而MinIO可正常工作？

内容的提问来源于stack exchange，提问作者Peter