You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

升级GCS Connector hadoop3-2.2.8后文件操作性能骤降求助

问题描述

我的Java应用实现了基于Hadoop文件系统的文件层封装类File,近期将GCS Connector从hadoop3-1.9.17升级至hadoop3-2.2.8(使用shaded jar)。该File类核心读写方法实现如下:

class File {
    private String path;
    private FileSystem fs;
}

@Override
public OutputStream write(boolean overwriteIfExists) throws IOException {
    return fs.create(path, overwriteIfExists);
}

@Override
public InputStream read() throws IOException {
    return fs.open(path);
}

通过多线程性能测试(每个线程创建对应特定路径的File实例并执行READ、RENAME、EXISTS、CREATE等操作)发现,新版本Connector的各项操作平均耗时远高于旧版本:

operation, hadoop3-1.9.17, hadoop3-2.2.8
READ       4542.71,        10171.26, (旧版本的2倍)
RENAME     1347.75,        4483.27,  (旧版本的4倍)
EXISTS     47.23,          1538.74,  (旧版本的50倍)
CREATE     570.1,          1539.81,  (旧版本的3倍)

已参考GitHub Issue #114调整配置但无性能改善,现咨询两个问题:

  1. 有无优化上述操作性能的参数配置指南?
  2. 即使使用shaded jar,类路径中的其他依赖jar是否可能引发兼容性问题?当前类路径包含以下jar:
  • gcs-connector-hadoop3-2.2.8-shaded.jar
  • google-extensions-0.7.1.jar
  • google-api-client-1.32.2.jar
  • google-http-client-apache-v2-1.40.1.jar
  • proto-google-common-protos-2.7.3.jar
  • google-http-client-1.41.8.jar
  • google-oauth-client-1.33.3.jar
  • google-http-client-jackson2-1.40.1.jar
  • grpc-google-cloud-storage-v2-2.2.2-alpha.jar
  • google-http-client-gson-1.41.8.jar
  • google-cloud-monitoring-1.82.0.jar
  • google-cloud-core-http-2.5.4.jar
  • proto-google-cloud-storage-v2-2.2.2-alpha.jar
  • google-api-client-jackson2-1.32.2.jar
  • google-api-services-iamcredentials-v1-rev20210326-1.32.1.jar
  • google-oauth-client-java6-1.27.0.jar
  • google-cloud-core-grpc-2.5.4.jar
  • google-http-client-appengine-1.34.2.jar
  • google-cloud-core-2.5.4.jar
  • google-auth-library-credentials-1.7.0.jar
  • google-cloud-storage-1.106.0.jar
  • proto-google-iam-v1-1.2.3.jar
  • google-api-services-storage-v1-rev20211018-1.32.1.jar
  • google-auth-library-oauth2-http-1.7.0.jar
  • proto-google-cloud-monitoring-v3-1.64.0.jar
  • grpc-services-1.43.2.jar
  • grpc-netty-shaded-1.43.2.jar
  • grpc-alts-1.43.2.jar
  • grpc-stub-1.43.2.jar
  • grpc-census-1.43.2.jar
  • grpc-protobuf-1.43.2.jar
  • grpc-api-1.43.2.jar
  • grpc-xds-1.43.2.jar
  • grpc-core-1.43.2.jar
  • grpc-protobuf-lite-1.43.2.jar
  • grpc-context-1.43.2.jar
  • opencensus-contrib-grpc-metrics-0.31.0.jar
  • grpc-auth-1.43.2.jar
  • gax-grpc-2.7.1.jar
  • grpc-grpclb-1.43.2.jar
  • api-common-2.1.4.jar
  • gax-2.7.1.jar
  • gax-httpjson-0.73.0.jar
  • util-2.2.8.jar
  • util-hadoop-hadoop3-2.2.8.jar
  • auto-value-annotations-1.9.jar

内容的提问来源于stack exchange,提问作者Selim Alawwa

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.08.16 09:20:29