升级GCS Connector hadoop3-2.2.8后文件操作性能骤降求助
问题描述
我的Java应用实现了基于Hadoop文件系统的文件层封装类File,近期将GCS Connector从hadoop3-1.9.17升级至hadoop3-2.2.8(使用shaded jar)。该File类核心读写方法实现如下:
class File { private String path; private FileSystem fs; } @Override public OutputStream write(boolean overwriteIfExists) throws IOException { return fs.create(path, overwriteIfExists); } @Override public InputStream read() throws IOException { return fs.open(path); }
通过多线程性能测试(每个线程创建对应特定路径的File实例并执行READ、RENAME、EXISTS、CREATE等操作)发现,新版本Connector的各项操作平均耗时远高于旧版本:
operation, hadoop3-1.9.17, hadoop3-2.2.8 READ 4542.71, 10171.26, (旧版本的2倍) RENAME 1347.75, 4483.27, (旧版本的4倍) EXISTS 47.23, 1538.74, (旧版本的50倍) CREATE 570.1, 1539.81, (旧版本的3倍)
已参考GitHub Issue #114调整配置但无性能改善,现咨询两个问题:
- 有无优化上述操作性能的参数配置指南?
- 即使使用shaded jar,类路径中的其他依赖jar是否可能引发兼容性问题?当前类路径包含以下jar:
- gcs-connector-hadoop3-2.2.8-shaded.jar
- google-extensions-0.7.1.jar
- google-api-client-1.32.2.jar
- google-http-client-apache-v2-1.40.1.jar
- proto-google-common-protos-2.7.3.jar
- google-http-client-1.41.8.jar
- google-oauth-client-1.33.3.jar
- google-http-client-jackson2-1.40.1.jar
- grpc-google-cloud-storage-v2-2.2.2-alpha.jar
- google-http-client-gson-1.41.8.jar
- google-cloud-monitoring-1.82.0.jar
- google-cloud-core-http-2.5.4.jar
- proto-google-cloud-storage-v2-2.2.2-alpha.jar
- google-api-client-jackson2-1.32.2.jar
- google-api-services-iamcredentials-v1-rev20210326-1.32.1.jar
- google-oauth-client-java6-1.27.0.jar
- google-cloud-core-grpc-2.5.4.jar
- google-http-client-appengine-1.34.2.jar
- google-cloud-core-2.5.4.jar
- google-auth-library-credentials-1.7.0.jar
- google-cloud-storage-1.106.0.jar
- proto-google-iam-v1-1.2.3.jar
- google-api-services-storage-v1-rev20211018-1.32.1.jar
- google-auth-library-oauth2-http-1.7.0.jar
- proto-google-cloud-monitoring-v3-1.64.0.jar
- grpc-services-1.43.2.jar
- grpc-netty-shaded-1.43.2.jar
- grpc-alts-1.43.2.jar
- grpc-stub-1.43.2.jar
- grpc-census-1.43.2.jar
- grpc-protobuf-1.43.2.jar
- grpc-api-1.43.2.jar
- grpc-xds-1.43.2.jar
- grpc-core-1.43.2.jar
- grpc-protobuf-lite-1.43.2.jar
- grpc-context-1.43.2.jar
- opencensus-contrib-grpc-metrics-0.31.0.jar
- grpc-auth-1.43.2.jar
- gax-grpc-2.7.1.jar
- grpc-grpclb-1.43.2.jar
- api-common-2.1.4.jar
- gax-2.7.1.jar
- gax-httpjson-0.73.0.jar
- util-2.2.8.jar
- util-hadoop-hadoop3-2.2.8.jar
- auto-value-annotations-1.9.jar
内容的提问来源于stack exchange,提问作者Selim Alawwa
相关产品推荐
相关产品推荐

