使用启动参数修改GCP Dataflow指定类日志级别时遇解析错误

需求 #

在Apache Beam Java编写的GCP Dataflow作业中，将com.google.api.client.http.HttpTransport类的日志级别设为ERROR，避免INFO级别下该类产生大量冗余日志，只保留自定义INFO日志。

代码配置成功方案 #

通过自定义继承自DataflowWorkerLoggingOptions的PipelineOptions可实现需求，代码示例：

var workerLogLevelOverrides = new DataflowWorkerLoggingOptions.WorkerLogLevelOverrides();
workerLogLevelOverrides.addOverrideForClass(HttpTransport.class, DataflowWorkerLoggingOptions.Level.ERROR);
options.setWorkerLogLevelOverrides(workerLogLevelOverrides);

启动参数配置失败情况 #

尝试通过启动参数--workerLogLevelOverrides={"com.google.api.client.http.HttpTransport":"ERROR"}配置时，触发JSON解析错误，报错信息如下：

Unable to parse JSON value {com.google.api.client.http.HttpTransport:ERROR}

java.lang.IllegalArgumentException: Unable to parse JSON value {com.google.api.client.http.HttpTransport:ERROR}
    at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1953)
    at org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:146)
    at org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:329)
    at com.macys.commonservices.ivs.ingestion.enrich.supplydemand.pipeline.EnrichSupplyDemandPipeline.main(EnrichSupplyDemandPipeline.java:26)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{com.google.api.client.http.HttpTransport:ERROR}')
 at [Source: UNKNOWN; line: -1, column: -1]
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1429)
    at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1059)
    at com.fasterxml.jackson.databind.deser.ValueInstantiator._createFromStringFallbacks(ValueInstantiator.java:371)
    at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromString(ValueInstantiator.java:258)
    at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:357)
    at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
    at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:345)
    at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
    at org.apache.beam.sdk.options.PipelineOptionsFactory.deserializeNode(PipelineOptionsFactory.java:1807)
    at org.apache.beam.sdk.options.PipelineOptionsFactory.tryParseObject(PipelineOptionsFactory.java:1839)
    at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1951)
    ... 9 common frames omitted

已尝试过已弃用的workerLogLevelOverrides和新的sdkHarnessLogLevelOverrides参数，均无法解决问题。

尝试过的命令 #

• mvn compile exec:java -Dexec.mainClass=org.example.WordCount -Dexec.args="--runner=DataflowRunner --project=my-project --stagingLocation=gs://my-bucket/staging --templateLocation=gs://my-bucket/templates/wordcount --tempLocation=gs://my-bucket/templates/temp/ --workerLogLevelOverrides={\"com.google.api.client.http.HttpTransport\":\"ERROR\"}"
• mvn compile exec:java -Dexec.mainClass=org.example.WordCount -Dexec.args="--runner=DataflowRunner --project=my-project --stagingLocation=gs://my-bucket/staging --templateLocation=gs://my-bucket/templates/wordcount --tempLocation=gs://my-bucket/templates/temp/ --workerLogLevelOverrides={'com.google.api.client.http.HttpTransport':'ERROR'}"
• mvn compile exec:java -Dexec.mainClass=org.example.WordCount -Dexec.args='--runner=DataflowRunner --project=my-project --stagingLocation=gs://my-bucket/staging --templateLocation=gs://my-bucket/templates/wordcount --tempLocation=gs://my-bucket/templates/temp/ --workerLogLevelOverrides={"com.google.api.client.http.HttpTransport":"ERROR"}'

所有命令均触发上述解析错误。

内容的提问来源于stack exchange，提问作者Ievgen Grozenok