使用K8s OpenTelemetry Operator时Python自动插桩报错求助

在Deployment模式下使用OpenTelemetry Collector，通过Kubernetes Operator对Python容器自动插桩时出现以下错误（Java和NodeJS容器插桩正常）：

manager {"level":"error","ts":"2023-08-01T19:00:00Z","msg":"failed to select an OpenTelemetry Instrumentation instance for this pod","namespace":"namespace","name":" ","error":"no OpenTelemetry Instrumentation instances available","stacktrace":"github.com/open-telemetry/opentelemetry-operator/pkg/instrumentation.(*instPodMutator).Mutate\n\t/workspace/pkg/instrumentation/podmutator.go:87\ngithub.com/open-telemetry/opentelemetry-operator/internal/webhookhandler.(*podSidecarInjector).Handle\n\t/workspace/internal/webhookhandler/webhookhandler.go:92\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/webhook.go:169\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/http.go:98\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2122\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2122\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:109\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2122\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2500\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2936\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1995"} 

Pod规格中的注解：

instrumentation.opentelemetry.io/container-names: otel-getting-started                                                                                  
instrumentation.opentelemetry.io/inject-python: opentelemetry-instrumentation/my-instrumentation    

对应的Instrumentation资源配置：

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  annotations:
    instrumentation.opentelemetry.io/default-auto-instrumentation-apache-httpd-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    instrumentation.opentelemetry.io/default-auto-instrumentation-dotnet-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:0.7.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-go-image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.2.1-alpha
    instrumentation.opentelemetry.io/default-auto-instrumentation-java-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.26.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-nodejs-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.40.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-python-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.39b0
  creationTimestamp: "2023-05-12T18:56:38Z"
  generation: 25
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: my-instrumentation
  namespace: opentelemetry-instrumentation
spec:
  apacheHttpd:
    configPath: /usr/local/apache2/conf
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    version: "2.4"
  dotnet:
    env:
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none
    - name: OTEL_TRACES_EXPORTER
      value: otlp
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:0.7.0
  exporter: {}
  java:
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://deployment-collector.opentelemetry.svc.cluster.local:4317
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none
    - name: OTEL_TRACES_EXPORTER
      value: otlp
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.26.0
  nodejs:
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://deployment-collector.opentelemetry.svc.cluster.local:4317
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none
    - name: OTEL_TRACES_EXPORTER
      value: otlp
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.40.0
  propagators:
  - tracecontext
  - baggage
  - b3multi
  - b3
  python:
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://deployment-collector.opentelemetry.svc.cluster.local:4318
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none
    - name: OTEL_TRACES_EXPORTER
      value: otlp
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.39b0
  resource: {}
  sampler: {}

排查原因 #

• 命名空间权限与匹配问题：Pod所在命名空间为namespace，但Instrumentation资源在opentelemetry-instrumentation命名空间。Operator默认优先在Pod同命名空间查找实例，跨命名空间引用需要Operator具备对应权限，否则无法访问其他命名空间的Instrumentation资源。
• 注解格式潜在问题：当前注解格式看似正确，但可能存在隐形空格或换行问题，导致Operator无法正确解析Instrumentation的引用路径。
• Python插桩镜像可访问性：Pod所在节点可能无法拉取指定的Python自动插桩镜像，导致Operator判定该Instrumentation实例不可用。

解决方法 #

1. 配置Operator跨命名空间访问权限

   • 检查Operator绑定的ClusterRole，确保包含对opentelemetry.io/v1alpha1组下instrumentations资源的get、list权限，且权限范围覆盖所有命名空间。若缺失则更新ClusterRole添加对应规则。

2. 修正Pod注解格式

   • 确保注解没有隐形字符，将两行注解分开书写，避免连在一起导致解析错误：

     instrumentation.opentelemetry.io/container-names: otel-getting-started
     instrumentation.opentelemetry.io/inject-python: opentelemetry-instrumentation/my-instrumentation
     

   • 同时确认Pod所在命名空间的ServiceAccount具备读取opentelemetry-instrumentation命名空间内Instrumentation资源的权限。

3. 复制Instrumentation到Pod所在命名空间

   • 快速验证方案：将my-instrumentation资源复制一份到Pod所在的namespace下，让Operator能直接在同命名空间找到可用实例。

4. 验证Python插桩镜像的可访问性

   • 在Pod所在节点执行docker pull ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.39b0，确认镜像能正常拉取。若无法拉取，检查节点网络或镜像仓库权限。

内容的提问来源于stack exchange，提问作者Dave Jablonski