OpenShift容器统一使用UID 101而非项目范围UID的问题排查及解决

Issue Overview #

After deploying OpenShift Container Platform 4.5.14 on AWS, we hit a critical problem: every pod across all projects was running with UID 101, instead of the project-specific UIDs that OpenShift normally assigns.

This caused permission failures when using images that already map UID 101 to a built-in user (like systemd-timesync in some base images). In those cases, the pod would run with GID 101 instead of the expected GID 0, making it impossible to write to directories permissioned for the root group.

How We Verified the Problem #

First, we spun up a test pod with a simple busybox image to confirm the UID issue:

oc run -it -n knative-serving --image busybox test1 sh

Once inside the pod, running id returned:

uid=101(101) gid=0(root) groups=1000600000

To show the permission failure scenario, we used an internal image that already has UID 101 mapped to systemd-timesync:

oc run -it -n kaiburr-app --image registry.kaiburr.com/dynamic-analysis-worker test3 sh

Inside this pod:

# Check user identity
id
uid=101(systemd-timesync) gid=101(systemd-timesync) groups=101(systemd-timesync),1000630000

# Check directory permissions
ls -ld
drwxrwxr-x. 1 zap root 4096 Oct 19 19:24 .

# Try to create a file (fails due to GID mismatch)
touch test
touch: cannot touch 'test': Permission denied

Our expected behavior was that pods would use project-scoped UIDs and run with GID 0, which is OpenShift's standard behavior.

Root Cause Analysis #

After digging into OpenShift's Security Context Constraints (SCCs), we found the culprit: the nginx-ingress-scc created by the nginx-ingress-operator.

This SCC had two problematic configurations:

1. It forced all users in the system:authenticated group (which includes all regular project users) to run as UID 101:

   runAsUser:
     type: MustRunAs
     uid: 101
   

2. Its priority was set to 20, which is higher than the default anyuid SCC's priority of 10. Since SCCs are evaluated from highest to lowest priority, this SCC was overriding the project-specific UID assignment logic.

You can inspect the full SCC details with:

oc get scc nginx-ingress-scc -o yaml

Fix Steps #

We first tried modifying the uid value in the SCC to 102, but that just forced all pods to run as UID 102 instead—hardly a real solution.

The correct fix was to lower the priority of nginx-ingress-scc so it doesn't take precedence over the default SCCs that handle project-scoped UIDs:

1. Edit the SCC:

   oc edit scc nginx-ingress-scc
   

2. Locate the priority field (default value 20) and change it to a number below 10—we used 5:

   priority: 5
   

3. Save the changes and exit the editor.

After making this adjustment, any new pods deployed will automatically use the project-scoped UIDs as expected, and the permission issues with images containing UID 101 will be resolved.

内容的提问来源于stack exchange，提问作者Oleg Khaschansky