HiveMR3 on Kubernetes：Worker Pod 卡在Init:0/1状态问题求助

Hey there, let's figure out why your HiveMR3 Worker Pod is stuck in the Init:0/1 state. Since your cluster is otherwise healthy (you can create databases/tables without issues), the problem is almost certainly tied to the init container that runs before the main Worker container starts. Here's a step-by-step breakdown to diagnose and fix this:

1. Check the Init Container's Logs First #

The most direct way to uncover the root cause is to inspect the logs from the init container. First, gather the necessary details:

• List all Worker Pods to get your target pod name: kubectl get pods -l <worker-pod-label> (replace <worker-pod-label> with the label used for HiveMR3 Worker pods, e.g., app=hivemr3-worker)
• Find the init container name: Run kubectl describe pod <worker-pod-name> and look for the Init Containers section (usually something like init-worker)

Then pull the logs for that init container:

kubectl logs <worker-pod-name> -c <init-container-name>

This will show you exactly where initialization is failing—whether it's a missing file, permission error, network timeout, or invalid command.

2. Verify Init Container Resource Allocation #

Sometimes the init container can't start because the Kubernetes node lacks available CPU or memory.

• Check the node's resource usage: kubectl top node <node-name> (replace <node-name> with the node running your Worker Pod)
• Review the pod's resource requests/limits in your HiveMR3 Kubernetes manifests. If the init container is requesting more resources than the node can provide, it'll hang waiting for available capacity. You may need to adjust the resources section for the init container or add more resources to the node.

3. Check Image Pull Status #

If the init container can't pull its Docker image, it'll get stuck in the init state. Inspect the pod's events to confirm this:

kubectl describe pod <worker-pod-name> | grep -A 10 Events

If you see ImagePullBackOff or ErrImagePull, this indicates:

• The image name/tag in your manifest is incorrect
• The image isn't available in your container registry
• The pod lacks credentials to access the registry (check if you need an ImagePullSecret configured)

4. Validate HiveMR3 Configuration & Volume Mounts #

HiveMR3 Worker pods depend on specific config files and storage volumes to initialize properly:

• Confirm volumes like HDFS mounts, config maps (e.g., mr3-site.xml, hive-site.xml) are correctly mounted in the init container. Check the Volumes and VolumeMounts sections in kubectl describe pod <worker-pod-name>.
• Verify the init script (usually defined in mr3.worker.init.script in mr3-site.xml) is accessible and has execute permissions. If the script tries to run missing commands or can't access required files, the init container will fail.

5. Check Kubernetes Permissions & Network Policies #

• Ensure the service account assigned to the Worker pod has sufficient permissions to perform initialization tasks (e.g., accessing cluster services, reading config maps).
• If you have network policies enabled, make sure they don't block the init container from accessing necessary resources like your container registry, HDFS namenode, or other cluster services.

Quick Fixes to Try #

• If logs show a permission error: Adjust the permissions on mounted volumes or the init script (e.g., add chmod +x to the script in your manifest).
• If resource limits are the issue: Lower the resource requests for the init container, or scale up your Kubernetes nodes.
• If image pull fails: Double-check the image path, confirm the registry is reachable, and add an ImagePullSecret if required.

内容的提问来源于stack exchange，提问作者Elpis