如何实现EFS/NFS目录独占访问？多容器共享存储写入管控方案探讨

Great question—this is a super common pain point when dealing with shared EFS volumes across containers, especially since NFS's built-in locking falls short for your strict reliability requirements. Let’s break down practical, low-refactor solutions that avoid the pitfalls you’ve outlined.

核心需求回顾 #

First, let’s align on what we’re solving for:

• Multiple containers share an EFS volume, each targeting a specific directory
• Exclusive write access: Only one container can modify a directory at any time
• Avoid NFS lock flaws (locks release on client disconnect, leading to race conditions if the client reconnects)
• Priority: No application code refactoring required

方案1：分布式锁Sidecar代理（最可靠，无应用改动） #

This is my top recommendation if you’re running on Kubernetes (or any orchestrator that supports sidecar containers). The idea is to pair every business container with a dedicated sidecar that handles lock management, so your app code stays untouched.

How it works #

• Sidecar + Business Container Binding: Deploy the sidecar in the same Pod as your business container—they share network/volume access, so the sidecar can tightly control the business container’s lifecycle.
• Distributed Lock Backend: Use a reliable KV store like etcd or Redis (with Redlock for multi-node redundancy) to track directory locks. Each lock is tied to a directory path and has a short TTL (e.g., 10 seconds).
• Lock Acquisition Flow:
  1. When the Pod starts, the sidecar first attempts to acquire the lock for the target directory.
  2. If successful, the sidecar launches the business container’s main process.
  3. The sidecar sends periodic heartbeats to the KV store to renew the lock’s TTL (e.g., every 3 seconds).
• Failure Handling:
  • If the sidecar loses connection to the KV store, it kills the business container immediately (via shared PID namespace) to prevent unprotected writes.
  • If the business container crashes, the sidecar explicitly releases the lock.
  • Kubernetes liveness/readiness probes monitor both containers—if either fails, the Pod is restarted, ensuring no orphaned locks or rogue writers.

Why no app refactoring? #

You can use an init container or sidecar entrypoint script to intercept the business container’s startup. For example, the sidecar runs a script that:

# Attempt to acquire lock for /efs/my-directory
if ! redis-cli set "lock:/efs/my-directory" "$POD_ID" NX EX 10; then
  echo "Lock held by another container—exiting"
  exit 1
fi

# Renew lock in background while app runs
while true; do
  redis-cli set "lock:/efs/my-directory" "$POD_ID" XX EX 10
  sleep 3
done &

# Launch the app's original command
exec "$@"

Just pass your app’s command as arguments to the sidecar, and it handles all lock logic.

方案2：EFS Access Points + 分布式锁（双重保障） #

Combine EFS’s native Access Points with distributed locking to add a permission layer on top of lock logic. This is great for enforcing read-only access for non-locked containers.

How it works #

• Per-Directory Access Points: Create two EFS Access Points for each target directory: one with read-write permissions, another with read-only.
• Lock-Driven Access: When a container starts, it first tries to acquire the distributed lock. If successful, it mounts the read-write Access Point; if not, it mounts the read-only one.
• Benefits: Even if a lock acquisition fails due to a race condition, the container can’t modify the directory—adding a safety net beyond just lock logic.

方案3：改进型文件锁（无额外依赖） #

If you don’t want to run a dedicated KV store, you can implement a file-based lock that fixes NFS’s native lock flaws with a lease mechanism.

How it works #

Instead of relying on NFS’s flock or fcntl, use atomic directory creation (mkdir is atomic on NFS) to implement locks:

1. To acquire a lock for /efs/my-directory, attempt to create /efs/my-directory/.lock/<pod-id>.
2. If the directory is created successfully, you have the lock.
3. Periodically update the directory’s modification time (e.g., every 10 seconds) to signal you’re still active.
4. To check if a lock is valid, other containers check the modification time of the lock directory. If it’s older than your TTL (e.g., 30 seconds), they delete the old lock directory and create their own.

Key fixes for NFS lock issues #

• No automatic lock release on disconnect: If a container dies, its lock directory will eventually expire, and another container can take over.
• No race conditions on reconnect: If a disconnected container comes back, it’ll see the new lock directory and exit instead of writing.

Example script (no app changes) #

Add this to your container’s entrypoint:

LOCK_DIR="/efs/my-directory/.lock/$POD_ID"
TTL=30

# Try to acquire lock
if ! mkdir "$LOCK_DIR"; then
  # Check if existing lock is expired
  OLD_LOCK=$(ls -dt /efs/my-directory/.lock/* | head -n1)
  if [ $(($(date +%s) - $(stat -c %Y "$OLD_LOCK"))) -gt $TTL ]; then
    rm -rf "$OLD_LOCK"
    mkdir "$LOCK_DIR" || { echo "Failed to acquire lock—exiting"; exit 1; }
  else
    echo "Lock held by active container—exiting"
    exit 1
  fi
fi

# Renew lock in background
while true; do
  touch "$LOCK_DIR"
  sleep 10
done &

# Launch app
exec "$@"

处理检测/终止延迟 #

All these schemes handle latency with intentional guardrails:

• TTL Tuning: Set lock TTLs to a value that balances responsiveness and safety (10-30 seconds is standard). Shorter TTLs mean faster lock release after failure, but require more frequent heartbeats/updates.
• Orchestrator Integration: Use Kubernetes liveness probes to detect unresponsive containers and restart them quickly. For example, a probe that checks if the sidecar’s heartbeat process is running, or if the app’s write endpoint is responsive.
• Explicit Cleanup: Sidecars and entrypoint scripts explicitly release locks when the app exits normally, avoiding orphaned locks even if there’s a small delay in detecting failures.

总结推荐 #

• Best for Reliability: Sidecar + Distributed Lock (etcd/Redis) with Kubernetes orchestration. It’s the most robust, handles edge cases like network partitions, and requires zero app changes.
• Best for Simplicity: Improved File Lock. No extra services to manage, works with any orchestrator or bare-metal setup.
• Extra Safety: Add EFS Access Points to either scheme to enforce read-only access for non-locked containers.

内容的提问来源于stack exchange，提问作者kassie