Nomad周期性作业运行历史持久化方案咨询（POC场景）

Ah, this is a common gotcha when working with Nomad periodic jobs—especially during POCs where you're still ironing out the persistence details. Let's walk through the most reliable solutions to make sure your job run history stays intact even if the Nomad process restarts or terminates:

By default, Nomad clients store job run state in memory, which is why it vanishes when the process stops. Fix this by configuring a persistent state directory:

• Open your Nomad client config file (typically /etc/nomad.d/client.hcl)
• Add or update the client block to specify a persistent filesystem path:

client {
  state_dir = "/var/lib/nomad/client" # Use a path on persistent storage (not tmpfs)
}

• Ensure the Nomad process has read/write permissions to this directory: run chown -R nomad:nomad /var/lib/nomad/client if needed
• Restart the Nomad client service. Job run state will now save to disk and survive process restarts.

Nomad servers can be set up to persist job run history across restarts using a dedicated storage backend. Here's how:

• Edit your Nomad server config file (e.g., /etc/nomad.d/server.hcl)
• Add a job_history block to enable persisted history:

job_history {
  enabled = true
  storage {
    type = "file" # Use "consul" instead if you have a Consul cluster for distributed storage
    path = "/var/lib/nomad/job-history" # Persistent path for file storage
  }
  retention {
    max_age = "720h" # Keep history for 30 days; adjust based on your needs
    max_entries = 10000 # Cap total history entries to avoid storage bloat
  }
}

• Restart the Nomad server. Now you can retrieve historical job runs with nomad job history <job-name> even after server restarts.

For an extra layer of persistence (or to aggregate history across clusters), set up a periodic task to export job history to a durable system:

1. Create a shell script export-nomad-history.sh:

#!/bin/bash
JOB_NAME="your-periodic-job-name"
EXPORT_DIR="/var/nomad/exported-history"
mkdir -p $EXPORT_DIR
nomad job history -json $JOB_NAME > "$EXPORT_DIR/history-$(date +%Y%m%d%H%M%S).json"

2. Make it executable: chmod +x export-nomad-history.sh
3. Create a Nomad periodic job to run this script (e.g., hourly):

job "export-history" {
  type = "batch"
  periodic {
    cron = "0 * * * *" # Run every hour
    prohibit_overlap = true
  }
  group "export" {
    task "export" {
      driver = "exec"
      config {
        command = "/path/to/export-nomad-history.sh"
      }
      volume_mount {
        volume = "history-storage"
        destination = "/var/nomad/exported-history"
        read_only = false
      }
    }
  }
  volume "history-storage" {
    type = "host"
    read_only = false
    source = "/persistent/storage/path/on/host" # Use a persistent host directory
  }
}

4. Submit the job with nomad job run export-history.nomad to auto-export history at your chosen interval.

If you need a detailed record of all job lifecycle events (not just run history), enable Nomad's audit logging:

• Edit your Nomad server config to add an audit block:

audit {
  enabled = true
  log_path = "/var/log/nomad/audit.log" # Persistent log file path
  log_format = "json" # Makes parsing logs easier later
}

• Restart the Nomad server. The audit log will capture every job-related action (submissions, starts, stops, failures) which you can store in a log aggregation system for long-term retention.

内容的提问来源于stack exchange，提问作者rajya vardhan