Absolutely! You can definitely use AWS Batch as the executor for all your Airflow jobs—this isn’t just a 2017 roadmap idea anymore; it’s a fully supported, production-grade setup that perfectly solves your low EC2 utilization problem.

Here’s how it works and how to set it up:

How AWS Batch fits your elastic needs #

AWS Batch is built exactly for this kind of on-demand workload execution. When paired with Airflow:

• Airflow submits tasks to a Batch job queue instead of running them on persistent EC2 workers.
• AWS Batch automatically provisions EC2 instances (or uses Fargate for serverless execution) to run tasks as they come in.
• Once tasks finish, Batch terminates the instances immediately—no idle resources sitting at 2% utilization.

Step-by-step setup basics #

1. Prepare your AWS Batch environment

   • Create a Batch compute environment: Choose between EC2 (On-Demand or Spot instances for cost savings) or Fargate. This defines the resources Batch will use to run your tasks.
   • Set up a Batch job queue linked to this compute environment.
   • Create a Batch job definition: This specifies the container image that runs your Airflow tasks. The image needs to include Airflow, your DAG dependencies (Python libraries, CLI tools, etc.), and any code your tasks rely on.

2. Configure Airflow to use Batch Executor

   • First, install the AWS provider package for Airflow:

     pip install apache-airflow-providers-amazon
     

   • Update your airflow.cfg file to use the Batch Executor and link to your Batch resources:

     executor = BatchExecutor
     aws_batch_job_queue = your-batch-job-queue-name
     aws_batch_job_definition = your-batch-job-definition-name
     aws_region_name = your-aws-region
     

3. Set up IAM permissions

   • The IAM role used by your Airflow scheduler needs permissions to submit jobs to AWS Batch (e.g., batch:SubmitJob, batch:DescribeJobs).
   • The IAM role attached to your Batch compute environment needs permissions for any resources your tasks use (like S3 access for data, CloudWatch Logs for logging).

Key benefits for your use case #

• Zero idle resources: You only pay for compute time when tasks are actually running—no more wasting money on underutilized EC2 instances.
• Built-in scalability: Batch handles scaling up/down automatically based on pending tasks, so you never have to manually adjust worker counts.
• Cost flexibility: Use Spot instances in your Batch compute environment to cut costs by up to 90% compared to On-Demand, with Batch handling instance interruptions gracefully.
• Less maintenance: No need to manage persistent Airflow worker EC2 instances—Batch takes care of all infrastructure lifecycle tasks.

Things to keep in mind #

• Container image consistency: Ensure your Batch job definition’s image matches the Airflow environment your DAGs are written for (same Python version, dependency versions) to avoid runtime errors.
• Logging setup: Configure your Batch job definition to send task logs to CloudWatch Logs—this lets you view task logs directly from the Airflow UI.
• Task resource allocation: Set appropriate vCPU/memory limits in your job definition to match your tasks’ needs—this ensures Batch provisions the right resources and avoids wasted capacity.

This setup is widely used in production for Airflow on AWS, and it’s exactly the elastic architecture you’re looking for.

内容的提问来源于stack exchange，提问作者romain-nio