Batch Jobs that Survive a Reclaimed Host

aws-batch

An AWS Batch compute environment, its queues and its job definitions. Works with Terraform and OpenTofu (>= 1.6), AWS provider >= 6.0, < 7.0.

A job definition with no retry_strategy loses the job on the first transient failure. Batch does not retry by default - attempts is 1 - so a spot instance reclaimed mid-run, a registry that timed out on the image pull, or a node that failed its health check all end as FAILED. The job is reported as having failed on its merits, which is the worst version, because somebody then debugs the code. This module defaults to 3 attempts with evaluate_on_exit rules that retry infrastructure failures and exit on any genuine non-zero code - so a broken job still fails fast instead of burning its attempts.

privileged is root on the host. It shares the Docker socket and the host namespaces, so a job that has it can read every other job's data on that instance and reach the instance metadata service as the node role. Batch runs other people's containers by design, which is exactly why this is refused unless stated, and why privileged_job_definitions is an output.

Environment variables are readable through DescribeJobDefinition by anyone holding batch:Describe*. secrets takes a Secrets Manager or Parameter Store ARN instead, resolved by the agent at launch. job_definitions_with_plaintext_environment names the ones worth checking.

Three more defaults chosen against AWS's:

• runnable_timeout_minutes on every queue. A job stuck in RUNNABLE is the commonest Batch failure - no capacity, no route out of the subnet, or a resource request nothing in the environment can satisfy - and by default it waits forever with no signal at all
• terminate_jobs_on_update = false, so changing the compute environment drains rather than killing work that has been running for an hour
• allocation_strategy = BEST_FIT_PROGRESSIVE. Plain BEST_FIT can wedge a queue: it will not switch instance type when the chosen one has no capacity

max_vcpus is the only thing bounding what a runaway queue costs, so it is a first-class variable and an output.

Verification

Static validation runs tofu fmt, init, validate, tflint and checkov. This module has not yet had a live test, so it is published as statically validated with its live test pending and does not carry the live-tested mark.