A Spark Application that Runs Inside Your VPC

aws-emr-serverless

An EMR Serverless application that runs inside your VPC, keeps its logs, and has a ceiling. Works with Terraform and OpenTofu (>= 1.6), AWS provider >= 6.0, < 7.0.

Without network_configuration the application runs on AWS-managed networking, and that means two things at once: it cannot reach anything private - not a database in your VPC, not a private Kafka, not an interface endpoint - and its outbound traffic does not pass through your routing, your NAT or your DNS firewall. Jobs that process your data then run somewhere you do not inspect. A precondition refuses it unless accept_aws_managed_network says it was meant, and a second one refuses half a network configuration, since subnets without security groups is not a partial answer but a broken one.

Monitoring is off by default. No monitoring_configuration means the driver and executor logs of a failed job go nowhere - and a Spark failure is diagnosed from executor logs, not from the job status. On here, with a KMS key, because the logs of a job that reads your data contain parts of your data.

maximum_capacity absent means the account service quota is the limit. A job with a bad join scales until it reaches that, and the first symptom is the bill. An explicit ceiling costs nothing to set and is the only thing that bounds it.

Pre-initialised capacity is deliberately not configured. initial_capacity keeps workers warm so the next job starts immediately, and bills for them whether or not a job ever arrives. That is a real trade-off rather than a default, so the module leaves it off and says so here instead of setting it quietly.

Smaller things: ARM64 rather than AWS's X86_64, because Graviton is materially cheaper for Spark on the same managed runtime image; auto_stop on, with a precondition that refuses an always-warm application unless accept_always_on states it; and the CloudWatch block's enabled is a plain variable rather than a derived expression, so a configuration scanner can actually read it.

Verification

Static validation runs tofu fmt, init, validate, tflint and checkov. This module has not yet had a live test, so it is published as statically validated with its live test pending and does not carry the live-tested mark.