A Delivery Stream whose Errors Are Separable

aws-kinesis-firehose

A Kinesis Data Firehose delivery stream into S3, with the error handling that decides whether the destination prefix can be trusted. Works with Terraform and OpenTofu (>= 1.6), AWS provider >= 6.0, < 7.0.

Without error_output_prefix, failed records land in the same prefix as good ones. Firehose does not drop what it could not process - a record that failed a transform, or that Firehose could not convert, is written anyway, wrapped in an error envelope, into the destination prefix. Whatever reads that prefix then reads failures as data, and nothing anywhere reports it. Giving errors their own prefix is one line, and it is the difference between a quiet corruption and a directory you can count. This module defaults it on and refuses null unless you say you meant it.

CloudWatch logging is off by default. A stream that cannot assume its role, cannot write to the bucket, or whose Lambda transform is throwing fails silently and retries until the retry window expires. The log group is the only place any of that appears, so log_group_name is likewise refused as null unless stated.

Compression is off by default too, and both Firehose and S3 bill by the byte. GZIP on log-shaped JSON is usually a 5-10x difference for no work, so the default here is GZIP.

Two more things the module handles:

• s3_backup_mode is on whenever a Lambda transform is configured, so the raw record is kept. Without it, the only copy of what arrived is the one the transform could not handle
• server_side_encryption is set for DirectPut streams and omitted for Kinesis-sourced ones, which inherit the source stream's encryption and reject it here

The outputs errors_are_separated, failures_are_visible and raw_records_are_kept exist to be asserted on in a policy test rather than read by a person.

Verification

Static validation runs tofu fmt, init, validate, tflint and checkov. This module has not yet had a live test, so it is published as statically validated with its live test pending and does not carry the live-tested mark.