Managed ETL Pipelines across clouds
Serverless data integration - Glue, Data Factory, Dataflow - with jobs that run as a scoped identity and pipelines that alert when they fail.
3 verified modules - all static-validated and publish-checked; live-test pending.
Compare by provider
| Provider | Module | Verification |
|---|---|---|
| AWS | Glue with Encrypted Job Bookmarks | static-validated |
| Azure | A Data Factory that Moves Data Privately and Keeps Pipelines in Git | static-validated |
| Google Cloud | A Dataflow Job on Private Workers that Drains on Destroy | static-validated |
How to choose
Compare the programming model (a visual pipeline, Spark, Beam), what a job runs as and whether it can reach the internet, and how failures surface: a pipeline that fails silently at 03:00 is the default on every one of these until an alert is wired.
When not to use
A pipeline copies data and with it every access problem the data had. Encrypt the intermediate storage, scope the job identity to its sources and sinks, and expect the service account to become the most privileged identity in the project.
Other solutions
Managed Relational Database25Load Balancer19Secrets & Key Management19Virtual Private Cloud (VPC)18Messaging & Pub/Sub17Cloud Firewall17Object Storage16Block Storage16Managed Kubernetes15Virtual Machines15DNS & Traffic Management13Serverless Containers8Container Registry7Managed CI/CD7Managed NoSQL Database7WAF & Edge Security6Serverless Functions5API Gateway5Identity & Access5Monitoring & Observability5Network Peering5CDN & Edge Delivery4In-Memory Cache4Shared File Storage4Budgets & Cost Alerts4Site-to-Site VPN4Private Certificate Authority4Security Posture & Findings4Dedicated Interconnect4Managed Search4Private Instance Access4Managed ML Platform4Organisation Guardrails4Transit Hub4NAT Gateway4Audit Log Retention4Managed Spark4Database Migration & CDC3Managed Network Firewall3Backup Service3Static Site Hosting3Managed TLS Certificates3Private Service Endpoints3Workflow Orchestration3Patch Management3Data Transfer Service3Managed Active Directory3Transactional Email3Golden Images3HPC Parallel File Systems3Managed App Platform3Cloud Data Warehouse3