Managed Spark across clouds
Managed Spark and Hadoop - EMR Serverless, HDInsight, Dataproc, Databricks on Azure - with the cluster in your network, the metastore outside it, and the bill understood before the first job.
4 verified modules, 1 of them live-tested apply→verify→destroy; the rest are static-validated, live-test pending.
Compare by provider
| Provider | Module | Verification |
|---|---|---|
| AWS | A Spark Application that Runs Inside Your VPC | static-validated |
| Azure | A Databricks Workspace in Your VNet with No Public IPs | static-validated |
| Azure | An HDInsight Spark Cluster Behind a Private Link with No Storage Key in State | static-validated |
| Google Cloud | Dataproc Single-Node Cluster | ✓ live-tested |
How to choose
Decide first whether you want a cluster or a job: EMR Serverless and Dataproc Serverless bill per job and idle at zero, HDInsight and a classic Dataproc cluster bill by the hour from creation, and Databricks is a platform with its own workspace and pricing on top of the VMs. Then compare where the metastore lives, whether the endpoints can be private, and what the storage identity is (a key in state, or a role).
When not to use
The cluster is not the data platform. Table definitions in an internal metastore die with the cluster, and a Spark cluster that is never deleted because the metastore is in it is the most expensive way to keep a schema.