Cloud-agnostic · polyglot · open source

There was no Spring for data pipelines. So I built one.

Sixteen contracts, two languages, one conformance suite every adapter must pass. Swap the cloud, keep the logic.

Culvert is a framework for pipelines defined once against a language-neutral contract set, then run on any cloud by swapping adapters instead of rewriting logic. Sixteen contracts, in Java and Python. GCP today, AWS alongside it, Azure on the way.

Install it

pip install culvert[gcp]

Install the core → The contract spec →

The idea01

Contracts are the portability boundary

Beam, Airflow and dbt are excellent engines. But an engine is not a framework. Nobody draws the line between the part of your pipeline that is data engineering and the part that is the cloud you happen to be renting this year, so teams weld business logic to one vendor's SDK out of habit, and the thin cloud layer quietly becomes a load-bearing wall.

Culvert draws that line where Spring drew it: a small core of contracts (BlobStore, Warehouse, Source, Sink, JobControlRepository, eleven more). Business logic depends only on the contracts. Adapters implement them for each cloud, and a shared conformance suite that every adapter must pass keeps the promise honest.

Where the line sits

Yours Business
logic

Written once. Depends only on the contracts, never on an SDK.

→↓

The boundary Sixteen
contracts

Java interfaces and Python Protocols. One spec, either runtime.

→↓

Swappable Cloud
adapters

Each one passes the shared conformance suite, or it is not an adapter.

Contracts

Both languages, one spec

16 interfaces plus a metrics record, realised as Java interfaces and Python Protocols. Same seam, either runtime.

Execution

Java owns Beam

The Dataflow/Beam execution layer is Java. Legacy Python Beam is not carried forward; the two languages deliberately don't do the same job.

Transform

dbt, reused

Transformation is SQL and macros, not "Java" or "Python". Packaged once; there is deliberately no Java transform module.

Local-first

Runs on your laptop

The whole stack runs against emulators with no cloud account. Cloud is where you prove and ship, not where you develop.

Install02

Install the core, add the clouds you need

One package, extras for the adapters and roles you want. Core installs with no cloud SDKs at all.

shellPython 3.10 or newer

# core contracts only, no cloud SDKs
pip install culvert
# + BigQuery, GCS, Pub/Sub, Secret Manager, observability
pip install culvert[gcp]
# + Airflow-side DAG factory / operators / sensors, or dbt
pip install culvert[orchestration]
pip install culvert[transform]

pythonauto-discovery

from data_pipeline_core import autoconfig
blob_store = autoconfig.discover().first("blob_store")
# -> GcsBlobStore, discovered via entry points when culvert[gcp] is installed

Contracts03

What each cloud implements

The core contracts and the adapters that satisfy them today. An adapter is only listed once it passes the shared conformance suite.

Contract	GCP	AWS	Azure
`BlobStore`	GCS	S3	roadmap
`Warehouse`	BigQuery	Athena	roadmap
`Source` · `Sink`	Pub/Sub	SQS	roadmap
`JobControlRepository`	Firestore	DynamoDB	roadmap
Observability	Cloud Monitoring	CloudWatch	roadmap
Secrets	Secret Manager	next	roadmap
Eleven more, plus a metrics record	Full implementation	Java adapter family	roadmap

The authoritative list is the spec itself: CONTRACT.md →. Adapters are discovered through entry points, so installing an extra is all the wiring there is.

Status04

What's real, stated plainly

This project has a rule: nothing is announced before it has run on real infrastructure. So here is the honest state, not the roadmap in disguise.

Python: Released: culvert 0.1.0 on PyPI live
Java: Released: com.enrichmeai.culvert:* on Maven Central live
Clouds: GCP: full implementation live · AWS: Java adapter family S3 · Athena · SQS · DynamoDB · CloudWatch · Azure roadmap
Proof: Ran end-to-end on a real GCP project (Cloud Run to BigQuery, event-driven) before release. It caught eight production-only bugs every local test had passed.
Licence: Apache 2.0

The release gate, in one sentence
"1,145 tests passed. The first real deploy failed in eight ways none of them saw. That is why nothing ships here until it has run on real infrastructure."

The book05

The book: building Culvert, honestly

A practitioner's account of designing this framework in the open: twenty-one chapters, in progress. Where the contract boundary sits and why. Why the two languages deliberately don't do the same job. An honest review of my own code. And what building with AI agents actually looks like when they're held to the same contracts and gates as everyone else.

Every technical claim cites the real source tree. It publishes once the 0.x line settles, the same rule the code ships under.

Essay · starting soon

Why is there no Spring for data pipelines?

Every other tier got its framework. The one that moves the data got engines and a habit of welding logic to whichever SDK was in fashion.

Essay · starting soon

1,145 green tests. The first real deploy failed 8 different ways.

What emulators cannot show you, written down as it happened, and why that deploy became the release gate.

Follow along at github.com/enrichmeai/culvert →.

FAQ06

Does this replace Beam, Airflow or dbt?

No. They are engines and Culvert is a framework around them. Beam runs under the Java execution layer, Airflow is wired through the orchestration extra, and dbt is packaged and reused rather than reimplemented.

Why not one language?

Because they do different jobs. Java owns Beam and the execution layer; Python owns the contracts-first pipeline code most teams write. Legacy Python Beam is deliberately not carried forward.

Do I need a cloud account to start?

No. The whole stack runs against emulators on your laptop. Cloud is where you prove and ship, not where you develop.

How portable is portable, really?

As portable as the conformance suite is strict. Every adapter has to pass the same suite, so a contract behaves the same on either cloud or it is not shipped as an adapter. GCP is fully implemented; AWS has the Java adapter family; Azure is roadmap.

Is 0.1.0 safe to build on?

It shipped only after running end-to-end on real infrastructure, which caught eight bugs the local tests never saw. It is a 0.x line, so expect the contracts to firm up before 1.0.

Does Culvert have anything to do with the AI work?

Not in the code. There is no AI in Culvert. It is here because it is where the practice the AI-era projects are held to was proven: contracts, a conformance suite, and a real deploy before a release. The register →