Contracts are the portability boundary
Engines like Beam, Airflow and dbt are magnificent — but an engine is not a framework. Nobody had drawn the line that says this part of your pipeline is about data engineering and that part is about the cloud you happen to be renting this year. So every team welds its business logic to one vendor's SDK by habit, and the thin cloud layer quietly becomes a load-bearing wall.
Culvert draws that line where Spring drew its: a small, honest core of contracts — BlobStore, Warehouse, Source, Sink, JobControlRepository, and eleven more. Business logic depends only on the contracts. Adapters implement them per cloud, and a shared conformance suite every adapter must pass keeps the promise honest.
Both languages, one spec
16 interfaces + a metrics record, realised as Java interfaces and Python Protocols. Same seam, either runtime.
Java owns Beam
The Dataflow/Beam execution layer is Java. Legacy Python Beam is not carried forward — the languages don't do the same job.
dbt, reused
Transformation is SQL + macros, not “Java” or “Python”. Packaged once; there is deliberately no Java transform module.
Runs on your laptop
The whole stack runs against emulators with no cloud account. Cloud is where you prove and ship, not where you develop.
Install the core, add the clouds you need
One package, extras for the adapters and roles you want. Core installs with no cloud SDKs at all.
# core contracts only — no cloud SDKs pip install culvert # + BigQuery, GCS, Pub/Sub, Secret Manager, observability pip install culvert[gcp] # + Airflow-side DAG factory / operators / sensors, or dbt pip install culvert[orchestration] pip install culvert[transform]
from data_pipeline_core import autoconfig blob_store = autoconfig.discover().first("blob_store") # -> GcsBlobStore, discovered via entry points when culvert[gcp] is installed
What's real — stated plainly
This project has a rule: nothing is announced before it has run on real infrastructure. So here is the honest state, not the roadmap in disguise.
culvert 0.1.0 on PyPI livecom.enrichmeai.culvert:* publish pending“The Java libraries aren't on Maven Central yet — and this page tells you so. That honesty is the point, not a caveat.”