Use a Contract Registry

A contract normally comes from a file the job carries with it — but a file has no shared place other jobs, CI pipelines, or platform tooling can look up, version, or govern independently of any one job’s own deployment. registry://<id>@<version> lets spark.invaract.contract name a contract stored in a contract registry — a small HTTP service that stores versions of a contract, computes a compatibility diff against the previous version on every new registration, and rejects a registration whose declared version bump doesn’t match the actual scope of change unless explicitly overridden.

Every engine

A registry:// contract works the same on Spark and on Beam. On Beam, pass it as --invaractContract with --invaractRegistryUrl (Java), or --invaract_contract with --invaract_registry_url (Python); see Configuration Settings.

The reference server is closed-source

Everything on this page other than the server itself is open, Apache 2.0, in this repo: the wire protocol, and the registry-client library that speaks it. The reference server implementation (invaract-registry) is a separate, private repository — its source isn’t published, not something you can git clone. Its own documentation, however, is public — see the Invaract Registry docs for installation, configuration, the full API reference, and known limitations of the reference implementation. Run your own implementation of the documented wire protocol, or reach out about access to the reference one; the client and the registry:// resolution mechanism work against either.

What you need running

A registry server, reachable over HTTP from wherever your job runs, implementing the wire protocol (routes, request/response shapes, status codes). Nothing else in this guide, or in spark-adapter’s resolution mechanism, assumes a specific server implementation — registry-client only speaks the documented HTTP protocol.

Register a contract

1. Write an ordinary contract, the same as any other — see Write a Contract.

2. PUT it to the registry:

   curl -X PUT http://localhost:7000/contracts/customer_orders/versions/1.0.0 \
     -H "Content-Type: application/yaml" \
     --data-binary @customer_orders.yaml

   A first-ever version returns 201 with an empty compatibility report:

   {"changes":[]}

3. Registering a later version reuses contract’s own ContractCompatibility diff — the same logic Check Contract Compatibility describes — to reject a version bump that doesn’t match the actual scope of change (?expectedPrevious=1.0.0 for the optimistic-concurrency check, ?force=true to override a rejected bump).

Point a job at it, purely via spark-submit --conf

No change to a job’s own source is needed — the same attachability guarantee every other spark.invaract.* conf key in this documentation gives you:

spark-submit \
  --class com.example.MyRealJob \
  --master yarn \
  --conf spark.sql.extensions=com.invaract.sparkadapter.InvaractSparkSessionExtension \
  --conf spark.invaract.contract=registry://customer_orders@2.1.0 \
  --conf spark.invaract.registryUrl=https://registry.corp.internal \
  --jars invaract-spark-adapter-0.5.0.jar,invaract-registry-client-0.1.0.jar \
  my-job.jar

registry://customer_orders@2.1.0 names an exact version; registry://customer_orders@latest resolves to whatever the registry currently considers the latest registered version for that id. Both parts are needed: a reference without a version, such as registry://customer_orders, is refused when the job starts with 'registry://customer_orders' is not a valid contract reference: write it as 'registry://<id>@<version>' (the version may be 'latest').

invaract-registry-client is only needed via --jars, not compiled in

spark-adapter has no compile-time dependency on registry-client — resolving a registry:// reference loads it reflectively, by class name, only once a job actually uses one. A job that never sets spark.invaract.contract to a registry:// value never needs invaract-registry-client-*.jar on its classpath at all, the same way notification-kafka’s jar is only needed by a job that configures a Kafka notification sink.

Need a resolver --conf can’t express?

Call registry-client directly in code, the same “conf path required, code path is a bonus” composition every other resolution mechanism in this documentation follows:

import com.invaract.registryclient.HttpContractRegistryClient
import com.invaract.sparkadapter.ContractEnforcementRule

val client = new HttpContractRegistryClient()
client.configure("https://registry.corp.internal")
val contract = client.getLatest("customer_orders")

val spark = SparkSession.builder()
  .withExtensions(_.injectCheckRule(ContractEnforcementRule.forContract(contract)))
  .getOrCreate()

Trying it against the demo job

./dev/registry-demo

Needs a local clone of the reference server implementation next to this repository (or INVARACT_REGISTRY_DIR pointing at one elsewhere) — see the caution above on why that repo isn’t publicly cloneable; this script is for anyone with access to it, or an equivalent implementation of the documented wire protocol at that path. See the Invaract Registry docs for how to build and run one from a checkout you do have access to. The script starts a real registry server, registers demo/contracts/invaract_output_registry.yaml into it over real HTTP, then runs DemoJobHarness against registry://invaract_demo_output_registry@1.0.0 — confirmed against a real run, not just described. Its contractVerification section:

{
  "status": "PASSED",
  "contract": "invaract_demo_output_registry@1.0.0",
  "contractPath": "registry://invaract_demo_output_registry@1.0.0",
  "violations": []
}

contractPath staying the literal registry://... string, rather than a resolved file path, is what confirms the run went through registry resolution end to end.

What’s next

A persistent registry storage backend (the reference implementation is in-memory, not durable across restarts), real SSO/OIDC auth, and cross-contract policies or fingerprint-based drift control consuming the registry are all named, disclosed future work — see the wire protocol’s “Non-goals” section and the Invaract Registry docs’ Known Limitations page for the current state of each.