Quick Start

This walks through the fastest path from a fresh clone to a real, contract-verified Spark write — using the example job this repository ships with. It assumes you’ve completed Installation.

1. Run the test harness

   From the repository root:

   ./dev/test

   This single command:

   1. Builds every module in dependency order: the engine (contract, ir, fingerprint, verification-core, spark-adapter, beam-adapter), the conformance kit, and the demo’s own plugin and runner.
   2. Runs a real Spark job — DemoJobHarness — via actual spark-submit, with Invaract’s ContractEnforcementRule installed in the SparkSession.
   3. The job reads demo/input/sample.csv, runs a small example transformation (InvaractPlugin, which adds two computed columns — value_squared and value_tier), and writes the result to demo/output/result.parquet — a write that Invaract verifies against demo/contracts/invaract_output.yaml before it’s allowed to happen.
   4. Generates and validates demo/output/report.json.

   Expect output ending like this (from a real run; the duration varies):

   Step 7/7: Validating execution report...
     Status: PASS
     Plugin Version: 0.1.0
     Duration: 13660ms
     Notification events: 3 line(s) in demo/output/events.jsonl (CONTRACT_VALIDATION, WRITE, and JOB_SUMMARY all present)
   
   ✓ All validation passed
   ✓ Plugin JAR is ready: plugin/target/scala-2.12/invaract-spark-plugin-0.2.0.jar
   ✓ Execution report: demo/output/report.json

   Exit code 0 means the engine actually verified a real Spark write — not just that the code compiled.

2. View the result

   Start the results UI:

   ./dev/report

   Open http://localhost:3000. In Codespaces, the port is auto-forwarded — open the forwarded URL, including on your phone, since the UI is responsive down to 375px screens.

   You’ll see:

   • A PASS status badge
   • Build info (plugin/Spark/Java versions, duration)
   • Input/output schema and sample rows
   • The translated Transformation IR — the actual Spark plan Invaract checked
   • Contract verification: PASSED (invaract_demo_output@1.0.0), no violations

3. See it actually block a bad write

   Everything so far proves a passing write goes through. To see Invaract reject a violating one:

   ./dev/regression

   This runs ten real jobs in pairs: each pair has one job that keeps to its contract (it passes and its output file is created) and one that breaks it. For example, demo/contracts/invaract_output_broken_example.yaml requires a customer_name column the demo transformation never produces. Each breaking job exits non-zero, and its output file is never created — the write was aborted before Spark touched disk. See Prove Enforcement with the Regression Pack for the full walkthrough and output.

Tip

Don’t have a local Spark/sbt setup handy? ./dev/regression-docker runs the same pass/fail proof inside a self-contained Docker image — nothing but Docker required.

What’s next

• Your First Contract — change the contract yourself and watch verification react.
• Infer a Starting Contract with Dry-Run Mode — run ./dev/dry-run to get a contract printed from a real run, instead of writing one from a blank file.
• Write a Contract — the full format, from scratch.
• Install the Enforcement Rule on Spark — wire Invaract into your own Spark job, not just the demo.
• Use Invaract with Apache Beam or with Beam Python pipelines — the same contracts, checked before a pipeline runs.