Common Problems

./dev/test exits non-zero

Work through this in order:

1. Check the console output — the script names which of its 7 steps failed.
2. Review the report:

   cat demo/output/report.json | jq .

   Look at status, contractVerification (is this an engine rejection, or something else?), and error.
3. Check the web UI (./dev/report) for the Contract Verification and Plugin Events sections.
4. Narrow down where the problem is:
   • Verification/translation issue → run cd spark-adapter && sbt test before assuming it’s the demo transformation.
   • A MISSING_OUTPUT_FIELD/OUTPUT_FIELD_TYPE_MISMATCH/similar violation → your contract and your job’s actual output disagree; see Reference → Violation Types.
   • UNVERIFIABLE_WRITE → your job writes via a shape Invaract doesn’t recognize; see below.

A write is rejected with UNVERIFIABLE_WRITE

This means Invaract found a command that changes data but couldn’t translate it into a recognized write — so it refused to guess, and rejected it. This is deliberate; see Fail-Closed by Default.

Check whether your write goes through a shape listed in Reference → Connector Support. If it doesn’t (a DataSourceV2 catalog write, or MERGE/UPDATE/DELETE outside the connectors this covers), support for it may not exist yet.

On Beam, the same violation names the step that ends the pipeline without a recognized write. If that step only reports (it sends metrics or a notification, and writes no governed data), a platform can let it through with the trusted steps setting.

A rule check is rejected with RULE_UNVERIFIABLE_DML

Your contract declares a DML rule (merge_condition, forbid_unconditional_delete, or allowed_update_columns) against an operation Invaract recognizes as real DML but can’t extract the specific fact that rule needs from — today, this is known to happen for Iceberg’s merge-on-read UPDATE. See Enforce Row-Level DML Rules for exactly which combinations are covered per connector.

InaccessibleObjectException running sbt test directly on JDK 17+

Caution

Spark reflectively accesses JDK-internal classes (sun.nio.ch.DirectBuffer, via org.apache.spark.storage.StorageUtils) that JDK 17+‘s module system closes by default.

spark-submit’s own launch scripts inject the required --add-opens flags automatically, which is why ./dev/test’s primary path needs no changes. If you invoke a module’s tests directly (sbt test inside spark-adapter or plugin) or use dev/test’s java -cp fallback path (used when spark-submit isn’t on PATH), those flags are already set in spark-adapter/build.sbt/plugin/build.sbt/dev/lib.sh — if you’re invoking Spark some other way and hit this error, add the same flag set yourself (see dev/lib.sh for the full list).

Contract parses but is rejected as invalid

A document can be syntactically valid YAML and still fail ContractValidator — missing outputs, a duplicate field name, an empty schema. Check INVALID_CONTRACT in your violations list, and see Reference → Contract Format for the full list of checks.

Something in the contract seems to be ignored

Invaract reads only the keys it knows and keeps the rest, because a contract may carry anything the Open Data Contract Standard or your own team defines. A misspelt key is therefore not an error: nulable: false is kept and ignored, and the field stays nullable. When a job starts with a contract, Invaract logs a warning for each key spelt close to one it reads, and for each other thing the contract validator warns about, such as a field type or format it doesn’t recognize:

WARN AttachSetup$: Invaract: contract orders@1.0.0: [Warning] outputs[0].schema.fields[0].nulable: Unknown key 'nulable' is ignored. Did you mean 'nullable'?
WARN AttachSetup$: Invaract: contract orders@1.0.0: [Warning] outputs[0].format: format 'parqet' is not one of the canonical formats (avro, bigquery, csv, delta, hive, hudi, iceberg, jdbc, json, kafka, orc, parquet, table, text); it can only match an adapter that reports exactly this string, so the same contract may fail on another engine. Did you mean 'parquet'?

Search the job’s log for Invaract: contract to find them. A contract fetched from a registry is checked the same way, except for misspelt keys: those are found only in a contract read from a file.

A setting is refused with “must be true or false”

Every setting that switches something on (dryRun, rejectUndeclaredFields, computeFingerprint, …) takes true or false, in any case. Anything else, such as yes or 1, is refused when the job starts, naming the setting:

'spark.invaract.dryRun' must be true or false, not 'yes'.

A notification config’s sink.enabled is held to the same rule, and the message names the file. A value Invaract can’t read is never quietly taken to mean false.

A location doesn’t match even though it “looks right”

A declared location matches the place the job reads or writes when it is that place, or the end of it starting at a /; slashes, a file: prefix and s3a:///s3:// don’t matter. See Reference → Contract Format for the rule and examples. The usual reasons it still doesn’t match:

• Case. Orders doesn’t match /data/orders.
• A full path against a cluster URL. /data/orders doesn’t match hdfs://nn:8020/data/orders: write data/orders, or the whole URL.
• A dotted table name. gold.orders is one segment and never matches the table. Write gold/orders (see Tables).
• A partition’s folder. A job that reads /data/orders/date=2024-01-01 reads that folder, not orders.

The violation names the location the job actually used, so compare the two from there.

ClassNotFoundException: ...SupportsNonDeterministicExpression installing the enforcement rule

This means you’re running a Spark version below this project’s verified floor. Invaract’s Delta Lake support depends on delta-spark 3.3.3, which requires a class (SupportsNonDeterministicExpression) that Spark only introduced in 3.5.6 — on an older 3.5.x patch (3.5.1–3.5.5), installing the enforcement rule into a Delta-extended SparkSession throws this exception before your job’s own code runs at all.

Upgrade to Spark 3.5.6 or later. See Reference → Spark Version Support for the exact verified range, and — if you’re on a version recent enough to also install ContractEnforcementRule’s own runtime compatibility guard — check your job’s logs for an Invaract is running against Spark ... warning naming the version it actually detected, which points at the same page.

does not support truncate in batch mode / TableCapabilityCheck writing to Delta

This is a Delta Lake/Spark version mismatch, not a bug in your job or your contract. delta-spark releases before 3.3.3 (3.3.0–3.3.2, and the entire 3.2.x line) were built against an older Spark line than this project’s supported Spark floor — running one of those delta-spark releases against a newer Spark session hits exactly this TableCapabilityCheck failure on an ordinary Delta write. There is currently no delta-spark release below 3.3.3 that works with this project’s supported Spark range at all.

Pin delta-spark to 3.3.3. See Reference → Connector Support for the current verified Delta version, and (same as above) check for an Invaract is running against Delta Lake ... warning in your job’s logs if you’re on a version new enough for the runtime compatibility guard to have run.

Dry-run reports SKIPPED_UNRECOGNIZED for a write that works fine

With a notification sink configured, dry-run mode reports every write-shaped command it couldn’t infer a contract from. Running this through spark-sql is a real example:

INSERT OVERWRITE DIRECTORY '/data/out/orders_doubled' USING parquet
SELECT id, value * 2 AS doubled FROM orders;

"'InsertIntoDataSourceDirCommand' may write data but Invaract has no translation for it; with a contract active, enforcement would reject it as an UNVERIFIABLE_WRITE"

The write itself works. The point is what enforcement would do with it: Invaract has no translation for that command, so with a contract active it would reject the write as an UNVERIFIABLE_WRITE. Dry-run is telling you so now, before you turn enforcement on. Write to a table or a path with the DataFrame API or INSERT INTO, or check Connector Support for the shape you need. (You will often see an INFERRED event for the same statement too: the command is executed as a supported write underneath.)

Dry-run reporting prints a warning and no events arrive

If spark.invaract.notifyConfig can’t be loaded, dry-run mode doesn’t fail your job. It logs one warning, and falls back to logging the inferred contract:

WARN AttachSetup$: Invaract dry-run mode: could not set up the notification sink named by 'spark.invaract.notifyConfig=/etc/invaract/notify.properties' (java.lang.IllegalArgumentException: 'spark.invaract.notifyConfig' names '/etc/invaract/notify.properties', which is not a file that exists.); falling back to logging inferred contracts. The job is not affected.

The cause is in the parentheses: here, a path that doesn’t exist on the machine where the driver runs. In a cluster deploy mode that is the cluster, not your laptop, so make sure the file is shipped there (for example with --files). A sink.class that doesn’t exist (Class '...' was not found: check the name, and that its jar is on the job's classpath), or a sink whose own required property is missing, is reported the same way. When Invaract is enforcing a contract rather than in dry-run mode, each of these stops the job at start-up instead, with the same message.

Beam: the pipeline runs, but nothing is checked

--runner=Invaract (Java) or --runner=invaract_beam.InvaractRunner (Python) only takes effect if the job builds its options from its command line and doesn’t set a runner in code afterwards. A job that does either runs on its own runner and Invaract never sees it: no log line, no events. The Java guide’s When the command line cannot reach the job shows the one-line check to add for such a job.

Beam Python: CheckNotRun

The pipeline was not submitted because the check could not be made. The message says why:

Message starts	Fix
Invaract could not start its checker (java: ...)	Install Java 11 or later where the job is submitted, or point --invaract_java at it.
Invaract has no checker to run	Build the checker jar (cd beam-adapter && sbt assembly) and set INVARACT_BEAM_CHECKER_JAR or --invaract_checker_jar.
--invaract_check_only=... must be true or false	Use true or false.
--invaract_delegate names InvaractRunner itself	Name the runner the job should really run on, for example DataflowRunner.
Invaract could not check the pipeline, ...	The checker’s own reason follows, for example a contract file it can’t read or a setting it doesn’t know.

A job submitted from a machine without the package stops earlier, with ModuleNotFoundError: No module named 'invaract_beam': install it there (What a job needs).

Beam: a check-only run in CI fails before Invaract runs

Beam validates some steps itself before handing the pipeline to any runner, including in a check-only run. A Java BigQuery read or write looks its dataset up in BigQuery, and wants --tempLocation; a Python file read checks that its pattern matches a file. Give the build what Beam asks for, or build those steps without validation. The Java and Python guides say exactly what each IO needs.

Still stuck

1. Look up the violation in Violation Types, and check What Each Engine Checks for whether your engine can make the check at all.
2. Open an issue on GitHub with the full violation message: it says what the contract expects, what the job contains, and why they disagree.