Fingerprint a Transformation's Business Logic

Two Spark jobs can produce byte-for-byte the same output schema while computing something completely different — value = amount * 1.20 and value = amount * 1.25 are both a DOUBLE column named value, but they’re not the same business logic. Schema verification (everything else in this documentation) cannot tell them apart, because it only checks shape. Semantic lineage fingerprinting can: it turns a transformation’s real, translated logic into a deterministic hash, so a later revision of the same job can be compared against it.

Spark only

A fingerprint is computed from the expressions a job computes each column with, which Spark’s plan gives and Beam’s pipeline graph doesn’t. On Beam, turning computeFingerprint on doesn’t stop the pipeline; it publishes no fingerprint, and a warning in the log says why. A fingerprint of the graph alone would miss the logic inside its steps.

This is a hash, not a diff

Enabling this gives you a fingerprint for this check — a value you can log, publish, and compare by hand or with your own tooling. Invaract does not yet store a history of fingerprints or diff two of them for you; see the design document for what’s built today versus what’s still future work.

Enable it

Fingerprinting is off by default — computing one does real work (translating and hashing the whole plan) on every check, so it’s opt-in the same way rejectUndeclaredInputs/rejectUndeclaredFields are. Turn it on via spark-submit --conf, the same as everything else already installed:

spark-submit \
  --class com.example.MyRealJob \
  --master yarn \
  --jars invaract-spark-adapter-0.11.0.jar \
  --conf spark.invaract.computeFingerprint=true \
  my-real-job.jar

As long as the job already installs Invaract the ordinary way —

import com.invaract.sparkadapter.ContractEnforcementRule

val spark = SparkSession.builder()
  .withExtensions(_.injectCheckRule(ContractEnforcementRule.forContract(contract)))
  .getOrCreate()

— that --conf is enough: forContract reads spark.invaract.computeFingerprint off the real SparkSession at the same moment it reads everything else Spark configuration can drive (see Resolve Dataset Locations at Runtime for the mechanism in full). A platform team can turn fingerprinting on for a job whose source they don’t control, the same way they can attach a location map.

Need to set it explicitly in code?

Pass a VerificationOptions(computeFingerprint = true) to ContractEnforcementRule.forContract yourself — see Install and Configure Invaract Explicitly in Code. The two compose: a flag ends up on if either side turns it on, so a platform attaching --conf spark.invaract.computeFingerprint=true can never be silently overridden by code that left it at its default, and code that deliberately opted in can never be silently turned off by a missing conf key. rejectUndeclaredInputs/rejectUndeclaredFields follow the identical pattern, via spark.invaract.rejectUndeclaredInputs/ spark.invaract.rejectUndeclaredFields — see Reference → Violation Types for what each one changes.

Where the fingerprint shows up

In a rejected write’s exception message. When a check with computeFingerprint = true fails, ContractViolationException’s message gains a Fingerprints section after the usual “how to correct it” list — the same fingerprint that check computed, not a diff against anything prior. This is the end of the real message for a job that reads raw/orders, computes value = amount * 1.20 and writes gold/customer_values, against a contract that declares value as long (the job produces a double):

How to correct it:
  1. Cast 'value' to 'long' in the transformation, or update the contract to declare 'double' if the new type is intentional.

Fingerprints (v4, SHA-256):
  overall: 7173bc8758e3b735afc616302a32c3351494c89d7d7d9a0c7be03d761275eb0b
  outputs:
    customer_id: d0ad273a8a49f68fdae80a82730e3f04f78e821bc953d53f3e274f59ae122549
    value: fb215b585bfe17ec620bd49cb9d5f6d115973c6326301199dfdd432ede138cf4

Only each output’s combined hash is printed, to keep the message readable — see “Reading the hierarchy” below for the finer-grained values available programmatically.

On a published ContractValidationEvent, if you’ve also set up a notification sink — for every check, PASS or FAIL, since a passing check’s fingerprint is exactly the baseline a later comparison needs:

{"eventType": "CONTRACT_VALIDATION", "status": "PASSED", "...": "...", "fingerprints": {"version": 4, "overall": {"version": 4, "algorithm": "SHA-256", "value": "7173bc8758e3b735afc616302a32c3351494c89d7d7d9a0c7be03d761275eb0b"}, "inputs": {"file:/tmp/invaract-fingerprint-example/raw/orders#0": {"version": 4, "algorithm": "SHA-256", "value": "12789decf21121758b30b19080f0a66e101dcb6624e08958447bc8124a385ac9"}}, "outputs": {"customer_id": {"expression": {"...": "..."}, "lineage": {"...": "..."}, "combined": {"version": 4, "algorithm": "SHA-256", "value": "d0ad273a8a49f68fdae80a82730e3f04f78e821bc953d53f3e274f59ae122549"}, "nonDeterministic": false}, "value": {"...": "..."}}, "rowMutation": null}}

Programmatically, on the VerificationResult itself — useful if you’re calling ContractEnforcementRule.verifyOrThrow directly, or just want structured access instead of parsing either of the above:

val result: VerificationResult = ??? // from your own check
result.fingerprints.foreach { fp =>
  println(fp.overall.value)
  fp.outputs.get("value").foreach(out => println(out.combined.value))
}

Reading the hierarchy

A TransformationFingerprint is not one opaque hash — it’s structured so a future change can be localized to a specific column, not just detected somewhere in the whole plan:

Level	What it answers
overall	Did anything about this transformation change — including things that don’t belong to any single output, like the write’s format or save mode?
inputs	Which physical datasets does this transformation read, and how many times (keyed "<location>#<occurrence>", so a self-join’s two reads of the same table never collapse into one entry)?
outputs(name).expression	Did this column’s own computation change, syntactically?
outputs(name).lineage	Did this column’s resolved dependency shape change — which source columns it derives from, how directly, and through which aggregate function(s)?
outputs(name).combined	The practical “did this column change” signal — moves whenever either of the two above does.
outputs(name).nonDeterministic	true/false/absent (unknown) — does this column’s definition contain a call Invaract recognizes as producing a different result every run (rand(), uuid(), current_timestamp(), …)? Recognition goes by a shared catalog of functions, so it is the same on every engine adapter (an engine’s own spelling — Spark’s random(), BigQuery’s GENERATE_UUID() — is mapped onto it)? Metadata only; it never affects any hash above.
rowMutation	Present only for a MERGE/UPDATE/DELETE: did the row-matching logic itself change — a MERGE’s ON condition, a conditional DELETE/UPDATE’s predicate, or which columns an UPDATE assigns? Absent (and overall unaffected by its absence) for an ordinary INSERT/overwrite write.

Changing amount * 1.20 to amount * 1.25 moves value’s expression/combined but not its lineage (same source column, same kind of computation) or customer_id’s fingerprint at all — the real, complete output from the run this example’s hex values were captured from is worked through in full in the design document’s own §13.

Every hash carries the version of the rules that produced it. Compare only fingerprints with the same version: a change to the rules changes every value, for the same job. Version 4 is the current one; it changed how a hash is built (from each part’s own hash), so a job whose plan merges branches of the same data, or defines a column from itself repeatedly, is fingerprinted as quickly as any other. A fingerprint saved under version 3 or earlier will not match the same job’s version 4 fingerprint.

A read’s hash is built from its location, so a job reading from a different place (another environment’s bucket, a renamed folder) has a different overall and inputs, while each output’s expression and combined are unchanged if only the location moved.

What this can and cannot detect

This hashes the transformation’s definition as Invaract’s translation layer represents it — never a runtime value, and never anything outside that representation:

• A literal, operator, function, join type/condition, filter, grouping, or cast changing — detected, reliably.
• A UDF’s registered name or its declared arguments changing — detected. Its implementation changing, with the same name and arguments — not detected: the translated plan carries no version or hash of a UDF’s actual body, so this is an honest limitation, not a bug. nonDeterministic is null/absent for any output touching a UDF, for the same reason — its body could do anything.
• A pure relabeling — renaming a self-join’s aliases consistently, for instance — does not move the fingerprint. Reordering two genuinely different things (which side of a join a table sits on, two arguments to a non-commutative operator) does, even in cases where the reorder happens to be logically equivalent — Invaract does not attempt to prove operand reordering safe, since getting that wrong would silently hide a real change.
• An unaliased self-join of the same table — whether both sides are read via spark.table(...)/SQL, or via a bare spark.read.format(...).load(...) — is handled correctly: even though Spark itself either assigns both physical occurrences the same default name internally, or (for a bare .load() read) no distinguishing label at all, Invaract disambiguates them at translation time so each side’s own columns are tracked, and fingerprint, distinctly — reading the left side’s column after such a join is never confused with reading the right side’s. This also covers joining two genuinely different tables (from the same catalog, or different ones) that were both read via a bare .load(...): each retains its own physical identity, so selecting one side’s column never fingerprints the same as selecting the other’s. Explicitly aliasing both sides (.as("a")/.as("b")) is still good practice for readability, but isn’t required for correct fingerprinting.
• A .checkpoint()/.localCheckpoint() in the middle of the job does not change the fingerprint: Invaract remembers the plan each checkpoint was made from and fingerprints the real transformation through it, so a checkpointed job fingerprints identically to the same job without the checkpoint — including when the job joins a checkpointed Dataset with itself. Some cases can’t be seen through — a checkpoint created before the enforcement rule was installed, one whose origin is ambiguous (several plans sharing its output columns while reading different datasets), and a Delta MERGE whose source is the checkpointed Dataset — and there the fingerprint still comes out stable and deterministic, but can’t reflect what happened before the boundary. Invaract logs a WARNING when a fingerprint crosses one, so it isn’t silently trusted.
• A MERGE’s ON condition, a conditional DELETE/UPDATE’s predicate, or which columns an UPDATE assigns — detected, via rowMutation (see the table above), and folded into overall too. This is not part of the translated Plan itself (Invaract translates a MERGE’s query as its source side only, never the ON clause), so it only appears when Invaract can extract it — the same extraction the merge_condition/ forbid_unconditional_delete contract rules already rely on for enforcement.

See the design document’s “Key risks and design trade-offs” section for the complete, itemized account of both categories.