Verify Static Data Quality

A contract can already require an output field to be non-null (nullable: false). Beyond that, a field can declare that it must always equal a specific constant, always be one of a finite set of values, always fall within a numeric range, always fall within a range relative to another field on the same row (e.g. end_date >= start_date), or (for a string-typed field) always have a specific length — and Invaract will attempt to prove, from the transformation’s own translated logic, that the requirement holds for every possible row, not just check it against whatever data happened to be present on one run.

Spark only

Static data-quality proofs need the expressions a job computes each column with, which Spark’s plan gives and Beam’s pipeline graph doesn’t. On Beam, turning staticDataQuality on refuses the pipeline with UNSUPPORTED_CONTRACT_FEATURE before it runs.

This proves guarantees, not predicts them

The objective is never “will this data quality rule probably pass?” — it’s “does the transformation’s own code guarantee this property?” When the analysis can’t prove something either way, it says so honestly (NotGuaranteed/NotStaticallyVerifiable) rather than pretending to have checked it. A false claim of a guarantee would be worse than not proving one at all — see the design document for the full reasoning.

Declare a constraint

outputs:
  - name: payments
    location: gold/payments
    schema:
      fields:
        - name: customer_id
          type: long
          nullable: false          # NOT NULL - no separate constraint needed
        - name: currency
          type: string
          constraints:
            - type: equals
              value: GBP
        - name: status
          type: string
          constraints:
            - type: oneOf
              values: [ACTIVE, INACTIVE]
        - name: amount
          type: double
          constraints:
            - type: range
              gte: 0
        - name: identifier
          type: string
          constraints:
            - type: length
              exact: 10
        - name: start_date
          type: long
        - name: end_date
          type: long
          constraints:
            - type: fieldRange
              gte: start_date

See Reference → Contract Format for the full constraints shape.

Enable it

Like fingerprinting, this is real additional analysis work Invaract shouldn’t impose on every existing caller by default — off unless you turn it on, via spark-submit --conf:

spark-submit \
  --class com.example.MyRealJob \
  --master yarn \
  --jars invaract-spark-adapter-0.11.0.jar \
  --conf spark.invaract.staticDataQuality=true \
  my-real-job.jar

As long as the job already installs Invaract the ordinary way, that --conf is enough — ContractEnforcementRule.resolveVerificationOptions reads spark.invaract.staticDataQuality off the real SparkSession, the identical mechanism spark.invaract.computeFingerprint already uses. A platform team can turn this on for a job whose source they don’t control.

Setting it explicitly in code works the same way too:

import com.invaract.sparkadapter.{ContractEnforcementRule}
import com.invaract.verification.{VerificationOptions}
val spark = SparkSession.builder()
  .withExtensions(_.injectCheckRule(
    ContractEnforcementRule.forContract(contract, VerificationOptions(staticDataQuality = true))
  ))
  .getOrCreate()

What “proven” actually means here

Invaract’s transformation IR already traces column-level provenance for lineage and sensitivity propagation. Static data-quality verification adds a second, parallel trace over the same IR: instead of asking “where did this column’s data come from,” it asks “what value-domain facts can I prove hold for this column, given what I can trust about the input.” A nullable: false/constraints entry on a contract input field seeds that trace as a trusted axiom; the trace then follows the plan’s Filter/Project/Conditional/Arithmetic/Join/… nodes the same way lineage does, narrowing or widening what’s provable at each step.

A few worked examples (the full set, mechanically traced, is in the design document):

• A WHERE customer_id IS NOT NULL filter upstream of the output proves nullable: false on customer_id, with no input axiom needed at all.
• .withColumn("currency", lit("GBP")) proves constraints: [{type: equals, value: GBP}] directly — the literal is the constant.
• A CASE WHEN is_active THEN 'ACTIVE' ELSE 'INACTIVE' END proves constraints: [{type: oneOf, values: [ACTIVE, INACTIVE]}] — both branches are constants and every case is covered (there’s an ELSE).
• A clamp (CASE WHEN amount < 0 THEN 0 ELSE amount END) proves constraints: [{type: range, gte: 0}], again with no input axiom.
• An input contract declaring constraints: [{type: length, exact: 10}] on an identifier field, passed straight through unchanged, proves the same length: exact: 10 constraint on the output — the axiom flows through a pure passthrough for free, the same mechanism nullable: false/range already use for Example 5’s shape.
• Negating a column the input contract already guarantees non-negative — .withColumn("amount", -col("amount")) — provably violates a gte: 0 constraint on the output: the transformation’s own logic guarantees the opposite of what’s required, which is exactly the case that aborts the write (see “The four verdicts” below).
• A nested field’s own constraint — struct(col("zip"), lit("US").as("country")) with constraints: [{type: equals, value: US}] declared on address.country’s own nested properties — proves Guaranteed the same way a flat column does, by tracing into the struct the transformation itself just built.
• A cross-field bound — an end_date field with constraints: [{type: fieldRange, gte: start_date}] proves Guaranteed whenever end_date’s own proven lower bound is already >= start_date’s own proven upper bound (e.g. an input contract declaring start_date’s own range: {lte: 1000} alongside end_date’s own range: {gte: 1000}, both passed straight through) — a direct comparison between each field’s own already-analyzed value range, not a new kind of tracing.

The four verdicts

Every field-level check reaches one of four verdicts — deliberately never collapsed to a simple pass/fail:

Verdict	Meaning	Blocks the write?
Guaranteed	The transformation’s own logic proves the property holds for every possible row.	No — nothing to block.
Violated	The transformation’s own logic proves the property is broken for some possible row.	Yes — DATA_QUALITY_VIOLATION, same abort path as any other violation.
NotGuaranteed	Neither proof nor disproof was possible — the honest, common “I don’t know.”	No.
NotStaticallyVerifiable	The column’s derivation involves something this analysis deliberately doesn’t reason about (a UDF, a window function, an aggregate).	No.

Only Violated produces a real Violation (see Violation Types) and aborts the write — the same as a schema mismatch or a broken plan-shape rule. Guaranteed/NotGuaranteed/NotStaticallyVerifiable are reporting only, populating VerificationResult.dataQuality, and never affect whether the check passes on their own. This matters more here than for most flags: a narrow, deliberately conservative first implementation will legitimately return NotGuaranteed/NotStaticallyVerifiable for a large fraction of real transformations — most don’t hit the whitelist of provable constructs yet — and that’s expected, not a regression.

On a published ContractValidationEvent, if you’ve also set up a notification sink — every entry from every field-level check, all four verdicts, on both a PASSED and a FAILED check:

{"eventType": "CONTRACT_VALIDATION", "status": "PASSED", "...": "...", "dataQuality": [{"field": "customer_id", "constraint": "notNull", "verdict": "Guaranteed"}, {"field": "amount", "constraint": "range(gte=0)", "verdict": "NotGuaranteed"}]}

Empty ("dataQuality": []) whenever spark.invaract.staticDataQuality isn’t enabled, or the contract declares no field constraints at all — the same opt-in-and-empty-by-default shape fingerprints uses.

Programmatically, on the VerificationResult itself:

val result: VerificationResult = ??? // from your own check
result.dataQuality.foreach { check =>
  println(s"${check.field} (${check.constraint}): ${check.verdict}")
}

Runtime data quality is still necessary

This does not replace runtime data-quality checking. Null rate, averages, uniqueness, row counts, freshness, referential integrity, and statistical distributions are facts about the data, not the transformation’s code — nothing about a plan’s shape can prove them, and this module correctly never tries. What it can prove is a narrower, but zero-cost, class of guarantee: properties that are true for every possible run because they’re baked into the transformation’s own logic, catchable before a single row is written rather than rediscovered downstream by a job that profiles the output after the fact.

What this doesn’t check yet

• Anything involving a UDF, a window function, or an aggregate function — always NotStaticallyVerifiable, deliberately: guessing at arbitrary function behavior is exactly the false-guarantee risk this feature exists to avoid.
• A CASE WHEN branch whose value is itself a computed expression (not a bare column reference or literal) doesn’t get the same conditional narrowing a bare column reference does — a documented MVP simplification, not a soundness gap (it only ever under-claims, never over-claims, a guarantee).
• A constraint declared on a nested (struct/record) field — constraints/nullable: false on a field’s own nested properties — is now proven for real whenever the parent struct is built by the transformation itself (e.g. struct(col("zip"), col("city")), even through an intervening rename like .withColumn("address", struct(...)).select("address", ...)): the same “construct, then extract” resolution a flat output column already gets, now reached one level into a nested field a contract declares its own obligation on. It still reports NotStaticallyVerifiable when the struct comes from anywhere this analysis can’t trace into — a struct-typed column read straight from the input with no transformation at all, the result of a UDF, or an Aggregate/Window/Union/Join output. The struct field itself (at the top level) is unaffected either way — its own nullable/ constraints are checked normally. A flat output column produced by extracting one field straight back out of a struct built in the same expression (e.g. struct(col("zip"), col("city")).getField("zip")) is checked for real too, the same as any other computed column.
• String constraints beyond length — pattern/regex requirements aren’t supported at all. length itself is proven through a pure passthrough (an unchanged input column), a string literal, LENGTH/CHAR_LENGTH/CHARACTER_LENGTH, UPPER/LOWER, and TRIM/LTRIM/RTRIM — not the full string-function surface (SUBSTRING, CONCAT, REPLACE, … all resolve to NotGuaranteed, the same honest “I don’t know” every other unrecognized construct gets).
• Sampling, historical runs, or current data — never consulted. Every proof (or disproof) comes from the transformation’s translated logic and the contract’s declared input properties alone.
• A fieldRange constraint on a nested field, or referencing a nested field — always NotStaticallyVerifiable: cross-field comparison only resolves between top-level output fields, deliberately, to avoid an ambiguous match against an unrelated field that happens to share a nested field’s bare name. Both the constrained field and the field it references must also be numeric-typed — the proof is a direct comparison of each side’s own analyzed numeric range.
• A formula relationship between fields (total = quantity * price, an arithmetic identity, not an ordering) — genuinely different from fieldRange: an ordering bound is proven from each side’s own already-known value range alone, while a formula identity would need to trace how one column’s expression actually relates to another’s. Not supported.