Enforce Row-Level DML Rules

Beyond schema, location, and format checks, a contract can constrain a table’s row-level MERGE/UPDATE/DELETE statements. These are checked against a real Spark plan the same way everything else is — the write is aborted before anything is committed if a rule is violated.

Spark only

Row-level DML exists only on Spark. A Beam pipeline reads collections and writes new output, so these rules don’t apply there, and neither do custom rule types (below).

The three rule types

rules:
  - type: merge_condition
    columns: [customer_id, region]
  - type: forbid_unconditional_delete
  - type: allowed_update_columns
    columns: [status, updated_at]

merge_condition

A MERGE’s ON clause must include a genuine equality match (t.col = s.col, or the null-safe t.col <=> s.col) on every listed column — not merely mention it. This catches several common, costly bugs: a MERGE silently missing a match key (for example, matching only on customer_id in a table that should also match on region, quietly overwriting the wrong rows), matching on a range check instead of an equality, pinning a column to a literal instead of joining target to source on it, only matching inside an OR branch instead of a required condition, or comparing a target column to itself — ON t.id = t.id, a copy-paste bug that always holds since it matches every row against itself, rather than genuinely matching target against source. A declared column may be paired against a differently-named column on the other side (t.customer_id = s.cust_id) — the contract can be authored against either side’s naming. An equality written with NOT, including via De Morgan’s laws (NOT (t.id != s.id), or NOT (t.id != s.id OR t.region != s.region) as an alternate way of writing a required double match), is recognized the same as its plain AND-of-equalities form — both are genuinely the same requirement.

Note

The YAML key is columns, not on. YAML 1.1’s resolver treats a bare on key as the boolean true (the “Norway problem” — on/off/yes/no all resolve to booleans), so on was avoided deliberately.

forbid_unconditional_delete

A DELETE may never omit a filtering predicate — DELETE FROM t with no WHERE is rejected. No properties needed.

allowed_update_columns

An UPDATE may only assign to the listed columns. Assigning to any column outside the list is rejected.

What’s checked, per connector

Rule	Delta (copy-on-write & merge-on-read)	Iceberg (copy-on-write)	Iceberg (merge-on-read)
merge_condition	✅	✅	✅
forbid_unconditional_delete	✅	✅	✅
allowed_update_columns	✅	✅	⚠️ fails closed

Iceberg’s merge-on-read UPDATE rewrites into a plan shape with no per-column before/after pairing to compare, so Invaract can’t extract which columns actually changed. Rather than silently reporting “no violation” with no real check behind it, a write matching this shape is rejected with RULE_UNVERIFIABLE_DML — see Fail-Closed by Default. This only fires when the active contract actually declares an allowed_update_columns rule; the same operation under a contract with no such rule executes normally.

What these rules don’t check yet

Set expectations correctly rather than assuming full coverage:

• Full predicate logic for merge_condition. It reasons about AND/OR/NOT, De Morgan equivalences, and target-vs-source-side qualifiers, but a match expressed inside a CASE WHEN is never recognized — that equality only holds conditionally, on some rows, not as a required part of the match.
• Which rows an UPDATE touches, or whether a non-empty DELETE predicate is trivially satisfiable (e.g. WHERE 1=1).
• Connectors without row-level DML support. Parquet, CSV, JDBC, and similar don’t support MERGE/UPDATE/DELETE via Spark SQL at all — Spark rejects them before any plan reaches Invaract.
• Governance rules (field-level masking, residency, purpose limitation) — no rule vocabulary exists for these yet; rules entries of other types are recorded on the contract but not interpreted.
• Transformation-shape rules (join, aggregation and filter shape) are a separate rule family, checked against the whole plan: see Enforce Transformation Shape Rules.

Write your own rule type

The three built-in types cover a lot of ground, but they’re a closed set — an organization’s own DML governance vocabulary (“never allow an UPDATE to touch this column directly,” “a DELETE against this table must always carry an archived_reason predicate”) isn’t limited to it. customRuleTypes maps a type: name your contract’s rules: list uses to a class implementing CustomRuleVerifier:

import com.invaract.contract.ContractRule
import com.invaract.ir.RowMutation
import com.invaract.verification.{CustomRuleVerifier, MutationKind, Violation, ViolationType}
class ForbidPasswordColumnUpdate extends CustomRuleVerifier {
  override def appliesTo(kind: MutationKind): Boolean = kind == MutationKind.Update

  override def verify(rule: ContractRule, mutation: RowMutation): List[Violation] =
    if (mutation.updatedColumns.contains("password"))
      List(
        Violation(
          ViolationType.InvalidContract,
          s"organizational rule '${rule.ruleType}' forbids updating the 'password' column directly.",
          remediation = "Route password changes through the dedicated credential-rotation job instead."
        )
      )
    else Nil
}

Point customRuleTypes at it, the same class-name-in-config shape sink.class uses for a notification sink and customPolicyTypes uses for a custom policy type:

customRuleTypes:
  forbid_password_update: com.acme.governance.ForbidPasswordColumnUpdate
rules:
  - type: forbid_password_update

Build and deploy it as its own small jar, and add it to --jars alongside the engine’s own — no Invaract source change, and no change to any governed job’s own source either:

spark-submit \
  --class com.example.MyRealJob \
  --master yarn \
  --jars invaract-spark-adapter-0.11.0.jar,my-governance-rules.jar \
  --conf spark.sql.extensions=com.invaract.sparkadapter.InvaractSparkSessionExtension \
  --conf spark.invaract.contract=/path/to/contract.yaml \
  my-real-job.jar

appliesTo decides which of MutationKind.Merge/Update/Delete your rule cares about — it’s also what decides whether a write this module recognizes as DML but can’t fully extract facts for (Iceberg’s merge-on-read UPDATE, above) fails closed under your rule, the same RULE_UNVERIFIABLE_DML treatment a built-in rule’s own kind-matching already gets. verify receives the same RowMutation a built-in rule type’s own check runs against — nothing about extraction is different for a custom type.

A typo is caught before any write is checked, not the next time a job runs

ContractEnforcementRule validates every customRuleTypes entry eagerly, the same moment it validates the rest of the contract — a misspelled class name, or one missing a public no-arg constructor, is rejected before any plan is ever checked, not silently ignored the next time a governed job runs.

A contract that names a class can run that class's constructor

customRuleTypes lives in the contract, so whoever can change the contract chooses which class Invaract instantiates in the driver. Invaract loads the named class without initializing it, checks that it implements CustomRuleVerifier, and only then constructs it, so a class that is not a verifier is rejected without any of its code running. A genuine verifier does run, in your job’s JVM: treat the contract source (a file in your repository, a registry) as trusted code, the same as the job’s jar, and restrict who can publish to it.

A bug in your own verifier isn't swallowed

An exception thrown from verify itself propagates, the same as a bug anywhere else in a job’s own installed check — it does not silently pass every mutation it was supposed to be checking.

A built-in type always wins a name collision

If customRuleTypes names a type: Invaract already interprets (merge_condition, forbid_unconditional_delete, allowed_update_columns), the built-in check runs — your custom verifier is never consulted for that rule type. ContractValidator warns on this rather than erroring, since the contract still evaluates correctly, just not the way the collision might suggest.

Try it

The regression pack (./dev/regression) demonstrates the structural PASS/FAIL guarantee end-to-end; see Prove Enforcement with the Regression Pack to run it yourself. For the exact violation types these rules produce, see Reference → Violation Types.