Declare Input and Output Types

A contract already says what a dataset looks like (location, format, schema). It can also say what a dataset represents — a governed, shareable data product; data entering your pipeline from outside your own responsibility; or data that only operates or controls processing rather than being the business data itself. Two datasets can be physically identical (both, say, Delta tables) and still play completely different roles — the type you declare is what tells Invaract, and anyone reading the contract, which one this is.

Engines

Declaring type works on every engine. The roleConsistency check, which compares a declared type with how the job really uses the dataset, is Spark only: on Beam, turning it on refuses the pipeline with UNSUPPORTED_CONTRACT_FEATURE before it runs.

Entirely optional, unless your organization says otherwise

Declaring type has zero effect on schema/location/format checking on its own. See Make it mandatory with an organizational policy below for how a platform team turns this from optional into required.

The three types

inputs:
  - name: customer
    location: gold/customer_master
    type: DATA_ASSET
  - name: vendor_feed
    location: bronze/vendor_feed
    type: SOURCE
  - name: processing_calendar
    location: control/processing_calendar
    type: CONTROL
outputs:
  - name: customer_position
    location: gold/customer_position
    type: DATA_ASSET

• DATA_ASSET — a governed, shareable dataset produced for consumption by other users, applications, or pipelines: a real data product (customer position, account exposure, a daily transaction dataset), not implementation state. A DATA_ASSET your contract produces can be consumed as a DATA_ASSET input to someone else’s contract — see Cross-contract validation below.
• SOURCE — data entering your pipeline that this contract doesn’t claim responsibility for producing: an external vendor feed, an upstream system extract, data received from another organizational boundary. Your contract can consume a SOURCE, but never declare one as its own output — see A SOURCE is never your own output below.
• CONTROL — data used to operate, control, or determine processing: a processing calendar, a watermark, a readiness or reconciliation signal, a configuration value represented as data. It can influence how your job runs without ever becoming part of the business data your job produces.

type is matched case-insensitively; anything other than these three values is rejected when the contract is parsed — a closed set, unlike the open-vocabulary format field.

Tip

Physical storage never decides this. A DATA_ASSET, a SOURCE, and a CONTROL dataset can all be the exact same table format in the exact same catalog — the type is a statement about ownership and responsibility, not about format/catalog.

A SOURCE is never your own output

outputs:
  - name: result
    location: gold/result
    type: SOURCE # ContractValidator warns: a SOURCE can't be your own output

ContractValidator warns (not an error — nothing is structurally broken) when an output declares type: SOURCE. By definition, a SOURCE is data your contract doesn’t claim responsibility for producing — declaring it as your own output contradicts that. If you meant “this is a real data product I produce,” declare DATA_ASSET; if you meant “this is control/state I maintain,” declare CONTROL.

Make it mandatory with an organizational policy

Dataset.datasetType itself always defaults to unset — every existing contract is unaffected. Whether your organization requires every dataset to declare one is a policy decision, using the same organizational policy mechanism that already makes description/catalog mandatory-or-optional per organization:

org-policy.yaml

version: "1.0"
policies:
  - id: dataset-type-required
    type: require_dataset_type
    description: "Every input and output must declare its role."
  - id: outputs-must-be-data-assets
    type: require_dataset_type
    scope: outputs
    types: [DATA_ASSET]
    description: "A governed output is a real data product, not control state."

types is optional — omit it to require any declared type; set it (a single value or a list) to require one of a specific subset. No policy attached (or one in Warn mode) means declaring type stays optional; an Enforce-mode require_dataset_type policy rejects a contract that doesn’t declare one, the same way any other org-policy violation does — see Enforce an Organizational Policy.

Enable role-consistency checking

Declaring a type is one thing; Invaract can also check whether your transformation’s own logic actually behaves the way the declared type claims — the mechanically-verifiable half of contract conformance. RoleConsistencyVerifier checks this today for exactly one high-confidence shape: a CONTROL-declared input whose data reaches a produced output column — pipeline-only data isn’t supposed to become part of the business data you produce.

inputs:
  - name: processing_calendar
    location: control/processing_calendar
    type: CONTROL
    schema:
      fields:
        - name: as_of_date
          type: date

If as_of_date only ever appears in a WHERE/join condition, this conforms. If it (or any column derived from it) ends up in a produced output column, that’s a real, blocking contradiction — Invaract calls this out rather than silently trusting the declared type.

Off by default — real additional analysis work, the same reasoning static data quality and fingerprinting already use:

spark-submit \
  --class com.example.MyRealJob \
  --master yarn \
  --jars invaract-spark-adapter-0.11.0.jar \
  --conf spark.invaract.roleConsistency=true \
  my-real-job.jar

As long as the job already installs Invaract the ordinary way, that --conf is enough — ContractEnforcementRule.resolveVerificationOptions reads spark.invaract.roleConsistency off the real SparkSession, the identical mechanism spark.invaract.staticDataQuality already uses. A platform team can turn this on for a job whose source they don’t control — including forcing it on for every governed job via inject.minVerificationOptions.roleConsistency in an organizational policy, alongside require_dataset_type above.

Setting it explicitly in code works the same way too:

import com.invaract.sparkadapter.{ContractEnforcementRule}
import com.invaract.verification.{VerificationOptions}
val spark = SparkSession.builder()
  .withExtensions(_.injectCheckRule(
    ContractEnforcementRule.forContract(contract, VerificationOptions(roleConsistency = true))
  ))
  .getOrCreate()

The three verdicts

Every check reaches one of three verdicts — deliberately never collapsed to a simple pass/fail, so an unproven contradiction is never mistaken for a confirmed one:

Verdict	Meaning	Blocks the write?
Contradicts	A CONTROL-declared input’s data reaches a produced output column.	Yes — ROLE_CONSISTENCY_VIOLATION, same abort path as any other violation.
Conforms	A CONTROL-declared input observed only in a filter/join condition, never in output data; or a DATA_ASSET/SOURCE-declared input that does contribute to output data.	No.
CannotDetermine	A DATA_ASSET/SOURCE-declared input observed only in a filter/join condition, never in output data. Genuinely ambiguous — a join-only input can still gate which rows survive without any column of its own reaching the output.	No.

An input with no declared type, or never observed at all in the plan (already a MISSING_INPUT violation’s own job), produces no role-consistency result at all.

An input is recognized however the job reads it: under an alias (df.as("calendar")), joined with itself, or upstream of a .checkpoint() (see Violation Types → Checkpoints). Only an alias shared by reads of different datasets (both sides of a union aliased t) is left unresolved, since it doesn’t say which one a column came from.

On a published ContractValidationEvent, if you’ve also set up a notification sink — every entry, all three verdicts, on both a PASSED and a FAILED check:

{"eventType": "CONTRACT_VALIDATION", "status": "FAILED", "...": "...", "roleConformance": [{"dataset": "processing_calendar", "datasetType": "CONTROL", "verdict": "Contradicts", "detail": "observed contributing to a produced output column - CONTROL data must not become part of the resulting business data asset it is declared to only operate/control"}]}

Empty ("roleConformance": []) whenever spark.invaract.roleConsistency isn’t enabled, or no input declares a type at all — the same opt-in-and-empty-by-default shape dataQuality/fingerprints use.

Dry-run mode observes usage, never declares a type

Dry-run mode never invents a type for an inferred dataset — Invaract never represents an unproven semantic property as proven. What it can do is note what it actually saw: an inferred input’s description carries a short observation — whether it contributed to a produced output column, or was only ever referenced inside a filter/join condition — for you to review before deciding what to declare:

inputs:
  - name: input_1
    location: control/processing_calendar
    schema:
      fields:
        - name: as_of_date
          type: date
          required: true
    description: >-
      Observed: referenced only in filter/join conditions; never observed
      contributing to a produced output column. Review whether this is a
      CONTROL input.

“Read and used to filter” is real, observed evidence — it just isn’t proof of organizational intent, so it stays a note attached to description, not a declared type: CONTROL.

Cross-contract validation

Once more than one contract exists, a role declaration can contradict another contract’s own declaration for the same physical location — one contract declaring customer_master a DATA_ASSET output, another declaring the same location a SOURCE input instead, even though a governed contract already establishes what it really is. CrossContractLintCli checks exactly this across every pair of contracts found under one or more paths, no Spark involved:

# run from the repository root; file paths are relative to contract/, or absolute
cd contract && sbt "runMain com.invaract.contract.cli.CrossContractLintCli contracts/"

Exit code 0 means every contract parsed and no contradiction was found; 1 means a parse failure, no contract files found, or at least one contradiction — printed with the shared location and both contracts involved.

Type guarantee checks, configured by organizational policy

Cross-contract validation above catches one specific, hardcoded contradiction. Type guarantee checks are the general, extensible version of the same idea: stronger semantic/guarantee checks across every contract discovered together, entirely configured through your organizational policy document — which checks run, and whether a real contradiction blocks the lint run or only gets reported.

org-policy.yaml

version: "1.0"
typeGuarantees:
  enabled: [data_asset_schema_consistency, data_asset_downstream_consumption]
  mode: enforce # or 'warn' - every enabled check still runs and is reported either way

cd contract && sbt "runMain com.invaract.contract.cli.CrossContractLintCli --org-policy org-policy.yaml contracts/"

Two checks are built in:

• data_asset_schema_consistency — every DATA_ASSET-typed dataset declared at the same physical location, across every contract in this run, must agree on schema. A common field with a different type, or a required field one side is missing, is a real contradiction.
• data_asset_downstream_consumption — for each DATA_ASSET output, is it declared as an input anywhere else? If yes, conforms. If no, that’s reported as CannotDetermine — never as a contradiction, since a consumer contract simply not written yet (or not included in this lint run) looks identical to one that will never exist. Invaract cannot prove a DATA_ASSET nobody currently consumes is dead pipeline-only state, so it never claims to.

Omit typeGuarantees entirely (or the --org-policy flag) and no type guarantee checking happens at all — the same “absent means off” default every other opt-in capability in this guide follows.

Mandatory or optional, per organization — the same theme as declaring a type at all

typeGuarantees.mode draws the same enforce/warn line require_dataset_type draws for declaring a type in the first place:

• mode: warn (or omit the whole block) — every enabled check still runs, every result still prints, but nothing blocks. Good for rolling out a new check safely before it starts failing builds.
• mode: enforce — a real Contradicts result from any enabled check fails the lint run (non-zero exit), the same way any other blocking violation does.

CrossContractLintCli prints every result tagged [FAIL] (actually blocking), [ OK ] (Conforms), or [WARN] (CannotDetermine, or a Contradicts result under warn mode) — a [FAIL] line only ever appears alongside a non-zero exit code.

Adding your own check

typeGuarantees is built as an open registry, the same reflective, no-source-change extension point writing your own policy type already uses for OrgPolicy itself: implement com.invaract.contract.TypeGuaranteeCheck (one method, check(contracts: List[Contract]): List[TypeGuaranteeResult]), put the compiled class on contract’s classpath, and name it in customTypeGuaranteeTypes:

typeGuarantees:
  enabled: [data_asset_schema_consistency, my_check]
  customTypeGuaranteeTypes:
    my_check: com.acme.governance.MyTypeGuaranteeCheck

A name in enabled that’s neither a built-in check nor a customTypeGuaranteeTypes entry is reported (not silently dropped) as a policy warning: “will never be evaluated” — the same warning an org policy’s customPolicyTypes already gives for the identical mistake.

What this doesn’t check yet

• Whether a DATA_ASSET output is genuinely a business data product, as opposed to pipeline-only state dressed up as one — data_asset_downstream_consumption only ever answers “is it consumed by a contract in this run,” never “should it exist at all,” an organisational judgment structure alone can’t make.
• Whether a DATA_ASSET is produced from inputs whose real transformation logic is consistent with its own declared contract — data_asset_schema_consistency only compares declared schemas across contracts, not the transformation itself against ir.Lineage.
• A DATA_ASSET/SOURCE input observed only gating rows (present in a join/filter, never contributing a column) is reported CannotDetermine, never asserted as a contradiction — genuinely ambiguous from structure alone.
• True business-guarantee reasoning beyond these mechanical, structural checks — a new TypeGuaranteeCheck can be added as confidence grows that a specific check is high-enough confidence not to overclaim, without changing this module at all.