Connector Support

Invaract only verifies a write it recognizes: it has to know where the write lands, in what format, and with which columns (see The Transformation IR). A write it doesn’t recognize is rejected rather than silently passed — see Fail-Closed by Default.

Spark connectors

Connector	Reads	Writes	Row-level DML (MERGE/UPDATE/DELETE)	Tested library versions	Notes
Delta Lake	✅	✅	✅	3.3.3 only	Row-level DML translated via a dedicated extractor; schema-evolution and generated-column edge cases investigated and fixed. Earlier releases (including 3.3.0–3.3.2) target an older Spark version this repo has already moved past — 3.3.3 is the only compatible release, not just the newest.
Iceberg	✅	✅	✅ (copy-on-write fully; merge-on-read UPDATE fails closed)	1.10.2 – 1.11.0	Catalog-based DSv2 connector. CALL procedures (e.g. rollback_to_snapshot) are classified individually — some verified against a contract, others safely allowed through, per procedure.
Parquet	✅	✅	—	—	Not a separate library — Spark’s own bundled file format.
CSV	✅	✅	—	—	Same shape as Parquet.
Hive	✅	✅	—	—	Table reads plus multiple write shapes (including CREATE TABLE ... AS SELECT and static-partition INSERT). Ships with Spark itself, not independently versioned here.
Avro	✅	✅	—	—	Reuses the same generic write recognition as Parquet/CSV — no Avro-specific code needed.
ClickHouse	✅	✅	DELETE only	—	UPDATE/MERGE are not yet translated for this connector. Verified against a real standalone ClickHouse server.
JDBC (generic)	✅	—	—	—	Reads recognized precisely, including the resolved connection/table identity.

Parquet, CSV, and JDBC don’t support MERGE/UPDATE/DELETE via Spark SQL at all — Spark itself rejects those statements before any plan reaches Invaract, so there’s nothing to verify there.

Tested library versions are CI-enforced, not aspirational — every push runs Invaract’s Delta and Iceberg tests against each version listed. The Delta and Iceberg notes in the repository say which versions were left out, and why.

Recognized Spark write commands

Any write that reaches one of these real Spark/Catalyst commands is recognized, regardless of which connector issued it:

Spark command	Typical trigger
InsertIntoHadoopFsRelationCommand	df.write.parquet(...), .csv(...), .avro(...) and similar file-format writes
SaveIntoDataSourceCommand	df.write.format("delta").save(...), and any other CreatableRelationProvider-based .save(...)
CreateDataSourceTableAsSelectCommand	df.write.saveAsTable(...) / CREATE TABLE ... AS SELECT against a new table
WriteToStream	df.writeStream...start(): a Structured Streaming query, checked once when it starts

Most new connectors need zero new recognition code — any connector built on CreatableRelationProvider’s .save(...) convention is already covered by SaveIntoDataSourceCommand above. A connector only needs dedicated support when it introduces a genuinely new write-command shape Spark has no existing generic node for (row-level DML, DSv2 catalog writes, and similar).

What happens for a write shape that isn’t recognized

If your contract is active and a write doesn’t match a recognized shape above, the write is rejected — UNVERIFIABLE_WRITE — rather than silently allowed through. See Fail-Closed by Default for why, and Reference → Violation Types for the full violation vocabulary.

Tip

A DataSourceV2 catalog write outside the shapes above falls into this fail-closed path too. Support for the connector you need may be straightforward to add: check the project’s issue tracker or open an issue.

Apache Beam IOs

The Beam adapter checks a pipeline before it runs, so it knows a read or a write by the IO transform the pipeline uses.

IO	Java pipelines	Python pipelines	Format checked as
Parquet	✅ ParquetIO	✅ ReadFromParquet, WriteToParquet	parquet
Avro	✅ AvroIO	✅ ReadFromAvro, WriteToAvro	avro
Text	✅ TextIO	✅ ReadFromText, WriteToText	text, csv or json (a text write can’t tell them apart)
CSV	✅ CsvIO.writeRows; a read is TextIO then CsvIO.parseRows	ReadFromCsv is seen but has no location; WriteToCsv is refused	csv
TFRecord	✅ TFRecordIO	✅ ReadFromTFRecord, WriteToTFRecord	tfrecord
Files	✅ FileIO.match, FileIO.write, FileIO.writeDynamic	Reads without a location (MatchFiles); fileio.WriteToFiles is refused	the sink’s format
BigQuery	✅ tables (a query has no location)	✅ tables (a query has no location)	bigquery, with its save mode
Kafka	✅ KafkaIO	—	kafka
JDBC	✅ JdbcIO	—	jdbc
Pub/Sub	✅ topics and subscriptions	✅ topics and subscriptions	—

How each IO’s location is written in a contract, and what Invaract can’t tell from it, is in the guide for each SDK: Java and Python. A step that ends the pipeline without being one of these writes is refused as UNVERIFIABLE_WRITE, unless a platform names it as a trusted step.