Engine Version Support

spark-adapter translates Spark’s Catalyst logical plan into Invaract’s engine-independent IR (see The Transformation IR) — an internal Spark API that isn’t guaranteed stable node-by-node release to release. This page states what’s actually been verified, not just what the code happens to compile against. For Apache Beam, see Apache Beam versions below.

Supported Spark versions

Spark version	Status	Notes
3.5.6	✅ Verified	Floor of the supported range.
3.5.7	✅ Verified	Verified patch.
3.5.9	✅ Verified	Newest verified patch, and the current primary version — what a real ./dev/test run installs and what spark-adapter is built against by default.

Every other 3.5.x patch at or above 3.5.6 is expected to work (Spark’s own patch releases don’t change Catalyst’s plan shapes), but is only a claim, not something CI checks for you — treat the three rows above as what’s actually proven.

Below 3.5.6 does not work

Spark 3.5.1 through 3.5.5 are not supported, confirmed by a real CI failure rather than assumed: Invaract’s Delta Lake support depends on delta-spark 3.3.3, which requires a class Spark only introduced in 3.5.6. Below that, installing the enforcement rule in a Delta-extended session throws ClassNotFoundException.

Spark 4.x is not supported

Invaract is built for Scala 2.12, and Spark 4.0 requires Scala 2.13. Running Invaract against a Spark 4.x cluster isn’t a configuration a real user has today.

How this is verified

Every one of the versions above is exercised by spark-adapter’s own full test suite, automatically, on every change — not a claim taken on faith. See docs/SPARK_ADAPTER.md’s “Spark version compatibility” section for how that CI job works if you’re contributing to spark-adapter itself.

ContractEnforcementRule also checks this same verified list at runtime, in your own job — see Install the Enforcement Rule → Version compatibility warnings. Running an unverified version doesn’t block your job, but logs a warning naming the version it detected, so you have a concrete signal if something related goes wrong.

Java and Scala

Component	Version
Java	21 (runtime); compiled to JVM 1.8 bytecode, so a JDK 11+ runtime also works
Scala	2.12.18

Scala 2.13 cross-compilation isn’t done — see the Spark 4.x note above for why it would be needed before a newer Spark line could be supported.

Apache Beam versions

The Beam adapters read Beam’s own description of a pipeline, which can change between releases, so each supported release is run through the full conformance suite in CI.

SDK	Tested releases	Notes
Java	2.60.0, 2.65.0, 2.70.0 and 2.77.0	The job brings its own Beam; the adapter does not bundle it.
Python	2.68.0, 2.71.0, 2.74.0 and 2.77.0	The checker it runs needs Java 11 or later on the machine that launches the pipeline.

The requirements of each SDK are in its guide: Java and Python.