holdenk/spark-testing-base

This system is a cross-language testing utility library for Apache Spark, providing a unified framework for validating Spark applications in both Scala and Python. It offers specialized test base classes and assertion helpers for Spark SQL, Streaming, and RDDs, with explicit compatibility layers for Spark versions 2.4, 3.0, 3.5, and 4.0. The library also supports modern Spark Connect gRPC client testing and includes infrastructure for managing embedded Kafka and Zookeeper instances for integration testing.

This period focused on expanding the project's testing infrastructure, specifically by releasing a new Python-based testing base that supports Python 3 and provides specialized test cases for SQL and Streaming. Concurrently, the project added support for Spark 3.5+ and 4.0+ via the Spark Connect client, enabling gRPC-based client/server architecture testing. The release also included significant project infrastructure updates, such as a Code of Conduct and automated build scripts.

5 changes

This period focused on stabilizing the testing framework by adding compatibility layers for Spark 2.4 and introducing utilities for Kafka 0.8. Additionally, logging verbosity was reduced to improve CI build logs, and the build system was updated to use local sbt launch scripts for consistent CI environments.

4 changes

The work focused on establishing comprehensive testing infrastructure for Spark versions 2.4 through 4.0, including specific test suites for DataFrames, RDDs, and streaming operations. New utilities were added to support Java APIs, Spark Connect, and expression codegen validation, ensuring broad compatibility across different Spark releases.

5 changes

holdenk/spark-testing-base was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.