twitter/scalding

Scalding is a Scala library for writing distributed data processing jobs, built on top of Apache Cascading and supporting multiple execution backends including Hadoop, Apache Beam, and Spark. It provides a type-safe, functional API for defining data pipelines that can read from and write to various data sources such as HDFS, Parquet, Avro, and JSON. The system includes utilities for argument parsing, date handling, serialization, and mathematical operations, along with a REPL for interactive development.

This period focused on modernizing the Scalding codebase by migrating the build system, removing legacy components, and introducing a comprehensive typed API. The work introduced new features for handling Avro, JSON, and LZO-compressed data, alongside a REPL for interactive development and improved command-line argument parsing.

27 changes

This period focused on expanding Scalding's data source capabilities, particularly through the introduction of Parquet support for both Thrift and Tuple types, as well as HBase and in-memory sources. Significant work was also done on the serialization layer, introducing macro-based ordered serialization for Scala types and improving the REPL and testing infrastructure.

34 changes

This period focused on decoupling the Scalding framework into a modular architecture, introducing new execution backends for Spark and Apache Beam, and adding support for Parquet and Thrift data formats. The codebase was refactored to isolate backend logic and establish a shared base module, while also integrating functional programming patterns via the Cats library and compile-time macros for code analysis.

14 changes

twitter/scalding was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.