twosigma/flint

Flint is a time-series analytics library for Apache Spark that enables efficient statistical summarization and windowing operations on large datasets. It provides a comprehensive set of built-in summarizers for calculations like exponential smoothing, correlation, and regression, while supporting custom user-defined functions and predicate-based filtering. The system optimizes performance through Arrow-based data serialization, tree-based aggregation, and memory-efficient RDD partitioning. It is accessible via both Scala and Python APIs, allowing users to read, join, and analyze time-series data with minimal memory overhead.

This period focused on expanding the library's analytical capabilities by introducing numerous new summarizers for time-series data, including exponential smoothing, weighted covariance, and geometric mean. Concurrently, the project established a Python API (ts-flint) with comprehensive unit tests and CI automation, while upgrading the underlying Spark and Scala dependencies to improve performance and stability.

24 changes

This period focused on enhancing data serialization and processing efficiency by introducing Apache Arrow-based components for zero-copy data exchange. The team also implemented batched window summarization, a builder API for time series data retrieval, and precise floating-point summation utilities.

6 changes

twosigma/flint was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.