awslabs/deequ

This system is a data quality and profiling library for Apache Spark, designed to verify data integrity, compute statistical metrics, and enforce constraints. It provides a fluent API for building and executing data analysis workflows, including column profiling, anomaly detection, and row-level quality checks. The system supports persistent storage of analysis results and integrates with the Glue DQDL format for rule definition.

This period focused on introducing fluent builder APIs for data verification, analysis, and constraint suggestions, significantly improving the developer experience. It also expanded storage capabilities by adding in-memory and file-system-based metrics repositories, alongside new statistical functions and Spark 3.5 compatibility.

12 changes

This period focused on expanding Deequ's capabilities by introducing a DQDL-based evaluation framework that supports row-level data quality checks and complex rule translation. Concurrently, the project added new analytical tools, including KLL sketch support for quantiles, data comparison utilities, and a codebase knowledge base generator.

8 changes

awslabs/deequ was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.