elixir-crawly/crawly

Crawly is an Elixir-based web scraping framework that manages the full lifecycle of headless browser and HTTP-based crawling. It provides a pluggable architecture for fetching content, processing requests through configurable middlewares, and storing scraped data. The system includes a built-in HTTP API and management UI for scheduling, monitoring, and controlling spider jobs, alongside tools for generating boilerplate code and publishing packages.

This period established the foundational structure of the Crawly web scraping framework, introducing the core crawling engine, pluggable fetcher architecture, and data storage systems. It also added essential middlewares, pipelines, and a management API to support spider execution and monitoring.

14 changes

This period focused on expanding the project's usability and infrastructure by introducing a standalone management UI and corresponding EEx templates. Developers were provided with new Mix tasks and a modern Elixir quickstart example to streamline spider creation and configuration. The release environment and test coverage were also enhanced to support these new features.

9 changes

elixir-crawly/crawly was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.