gocolly/colly

This system is a Go-based web scraping and crawling framework that provides a flexible API for extracting data from websites. It supports concurrent and parallelized crawling through request queues and controlled concurrency, while offering robust configuration options like rate limiting, proxy rotation, and URL filtering. The library also includes utilities for debugging, caching, and handling authenticated or multipart requests, enabling users to build scalable scrapers for diverse content types.

This period focused on modernizing the Colly API by introducing functional options for collector initialization and adding comprehensive examples for various scraping scenarios. Key features included HTTP tracing, round-robin proxy switching, and debuggers, alongside a significant expansion of the examples directory to demonstrate best practices.

20 changes

This period focused on expanding the library's capabilities for concurrent crawling by introducing a request queue and an in-memory storage backend. The team also added several new scraping examples for platforms like Reddit, Shopify, and local files, while updating Go dependencies to support modern module management.

9 changes

gocolly/colly was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.