D4Vinci/Scrapling

Scrapling is a Python web scraping and crawling framework that provides a unified interface for HTTP requests, headless browser automation, and automated multi-page crawling. It features stealth capabilities to bypass anti-bot protections, an interactive CLI for testing and debugging, and an MCP server for AI integration. The system also includes a full-featured crawler engine with state management and supports integration with Scrapy.

This period focused on modernizing the project's build configuration and development workflow, alongside significant architectural improvements to the scraping engine. New capabilities were introduced, including a CLI shell, an MCP server for AI integration, and a static HTTP fetching engine with advanced proxy and ad-blocking features. Comprehensive test coverage was added for the new core modules, parsers, and fetcher implementations.

11 changes

This period focused on introducing a comprehensive crawling framework with checkpointing, scheduling, and export capabilities, alongside a new modular browser-based fetching architecture. The work also established a Scrapy integration layer and expanded test coverage for the new spiders and utility modules.

11 changes

D4Vinci/Scrapling was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.