heritrix3

Heritrix is an open-source, web-scale archival-quality web crawler developed by the Internet Archive, designed primarily for collecting and preserving digital artifacts for future research. It features respect for `robots.txt` directives and META nofollow tags, ensuring ethical crawling practices, and offers extensive documentation for configuration and operational guidance, as well as a REST API for integration. The tool is extensible and supports customization through various modules and a comprehensive developer manual.