grab-site

grab-site is a preconfigured web crawler designed for backing up websites by recursively crawling and capturing content into WARC files. It features a user-friendly dashboard for monitoring crawl status, allows dynamic ignore pattern adjustments during runtime, and includes built-in duplicate page detection to enhance efficiency. The tool is capable of managing extensive web archives, making it suitable for large-scale website preservation efforts.