scrapy-crawl-once

The `scrapy-crawl-once` middleware for Scrapy optimizes web scraping by preventing the re-crawling of pages that have already been downloaded in previous runs. It allows users to control which requests to skip through a `request.meta['crawl_once']` flag, storing fingerprints of crawled requests in a SQLite database, thereby enhancing efficiency and reducing unnecessary load on web resources. Notably, it offers customizable settings for storage paths and default behaviors, distinguishing it from similar tools like `scrapy-deltafetch` and the built-in Scrapy HTTP cache.