cc-downloader

CC-Downloader is a command-line tool written in Rust designed for efficiently downloading Common Crawl data, featuring a polite retry mechanism that employs exponential backoff and jitter to mitigate infrastructure overload. Key functionalities include concurrent downloading with the capability of processing multiple files simultaneously, preservation of the folder structure, and cross-platform compatibility. The tool is optimized for use outside of AWS, ensuring all requested files are reliably downloaded.