gotor

GoTor is a concurrent web crawler designed for robust web scraping through the Tor network, utilizing SOCKS5 support for anonymity. Its primary use case is to crawl websites while providing a JSON crawl report and a local HTTP API for job control, making it suitable for integration with applications like TorBot. Notable features include customizable crawl depth, worker count, and the ability to randomize headers, along with built-in diagnostics for Tor connectivity and health checks.