gospider

GoSpider is a high-performance web spider developed in Go, designed to facilitate reconnaissance workflows by efficiently crawling websites. Its notable features include the ability to parse sitemaps and robots.txt, support for link finding in JavaScript files, the discovery of AWS S3 buckets, and integration with third-party URL sources such as the Wayback Machine. Additionally, it offers parallel crawling, customizable user agents, and extensive output formatting options for streamlined data processing.