xcrawl3r

`xcrawl3r` is a command-line tool that recursively spiders websites to discover URLs by actively traversing webpages and parsing files such as sitemaps and `robots.txt`. This active spidering approach distinguishes it from similar tools by revealing hidden or unindexed links, making it particularly useful for security researchers and IT professionals. Notable features include support for multiple output formats, cross-platform compatibility, and integration with automated workflows through standard input and output options.