RobotsDisallowed

RobotsDisallowed is a tool that consolidates disallowed directories from the robots.txt files of the world's top 100,000 websites, specifically sourced from the Alexa and Majestic lists. Its primary use case is to assist cybersecurity professionals in identifying potential high-value targets during web security assessments or bug bounties by providing a curated list of sensitive areas that website owners have marked as off-limits. Notable features include a cleaned-up dataset, a curated top-500 list emphasizing critical directory names, and a focus on usability for effective content discovery.