urlgrab

urlgrab is a Go-based utility designed for web crawling that efficiently discovers additional links on websites, including those utilizing JavaScript rendering. It offers notable features such as customizable parallelism, the ability to render JavaScript for Single Page Applications, and various configuration options for controlling crawling behavior, such as request delays and recursion depth. This tool is particularly useful for security researchers and developers looking to gather comprehensive link data from modern web applications.