deep-deep

Deep-Deep is an adaptive crawler built on the Scrapy framework that employs Reinforcement Learning techniques to optimize link selection during web crawling. Its primary use case involves learning effective crawling strategies for various scenarios using seed URLs and relevancy functions, offering features like custom scripts for different crawling tasks and integration with TensorBoard for monitoring learning statistics. Additionally, Deep-Deep supports the usage of trained link models in other crawlers, enhancing overall efficiency by allowing models to be frozen and utilized in different contexts.