trafilatura

Trafilatura is a Python package and command-line tool designed for web crawling, downloading, and extracting structured text data from raw HTML. It features advanced capabilities such as support for sitemaps and feeds, parallel processing, and robust extraction of key elements like metadata and main text, while offering multiple output formats including TXT, JSON, and Markdown. The tool is modular and efficient, aiming to minimize noise and enhance content quality, making it widely adopted in various projects by notable companies and institutions.