wikiextractor

WikiExtractor is a Python tool designed to extract and clean text from Wikipedia database dump files, offering functionality for both standard Wikipedia dumps and Cirrus dumps with expanded templates. Key features include multiprocessing for parallel processing, customizable output formats (JSON or XML), and options for template management and the specification of output file sizes. The script can be easily installed via PyPi and directly invoked to streamline the extraction process.