DeepSafe

DeepSafe is an all-in-one safety evaluation toolkit designed specifically for large language models (LLMs) and multimodal language models (MLLMs), integrating over 25 safety datasets along with the ProGuard evaluation model for comprehensive assessments. Its modular architecture allows for extensive customization and rapid integration of new components, promoting a streamlined, automated evaluation process that generates detailed reports to enhance AI safety research. Key features include a configuration-driven approach, facilitating user-friendly YAML-based execution and in-depth analysis capabilities, aimed at positioning itself as a significant tool in the construction of trustworthy AI systems.