recode

ReCode is a benchmark tool designed for the robust evaluation of code generation models, providing over 30 types of semantic perturbations to docstrings, function names, and code. Its primary use case is to assess the resilience and robustness performance of both public and custom code generation models against various natural semantic changes, offering comprehensive analytical comparisons across different models. Key features include automatic dataset perturbation generation, configuration flexibility via `config.json`, and the ability to run evaluations with well-defined metrics, enhancing the usability for model robustness analysis.