Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

Existing AI evaluation methods for math are inadequate: Current approaches to testing LLMs on mathematical proofs rely either on checking only final answers or require expensive expert grading, creating a significant gap in our ability to trustworthily measure whether AI systems can actually reason through complex proofs step-by-step.

Mask-Proof provides a scalable solution for verification: The new pipeline automatically converts real proofs into testable tasks by masking key steps, allowing researchers to evaluate whether AI models can reconstruct missing mathematical reasoning—achieving 96.8% agreement with human experts while being reproducible and cost-effective.

Advanced reasoning models show meaningful improvements: When tested on the resulting Mask-ProofBench dataset of 292 problems, reasoning-enhanced models outperformed standard models by 12-27%, suggesting that specialized AI architectures do genuinely improve mathematical reasoning capabilities.

Reproducibility and transparency are critical for AI in science: By making the benchmark, expert annotations, and code publicly available, this work demonstrates how responsible AI development requires not just better models, but also standardized, verifiable evaluation methods that enable independent validation and comparison.

Step-level evaluation matters more than final answers: The focus on intermediate reasoning steps rather than just correct conclusions is essential for trustworthy AI assistance in research, since understanding how an AI reaches an answer is crucial before relying on it for scientific progress.