Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewing

Mechanistic interpretability findings lack standardized auditing systems, which prevents their adoption in safety-critical applications like medical AI and autonomous systems where stakeholders need to verify validity and correctness guarantees.

Conflicting research conclusions demonstrate the problem concretely: the paper cites cases where two studies reached opposite conclusions about the same neural network behavior, and a third study found both were "partially correct but incomparable due to methodological inconsistencies"—highlighting how methodological gaps undermine trust in findings.

The field needs a "Collaborative Reviewing Platform" that goes beyond traditional peer review to continuously organize meta-science results including critiques, negative results, reproductions, and replications that currently fall outside formal papers, allowing ongoing discussion and revision.

Expert-verified guidelines and protocols should be developed from successful practices on collaborative platforms to standardize auditing procedures and improve efficiency, making it easier for stakeholders to evaluate mechanistic interpretability claims.

Source-based auditing systems are needed to track which specific arguments and evidence support particular claims about AI systems, creating transparency about the dependencies underlying mechanistic interpretability findings used in high-stakes decisions.