Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantization

Scale-invariant statistics like cosine similarity can be misleading without reporting the underlying noise floor: The paper demonstrates that a published cosine similarity of 0.996 between full-precision and quantized AI safety features cannot be interpreted as evidence of preservation without knowing the sample size (n) and class separation (ρ) used to compute it—metrics that compression studies typically omit.

Current interpretability transfer studies lack critical methodological transparency: Researchers calibrate interpretability artifacts on full-precision model weights, deploy them on quantized versions, and report similarity metrics without the dimensionless parameter κ = nρ²/d needed to determine whether observed agreement exceeds random noise, making it impossible to verify their claims.

Quantization can cause meaningful directional changes in safety-critical features: Testing on Qwen2.5-1.5B-Instruct revealed that at INT4 quantization, refusal directions rotated beyond the estimator's own noise floor, suggesting the model's safety mechanisms may not survive aggressive compression despite high reported similarity scores.

Proper statistical validation requires comparing against quantized-model baselines, not full-precision ones: The paper argues that null hypotheses should be measured within the quantized model itself rather than assuming the low-bit estimator has the same variance as the full-precision version—an assumption that should be tested, not assumed.

At INT8 quantization, no detectable movement was found, but this is not equivalent to proof of preservation: The authors emphasize the distinction between failing to detect change and proving equivalence, highlighting the need for careful interpretation of negative results in compression studies.