Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences

Finetuned models leave detectable fingerprints: When LLMs are narrowly finetuned on specific domains (like false information or unusual behaviors), the changes create identifiable patterns in the model's internal activations that can be discovered using relatively simple analysis tools, making it difficult to hide what a model was trained on.

These traces reveal training objectives: Researchers were able to use an AI agent to understand what domain a model was finetuned on by analyzing these activation differences, suggesting that a model's hidden training objectives and specialized knowledge are "readable" from its internal workings—raising transparency and accountability concerns.

The effect spans multiple model types and sizes: The findings hold across different LLM architectures (Gemma, LLaMA, Qwen) and scales (1B to 32B parameters), indicating this is a fundamental characteristic of finetuning rather than an isolated quirk, making it a widespread phenomenon to consider.

Overfitting appears to be the culprit: The researchers found that mixing in original pretraining data during finetuning substantially reduces these detectable biases, suggesting the traces stem from overfitting—though some residual risks may persist even with this mitigation.

Implications for model safety and misuse: The ability to detect and interpret what models were finetuned on has dual implications: it could help identify when models are secretly trained for harmful purposes, but also highlights risks if adversaries use this knowledge to better hide problematic finetuning.