TX-Ray: Quantifying and Explaining Model-Knowledge Transfer in (Un-)Supervised NLP

TX-Ray enables tracking of knowledge transfer across training stages: The method allows researchers to visualize and quantify how NLP models build knowledge abstractions during pretraining, transfer them to new domains, and adapt them during fine-tuning—addressing a gap in existing explainability methods that only focus on individual predictions.

Neural networks learn linguistic structure automatically: The research found that early stages of self-supervised training automatically learn linguistic abstractions like parts-of-speech without explicit instruction, providing evidence that language models develop meaningful linguistic knowledge during pretraining.

Explainability insights can improve model performance: TX-Ray-based pruning (removing less important neurons) improved test set generalization comparable to Lottery Ticket pruning methods, suggesting that understanding which neurons matter most can lead to more efficient and better-performing models.

Current AI explainability methods have blind spots: Existing NLP explainability approaches focus on explaining individual decisions rather than understanding how models accumulate and transfer knowledge, which TX-Ray addresses by adapting computer vision visualization techniques to track learning across training phases.

Fine-grained neuron analysis enables human oversight: By expressing neurons as "feature preference distributions," TX-Ray provides interpretable information that can guide human analysis of model behavior, supporting more informed decisions about model development and deployment.