Interpretable Self-Supervised Learning via Representer Landmarks and Nyström Approximation

SSL models can be interpreted through "Representer Landmarks": Rather than treating self-supervised learning models as black boxes, KREPES makes their learned representations interpretable by identifying which unlabeled training examples most influence the model's decisions—enabling direct auditing of what the model has learned without needing labeled data.

The framework revealed algorithmic bias in real datasets: When applied to the Adult-1M dataset, KREPES detected that the SSL model was using demographic proxies (like age or gender) to predict income, demonstrating the practical value of this interpretability approach for identifying hidden biases that might otherwise go undetected.

New transparency metrics quantify representation quality: The paper introduces three novel metrics—Sample-Specific Influence Score, Concept-Conditioned Influence Score, and Feature Alignment Gap—that provide measurable ways to assess how transparent and trustworthy a self-supervised model's learned representations are.

The approach scales to real-world datasets: By incorporating Nyström approximation techniques, KREPES can analyze massive datasets with over 1 million samples (like ImageNet-1K and Adult-1M), making interpretability practical for production-scale AI systems rather than just small research datasets.

Works across multiple SSL methods: KREPES provides a unified framework for interpreting different self-supervised learning approaches (SimCLR, BYOL, VICReg), suggesting the interpretability insights could apply broadly across the SSL landscape.