Research Papers

From Interpretability Methods to Interpretable Models

Julien Colin, Nuria Oliver, Thomas Serre

More than a decade in, explainable AI (XAI) for computer vision has assembled a mature toolbox: attribution, feature visualization, concept-based, and circuit-based methods. Yet almost all of the field's effort has gone into building and comparing these methods, and little into the question they were meant to answer---how interpretable are our models, and are we making progress as they evolve? We argue for shifting the field's focus from methods to models, along two complementary lines. One is already within reach: existing tools let us characterize and compare what different models represent and compute. The other is harder, and largely neglected: whether a model can actually be understood by the humans who rely on it---the independent evaluators on whom trust and certification depend, not the experts confirming what they already expect. It can only be measured, not inferred. We review why the toolbox is mature enough to support both, survey the thin body of work comparing models, draw a parallel to systems neuroscience, and close with a model-centric XAI agenda.

Diffusion TV is an interactive AI art installation that offers a tangible and embodied experience of diffusion models through a modified CRT TV. By physically manipulating the TV's antenna, audiences control the clarity of AI-generated images and sounds, metaphorically enacting the denoising process that underlies diffusion-based generation. Using the tuning knob, participants switch between three channels featuring AI-generated animals from the Past (extinct species), Present (endangered species), and Future (speculative creatures), situating the interaction within a temporal and ecological narrative. Through continuous audiovisual feedback and physical interaction, Diffusion TV foregrounds the generative process over final outputs, allowing audiences to explore intermediate states as experiential material. Rather than providing explicit technical explanation, the work presents an alternative, embodied mode of explainable AI that invites exploratory engagement with and reflection on generative technologies.

Requirements engineers for agentic-AI domains face challenges in evaluating, specifying, and operationalizing safe autonomy. Mainstream frameworks, such as Goal-Oriented Requirements Engineering (GORE), lack mechanisms to systematically address these challenges under epistemic uncertainty. We contribute an approach that builds on GORE to model and simulate safe autonomy in agentic-AI systems. We introduce a novel Regret-Dominance Mechanism (MS-RGR) to operationalize safe autonomy. MS-RGR uses two signals: epistemic surprise (novelty detection) and cognitive regret (evaluative risk) to address the trilemma problem: should the agent operate in routine autonomy, undergo reflective reasoning, or escalate to human? We instantiate MS-RGR in elderly care monitoring and autonomous driving. A 100-seed stochastic simulation shows MS-RGR reduces silent failures to near-zero and detects risk approximately 17.5 times faster than a sensor-only baseline, remaining formally traceable via LTL safety properties. A retrospective proxy instantiation applying the DRI gate post-hoc over execution traces from 208 AGENTHARM scenarios across seven LLMs shows the gate improves harmful-task refusal only for models with strong baseline safety (over 80% pre-gate refusal, e.g., 84.1% to 90.9%), indicating MS-RGR amplifies rather than substitutes for model-level safety training. We discuss threats to validity, positioning MS-RGR as initial feasibility evidence for design-time safety constraints in agentic-AI requirements engineering.

Predicting olfactory qualities from molecular structure is an open problem in chemoinformatics. Although linear models can link molecular features to odor descriptors, they often fail when extrapolating to novel chemical scaffolds, extreme molecular weights, or complex odor mixtures. To address this, we introduce GraphNOSE, an open-source graph transformer framework that predicts multi-label odor descriptors from simplified molecular-input line-entry system (SMILES) strings for single molecules and binary mixtures. By integrating positional and structural encodings within a transformer-based graph architecture, GraphNOSE achieves strong performance with six times fewer parameters than standard graph neural network (GNN) baseline while consistently outperforming linear models, molecular language model embeddings, molecular fingerprints, and baseline GNNs by an average area under the ROC curve (AUROC) margin of 4.52% (p < 0.01). GraphNOSE achieves an AUROC of 84% on out-of-distribution compounds (OODs). This exceeds the current state-of-the-art GNN for OOD in olfaction (Open-POM: 81%, p < 0.001), and identifies conditions under which linear models empirically fail. Finally, we apply XAI (explainable AI) methods to identify which substructures and molecular features drive odor predictions, yielding insights consistent with chemical intuition and grounded in the model's learned representations. Together, these results establish GraphNOSE as a scalable and interpretable architecture for olfactory prediction that generalizes to structurally distinct compounds underrepresented in current perceptual databases.

XAI-SDN: An Explainable Entropy-Guided Machine Learning Framework for Real-Time DDoS Detection in Software Defined Networks

Adeel Ahmad, Ali Akarma, Ahmad Ali +2

One of the biggest risks faced by Software Defined Networks (SDN) is the Distributed Denial of Service (DDoS) attack in which a compromised controller can make an entire network unusable. To address these challenges, we suggest an entropy-guided machine learning framework, called XAI-SDN, for real-time DDoS detection in SDN environments which is lightweight and explainable. The framework extends the flow features extracted by CICFlowMeter with eight Shannon entropy metrics obtained by an $\mathcal{O}(1)$ rolling algorithm and uses a Random Forest classifier with SHAP TreeExplainer for providing transparency at the prediction level. On a fixed temporal split, XAI-SDN achieves an accuracy of 99.9987\%, a macro F1-score of 99.9621\%, and an AUC-ROC of 1.0000 on the full 3.59 million flows of the CIC-DDoS2019 SYN benchmark. The pipeline sustains 0.0165~ms per flow (60{,}606 flows/s) without the use of SHAP and 0.5122~ms per flow (1{,}953 flows/s) with full support of SHAP under the 99.14\% prevalence of DDoS traffic, which is a step towards achieving a balance between the detection performance and operational transparency in next-generation SDN security.

Scientific and clinical decision-making depends on evidence from the primary literature, but existing standards for representing that evidence (FHIR Evidence, ECO, SEPIO, and the GA4GH Genomic Knowledge Standards) are oriented toward clinical-trial workflows, evidence codes, or single-variant assertions, and do not capture the fine-grained, domain-specific structure of claims in basic and pre-clinical research. We introduce a semantic model for scientific evidence with three core classes, specialize it for genetics, align it structurally to FHIR Evidence with a SEPIO-anchored credibility decomposition, and attach a compact dimensional vocabulary whose conditional-activation rules are validated by a SHACL schema for the implemented constraints. Using clinical variant interpretation as the driving use case, we evaluate the model through a human-AI annotation pilot over six genetics papers, yielding 28 evidence items and 95 source-anchored assertions, with a workflow that keeps curator-authored reference annotations distinct from AI-drafted annotations. Treating the pilot as a feasibility study rather than a benchmark, we argue that the model is a useful increment toward trustworthy, AI-ready infrastructure for variant interpretation: a reference data model and validation schema for representing genetic evidence.

With the rapid proliferation of generative models on Machine Learning as a Service (MLaaS) platforms, reliably tracing the provenance of synthetic media without modifying generator architectures or parameters remains a major challenge. In this work, we propose a self-referential retrosynthesis framework for explainable AI provenance forensics under a fixed-generator setting. The framework leverages a jointly optimized encoder-decoder pair to implement a self-embedding mechanism that enables round-trip consistency verification. During inference, client inputs are first encoded and then processed by the generator to produce outputs with high visual fidelity. For forensic verification, the consistency between the resynthesized image and the query image is analyzed to determine whether the image originates from the target generative model. Our approach eliminates the need for watermark embedding or modifications to the generation process. Experimental results show that images generated from encoded inputs maintain visual quality comparable to original generator outputs, while decoded images reliably trace back to their corresponding source inputs. Furthermore, the framework provides interpretable evidence for generative content provenance, establishing a practical tool for explainable generative AI forensics.

Autonomous robots powered by deep learning face a fundamental auditability challenge: when incidents occur, investigators cannot reconstruct why the system made specific decisions. This paper presents TRACE (Transparent Reasoning Architecture for Credible Execution), a decision framework that ensures every autonomous action can be traced back to sensor evidence through documented causal chains. The framework organizes decision-making into four auditable layers: Semantic Perception for evidence-grounded entity recognition, Belief Reasoning for probabilistic state estimation with causal graphs, Action Synthesis for constraint-aware planning with counterfactual documentation, and Execution Verification for compliance monitoring. TRACE is model-agnostic yet designed to integrate learning-based perception modules (CNNs, transformers) while preserving decision-level auditability. We evaluate the framework using three objective metrics: Evidence Traceability (sensor-to-decision linkage), Decision Reconstructability (post-hoc analysis capability), and Temporal Continuity (audit trail completeness). Experimental evaluation on warehouse robot navigation demonstrates that TRACE achieves 98.6% evidence traceability, 99.0% temporal continuity, and 98.1% decision reconstructability across 500 simulated decision cycles. Post-hoc methods like LIME provide feature attributions but lack the artifact structure needed for decision-level reconstruction. The framework addresses EU AI Act requirements for high-risk system transparency and contributes to Explainable AI for safety-critical autonomous systems.

In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?

Koshiro Aoki, Ryota Takatsuki, Gouki Minegishi +2

Whether large language models (LLMs) can control their own internal representations matters for both machine metacognition and AI safety. A recent study applied neurofeedback to LLMs and claimed that they can control their internal representations. However, the reported control may rely on superficial mechanisms rather than genuine internal access because the control targets in that study are not privileged, meaning that a third party can infer them from the prompt. We redesign the neurofeedback paradigm for LLMs so that the control target satisfies the privileged access requirement, which is closer to neurofeedback experiments in human cognitive neuroscience. Under this stricter setting, the models do not demonstrate reliable control over privileged internal representations, suggesting that previously reported control cannot exclude the possibility that it relies on superficial mechanisms. Our results indicate that rigorous assessments of metacognition in LLMs require evaluation methods that demand privileged access.

Network slicing in multidomain 5G environments introduces critical challenges in guaranteeing Service Level Agreements (SLAs) under dynamic resource variability and heterogeneous service requirements. This article presents TriSLA, a closed-loop, preventive, SLA-aware architecture designed to evaluate feasibility at request time and continuously ensure SLA compliance during operation. The architecture combines ontology-driven semantic intent interpretation, multidomain machine learning feasibility risk inference, Explainable Artificial Intelligence (XAI) feature attribution, and closed-loop runtime SLA assurance into a unified operational pipeline. A fully operational prototype was evaluated in a multi-node cloud-native environment integrating Radio Access Network (RAN), Transport Network (TN), and 5G Core (5GC) domains with real-time telemetry collection. Experimental evaluation demonstrates that TriSLA guarantees a 100% SLA satisfaction rate for admitted slices, completely eliminating post-deployment violations compared to reactive (51.2%) and static threshold (80.4%) admission baselines. The predictive feasibility assessment achieved a classification accuracy of up to 99.51% (98.68% for the default explainable Random Forest classifier), enabling preventive admission decisions before infrastructure commitment. Furthermore, the cognitive admission pipeline introduces minimal processing overhead, requiring 25.37 ms for ontology-driven semantic parsing and 231.66 ms for XAI-assisted feasibility inference. Concurrently, the closed-loop assurance engine resolves 100% of runtime telemetry anomalies within a 4.22 s recovery cycle. These results demonstrate that TriSLA provides reliable, explainable, transparent, and preventive SLA management through integrated predictive admission and closed-loop runtime assurance for next-generation 5G networks.

Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment

Deepak Pandita, Christopher M. Homan

Humans play a vital role at every stage of AI development, from data collection and curation to model development and evaluation. However, humans often disagree with each other and sometimes with themselves over time. It is essential to take disagreement into account when building human-centered AI systems, especially in domains where it is prevalent, such as AI safety, content moderation, or sentiment analysis. Disagreement often arises from subjective human opinion and can vary with one's identity, beliefs, and social environment. Despite this, current LLM evaluation approaches frequently rely on aggregating labels (often via plurality voting) to represent consensus, thereby obscuring minority perspectives. By failing to account for human disagreement, these evaluation methods contribute to the reproducibility crisis in AI. Human feedback is also crucial for ensuring that AI systems align with human values. For these systems to be trustworthy, it is critical to ensure that they reflect diverse human values and perspectives. In this thesis proposal, we present a human-centered and perspective-aware framework for reproducible ML evaluation and AI alignment.

Wi-Fi Channel State Information (CSI) has emerged as a key enabler for device-free Human Activity Recognition (HAR), enabling low-cost, unobtrusive sensing using existing communication infrastructure. However, Deep Learning (DL) models trained on CSI data often struggle to generalize across users, environments, and device setups due to the context sensitivity of wireless propagation. Despite this challenge, limited attention has been devoted to understanding model decisions and generalization failures. This paper introduces XAI2CSI, a framework that leverages eXplainable Artificial Intelligence (XAI) to analyze DL-based CSI sensing systems. XAI2CSI employs SAGE, a model-agnostic explainability method, to quantify temporal, spectral, and spatial CSI contributions to HAR decisions under nominal and cross-context evaluations on IEEE 802.11ax data. Our analysis reveals that the considered DL model exhibits limited robustness to unseen conditions due to an over-reliance on context-specific CSI patterns, causing models to misinterpret the underlying signal dynamics when deployment conditions change. The proposed methodology and findings provide a reference framework to explore alternative solutions and guide the development of robust, transparent Wi-Fi sensing systems.