Research Papers

Advances in artificial intelligence (AI) present significant risks and opportunities, requiring improved governance to mitigate societal harms and promote equitable benefits. Current incentive structures and regulatory delays may hinder responsible AI development and deployment, particularly in light of the transformative potential of large language models (LLMs). To address these challenges, we propose developing the following three contributions: (1) a large multimodal text and economic-timeseries foundation model that integrates economic and natural language policy data for enhanced forecasting and decision-making, (2) algorithmic mechanisms for eliciting diverse and representative perspectives, enabling the creation of data-driven public policy recommendations, and (3) an AI-driven web platform for supporting transparent, inclusive, and data-driven policymaking.

Deep Learning (DL) models are often black boxes, making their decision-making processes difficult to interpret. This lack of transparency has driven advancements in eXplainable Artificial Intelligence (XAI), a field dedicated to clarifying the reasoning behind DL model predictions. Among these, attribution-based methods such as LRP and GradCAM are widely used, though they rely on approximations that can be imprecise. To address these limitations, we introduce One Matrix to Explain Neural Networks (OMENN), a novel post-hoc method that represents a neural network as a single, interpretable matrix for each specific input. This matrix is constructed through a series of linear transformations that represent the processing of the input by each successive layer in the neural network. As a result, OMENN provides locally precise, attribution-based explanations of the input across various modern models, including ViTs and CNNs. We present a theoretical analysis of OMENN based on dynamic linearity property and validate its effectiveness with extensive tests on two XAI benchmarks, demonstrating that OMENN is competitive with state-of-the-art methods.

Explainable AI methods for vision models aim to identify the parts of the input that are important for the final prediction and subsequently relate these regions to human-understandable concepts. Here, we propose focusing solely on the encoder and relating its intermediate outputs to the input, instead. We introduce Neuro-Activated Vision Explanations (NAVE), a post-hoc, unsupervised, and architecture-agnostic (across CNNs and ViTs) method for extracting and visualizing internal representations from frozen vision model encoders. Specifically, NAVE clusters composite feature activations from multiple encoder depths to produce interpretable segmentation maps with controllable granularity, requiring no fine-tuning or architectural modifications. Through extensive experiments, we quantitatively demonstrate that NAVE's concepts align with input semantics and can be used in downstream tasks. We further demonstrate NAVE as an inspection tool by analyzing how training strategies and architectures affect encoder representations. Overall, our results establish NAVE as an effective tool for post-hoc model inspection and enhancing transparency in vision models. \texttt{https://github.com/Ahcene-B/NAVE}

Probe-Me-Not: Protecting Pre-trained Encoders from Malicious Probing

Ruyi Ding, Tong Zhou, Lili Su +3

Adapting pre-trained deep learning models to customized tasks has become a popular choice for developers to cope with limited computational resources and data volume. More specifically, probing--training a downstream head on a pre-trained encoder--has been widely adopted in transfer learning, which helps to prevent overfitting and catastrophic forgetting. However, such generalizability of pre-trained encoders raises concerns about the potential misuse of probing for harmful intentions, such as discriminatory speculation and warfare applications. In this work, we introduce EncoderLock, a novel applicability authorization method designed to protect pre-trained encoders from malicious probing, i.e., yielding poor performance on specified prohibited domains while maintaining their utility in authorized ones. Achieving this balance is challenging because of the opposite optimization objectives and the variety of downstream heads that adversaries can utilize adaptively. To address these challenges, EncoderLock employs two techniques: domain-aware weight selection and updating to restrict applications on prohibited domains/tasks, and self-challenging training scheme that iteratively strengthens resistance against any potential downstream classifiers that adversaries may apply. Moreover, recognizing the potential lack of data from prohibited domains in practical scenarios, we introduce three EncoderLock variants with different levels of data accessibility: supervised (prohibited domain data with labels), unsupervised (prohibited domain data without labels), and zero-shot (no data or labels available). We verify EncoderLock's effectiveness and practicality with a real-world pre-trained Vision Transformer (ViT) encoder from Facebook. These results underscore the valuable contributions EncoderLock brings to the development of responsible AI.

This chapter explores the symbiotic relationship between Artificial Intelligence (AI) and trust in networked systems, focusing on how these two elements reinforce each other in strategic cybersecurity contexts. AI's capabilities in data processing, learning, and real-time response offer unprecedented support for managing trust in dynamic, complex networks. However, the successful integration of AI also hinges on the trustworthiness of AI systems themselves. Using a game-theoretic framework, this chapter presents approaches to trust evaluation, the strategic role of AI in cybersecurity, and governance frameworks that ensure responsible AI deployment. We investigate how trust, when dynamically managed through AI, can form a resilient security ecosystem. By examining trust as both an AI output and an AI requirement, this chapter sets the foundation for a positive feedback loop where AI enhances network security and the trust placed in AI systems fosters their adoption.

The increasing integration of artificial intelligence into various domains, including design and creative processes, raises significant ethical questions. While AI ethics is often examined from the perspective of technology developers, less attention has been paid to the practical ethical considerations faced by technology users, particularly in design contexts. This paper introduces a framework for addressing ethical challenges in creative production processes, such as the Double Diamond design model. Drawing on six major ethical theories - virtue ethics, deontology, utilitarianism, contract theory, care ethics, and existentialism - we develop a "compass" to navigate and reflect on the ethical dimensions of AI in design. The framework highlights the importance of responsibility, anticipation, and reflection across both the AI lifecycle and each stage of the creative process. We argue that by adopting a playful and exploratory approach to AI, while remaining anchored in core ethical principles, designers can responsibly harness the potential of AI technologies without overburdening or compromising their creative processes.

Causal Responsibility Attribution for Human-AI Collaboration

Yahang Qi, Bernhard Schölkopf, Zhijing Jin

As Artificial Intelligence (AI) systems increasingly influence decision-making across various fields, the need to attribute responsibility for undesirable outcomes has become essential, though complicated by the complex interplay between humans and AI. Existing attribution methods based on actual causality and Shapley values tend to disproportionately blame agents who contribute more to an outcome and rely on real-world measures of blameworthiness that may misalign with responsible AI standards. This paper presents a causal framework using Structural Causal Models (SCMs) to systematically attribute responsibility in human-AI systems, measuring overall blameworthiness while employing counterfactual reasoning to account for agents' expected epistemic levels. Two case studies illustrate the framework's adaptability in diverse human-AI collaboration scenarios.

The Artificial intelligence in critical sectors-healthcare, finance, and public safety-has made system integrity paramount for maintaining societal trust. Current verification methods for AI systems lack comprehensive lifecycle assurance, creating significant vulnerabilities in deployment of both powerful and trustworthy AI. This research introduces Meta-Sealing, a cryptographic framework that fundamentally changes integrity verification in AI systems throughout their operational lifetime. Meta-Sealing surpasses traditional integrity protocols through its implementation of cryptographic seal chains, establishing verifiable, immutable records for all system decisions and transformations. The framework combines advanced cryptography with distributed verification, delivering tamper-evident guarantees that achieve both mathematical rigor and computational efficiency. Our implementation addresses urgent regulatory requirements for AI system transparency and auditability. The framework integrates with current AI governance standards, specifically the EU's AI Act and FDA's healthcare AI guidelines, enabling organizations to maintain operational efficiency while meeting compliance requirements. Testing on financial institution data demonstrated Meta-Sealing's capability to reduce audit timeframes by 62% while enhancing stakeholder confidence by 47%. Results can establish a new benchmark for integrity assurance in enterprise AI deployments. This research presents Meta-Sealing not merely as a technical solution, but as a foundational framework ensuring AI system integrity aligns with human values and regulatory requirements. As AI continues to influence critical decisions, provides the necessary bridge between technological advancement and verifiable trust. Meta-Sealing serves as a guardian of trust, ensuring that the AI systems we depend on are as reliable and transparent as they are powerful.

Differentiating between Intestinal Tuberculosis (ITB) and Crohn's Disease (CD) poses a significant clinical challenge due to their similar symptoms, clinical presentations, and imaging features. This study leverages Computed Tomography Enterography (CTE) scans, deep learning, and traditional computer vision to address this diagnostic dilemma. A consensus among radiologists from renowned institutions has recognized the visceral-to-subcutaneous fat (VF/SF) ratio as a surrogate biomarker for differentiating between ITB and CD. Previously done manually, we propose a novel 2D image computer vision algorithm for auto-segmenting subcutaneous fat to automate this ratio calculation, enhancing diagnostic efficiency and objectivity. As a benchmark, we compare the results to those obtained using the TotalSegmentator tool, a popular deep learning-based software for automatic segmentation of anatomical structures, and manual calculations by radiologists. We also demonstrated the performance on 3D CT volumes using a slicing method and provided a benchmark comparison of the algorithm with the TotalSegmentator tool. Additionally, we propose a scoring approach to integrate scores from radiological features, such as the fat ratio and pulmonary TB probability, into a single score for diagnosis. We trained a ResNet10 model on a dataset of CTE scans with samples from ITB, CD, and normal patients, achieving an accuracy of 75%. To enhance interpretability and gain clinical trust, we integrated the explainable AI technique Grad-CAM with ResNet10 to explain the model's predictions. Due to the small dataset size (100 total cases), the feature-based scoring system is considered more reliable and trusted by radiologists compared to the deep learning model for disease diagnosis.

The impressive performance of deep learning models, particularly Convolutional Neural Networks (CNNs), is often hindered by their lack of interpretability, rendering them "black boxes." This opacity raises concerns in critical areas like healthcare, finance, and autonomous systems, where trust and accountability are crucial. This paper introduces the QIXAI Framework (Quantum-Inspired Explainable AI), a novel approach for enhancing neural network interpretability through quantum-inspired techniques. By utilizing principles from quantum mechanics, such as Hilbert spaces, superposition, entanglement, and eigenvalue decomposition, the QIXAI framework reveals how different layers of neural networks process and combine features to make decisions. We critically assess model-agnostic methods like SHAP and LIME, as well as techniques like Layer-wise Relevance Propagation (LRP), highlighting their limitations in providing a comprehensive view of neural network operations. The QIXAI framework overcomes these limitations by offering deeper insights into feature importance, inter-layer dependencies, and information propagation. A CNN for malaria parasite detection is used as a case study to demonstrate how quantum-inspired methods like Singular Value Decomposition (SVD), Principal Component Analysis (PCA), and Mutual Information (MI) provide interpretable explanations of model behavior. Additionally, we explore the extension of QIXAI to other architectures, including Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, Transformers, and Natural Language Processing (NLP) models, and its application to generative models and time-series analysis. The framework applies to both quantum and classical systems, demonstrating its potential to improve interpretability and transparency across a range of models, advancing the broader goal of developing trustworthy AI systems.

How Performance Pressure Influences AI-Assisted Decision Making

Nikita Haduong, Noah A. Smith

Many domains now employ AI-based decision-making aids, and although the potential for AI systems to assist with decision making is much discussed, human-AI collaboration often underperforms due to factors such as (mis)trust in the AI system and beliefs about AI being incapable of completing subjective tasks. One potential tool for influencing human decision making is performance pressure, which hasn't been much studied in interaction with human-AI decision making. In this work, we examine how pressure and explainable AI (XAI) techniques interact with AI advice-taking behavior. Using an inherently low-stakes task (spam review classification), we demonstrate effective and simple methods to apply pressure and influence human AI advice-taking behavior by manipulating financial incentives and imposing time limits. Our results show complex interaction effects, with different combinations of pressure and XAI techniques either improving or worsening AI advice taking behavior. We conclude by discussing the implications of these interactions, strategies to effectively use pressure, and encourage future research to incorporate pressure analysis.

Explainable Artificial Intelligence (XAI) techniques are used to provide transparency to complex, opaque predictive models. However, these techniques are often designed for image and text data, and it is unclear how fit-for-purpose they are when applied to tabular data. As XAI techniques are rarely evaluated in settings with tabular data, the applicability of existing evaluation criteria and methods are also unclear and needs (re-)examination. For example, some works suggest that evaluation methods may unduly influence the evaluation results when using tabular data. This lack of clarity on evaluation procedures can lead to reduced transparency and ineffective use of XAI techniques in real world settings. In this study, we examine literature on XAI evaluation to derive guidelines on functionally-grounded assessment of local, post hoc XAI techniques. We identify 20 evaluation criteria and associated evaluation methods, and derive guidelines on when and how each criterion should be evaluated. We also identify key research gaps to be addressed by future work. Our study contributes to the body of knowledge on XAI evaluation through in-depth examination of functionally-grounded XAI evaluation protocols, and has laid the groundwork for future research on XAI evaluation.