AI2030 Responsible AI Tools Marketplace

Key Takeaways

Fusion in neural networks lacks transparency: The paper identifies that most current approaches to combining information in deep learning systems are "ad hoc" and poorly understood, with little to no explainability—a significant gap for responsible AI deployment.
Mathematical foundations enable interpretability: By representing the fuzzy Choquet integral as a neural network architecture (ChIMP), the researchers demonstrate that principled mathematical approaches can make fusion processes interpretable, allowing users to understand how and why models combine different information sources.
Explainability reveals data and model quality issues: The paper shows that XAI indices derived from their approach can illuminate "the quality of our data, model, and its decisions"—suggesting that explainable fusion doesn't just improve transparency but can actively identify problems in training data or model behavior.
Improved accuracy without sacrificing interpretability: Testing on remote sensing tasks, iChIMP achieved better model accuracy while maintaining explainability, demonstrating that interpretable AI approaches don't necessarily require trading off performance.
Applicable to heterogeneous decision-making: The method successfully fuses outputs from multiple different deep learning architectures, which is relevant for real-world systems that often combine diverse models and data sources—a common scenario in high-stakes applications.

Abstract

Information fusion is an essential part of numerous engineering systems and biological functions, e.g., human cognition. Fusion occurs at many levels, ranging from the low-level combination of signals to the high-level aggregation of heterogeneous decision-making processes. While the last decade has witnessed an explosion of research in deep learning, fusion in neural networks has not observed the same revolution. Specifically, most neural fusion approaches are ad hoc, are not understood, are distributed versus localized, and/or explainability is low (if present at all). Herein, we prove that the fuzzy Choquet integral (ChI), a powerful nonlinear aggregation function, can be represented as a multi-layer network, referred to hereafter as ChIMP. We also put forth an improved ChIMP (iChIMP) that leads to a stochastic gradient descent-based optimization in light of the exponential number of ChI inequality constraints. An additional benefit of ChIMP/iChIMP is that it enables eXplainable AI (XAI). Synthetic validation experiments are provided and iChIMP is applied to the fusion of a set of heterogeneous architecture deep models in remote sensing. We show an improvement in model accuracy and our previously established XAI indices shed light on the quality of our data, model, and its decisions.

IEEE Transactions on Fuzzy Systems

Key Takeaways

Fusion in neural networks lacks transparency: The paper identifies that most current approaches to combining information in deep learning systems are "ad hoc" and poorly understood, with little to no explainability—a significant gap for responsible AI deployment.
Mathematical foundations enable interpretability: By representing the fuzzy Choquet integral as a neural network architecture (ChIMP), the researchers demonstrate that principled mathematical approaches can make fusion processes interpretable, allowing users to understand how and why models combine different information sources.
Explainability reveals data and model quality issues: The paper shows that XAI indices derived from their approach can illuminate "the quality of our data, model, and its decisions"—suggesting that explainable fusion doesn't just improve transparency but can actively identify problems in training data or model behavior.
Improved accuracy without sacrificing interpretability: Testing on remote sensing tasks, iChIMP achieved better model accuracy while maintaining explainability, demonstrating that interpretable AI approaches don't necessarily require trading off performance.
Applicable to heterogeneous decision-making: The method successfully fuses outputs from multiple different deep learning architectures, which is relevant for real-world systems that often combine diverse models and data sources—a common scenario in high-stakes applications.

Abstract

IEEE Transactions on Fuzzy Systems

Enabling Explainable Fusion in Deep Learning with Fuzzy Integral Neural Networks

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

Enabling Explainable Fusion in Deep Learning with Fuzzy Integral Neural Networks

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links