Seeking Interpretability and Explainability in Binary Activated Neural Networks

Binary activated networks offer genuine interpretability: Unlike black-box neural networks, binary activated networks (which output only 0 or 1) are inherently simpler and more transparent, making it possible to understand how they arrive at predictions for regression tasks on tabular data.

Feature importance can be rigorously quantified: The paper demonstrates how to efficiently compute SHAP values for these networks to measure the relative importance of individual features, hidden neurons, and even specific weights—providing stakeholders with concrete explanations for model decisions.

Simpler models are built automatically without over-engineering: The proposed greedy algorithm constructs networks incrementally (one neuron at a time) rather than requiring architects to guess the right size upfront, resulting in models that are "not needlessly complex for a given task"—reducing the risk of unnecessary opacity.

Mathematical guarantees support trustworthiness: The paper provides formal expressiveness guarantees for binary activated networks, offering theoretical backing that these simplified models can still perform their intended function reliably.

Interpretability and performance need not be mutually exclusive: By focusing on tabular data and regression tasks where binary networks are well-suited, this work demonstrates that organizations can achieve both model explainability and practical utility without sacrificing one for the other.