SkelEx and BoundEx - Geometrical Framework for Interpretable ReLU Neural Networks
Pawel Pukowski, Joachim Spoerhase, Haiping Lu · 2024
Every ReLU Neural Network (NN) tessellates its input space into activation regions. Studying this tessellation provides insights into some of the architecture’s properties. Recent research has focused on computing the tessellation generated by the output neurons, with the main objective of counting the number of generated regions. This tessellation is achieved through the encoding of each activation region using its bounding hyperplanes. In contrast, we introduce SkelEx, a novel variation of this extraction technique that encodes the extracted regions using their vertices. Next, we introduce BoundEx, which is the first algorithm designed to transform the tessellations of the output neurons into the learned decision boundary defined via the membership polytopes. We highlight the geometric perspective on forward propagation and inference introduced by SkelEx and BoundEx, allowing for more interpretable and intuitive insights. We do so by providing: 1) explanations to the impact of earlier layers; and 2) new perspective on the existence of adversarial examples together with their categorization. The code is available on https://github.com/PawPuk/SkelEx-BoundEx.