TopoMap: A Feature-based Discriminator of the Topographical Regions in the Test Input Space

The gist Testing Deep Learning (DL)-based systems is an open challenge, and this paper introduces TOPOMAP, an explicit topographical map of the input feature space designed to explore all regions that may potentially trigger a misbehaviour of a DL model<ref:2509.03242#pg2>.

How it works

TOPOMAP is described as a black-box and model-agnostic approach that relies solely on features that characterise the input space<ref:2509.03242#pg2>. The process involves several key steps to create the topographical map<ref:2509.03242#pg7>.

1. Input Embedding Computation: The first step is the automated extraction of an input embedding that has lower dimensionality than the input space, while still retaining the key features that characterise each input and trigger a specific behaviour of the DL system under test<ref:2509.03242#pg7>. Various embedding models are considered, including Principal Component Analysis (PCA), Uniform Manifold Approximation and Projection (UMAP), t-distributed Stochastic Neighbour Embedding (t-SNE), Singular Value Decomposition (SVD), Linear Discriminant Analysis (LDA), Isomap, Autoencoder (AE), and Word2vec<ref:2509.03242#pg11>.

2. Input Data Clustering: Inputs are then grouped according to the features represented in their embeddings using unsupervised learning methods, which infer the membership of the instances to a certain region (i.e., a cluster), based on the characteristics of the lower-dimensional data<ref:2509.03242#pg11>. Algorithms considered include K-means, BIRCH, HDBSCAN, and affinity propagation<ref:2509.03242#pg12>.

3. Optimal Map Selection: To automate the selection of the optimal set of clusters by mimicking human assessment, a Deep Neural Network (DNN) is used to classify pairs of clusters based on their features<ref:2509.03242#pg11>. This DNN is trained to classify inputs according to pseudo-labels (regions) produced by clustering<ref:2509.03242#pg12>. The goal is to select the clustering configuration that produces the highest DNN prediction accuracy on the test set<ref:2509.03242#pg7>.

Evaluation and Effectiveness

The effectiveness of TOPOMAP is evaluated through mutation analysis to assess whether the clusters in our topographical map allow for an effective selection of mutation-killing inputs<ref:2509.03242#pg12>. Experimental results show that our approach outperforms random selection by 35% on average on killable mutants and by 61% on non-killable ones<ref:2509.03242#pg18>.

The research addresses four main questions:

RQ1 (Discrimination): How useful is the proposed approach to select the optimal clustering configuration, i.e., the clustering configuration that better discriminates the test input instances?

The results show that for each subject, TOPOMAP can generate a map with meaningful and distinguishable clusters<ref:2509.03242#pg18>. The best configuration is reported in the rows ‘Embedding’ and ‘Clustering’, while the row ‘k∗’ shows the optimal number of clusters identified for each corresponding configuration<ref:2509.03242#pg17>.

RQ3 (Fault Detection): Are clusters indicative of failure regions in DL systems?

The results show that TOPOMAP is highly effective in locating input space regions with high density of inputs sharing features that expose faults in DNN models<ref:2509.03242#pg19>. The average advantage across subjects of TOPOMAP compared to a random sampling is 35% on killable mutants and 74% on non-killable ones, using on average just 9% of the input data<ref:2509.03242#pg19>.

Comparative Analysis

The paper compares TOPOMAP with state-of-the-art techniques like DeepHyperion to assess complementarity<ref:2509.03242#pg20>. The similarity score between the maps generated by TOPOMAP and DeepHyperion is 0.605 ± 0.006, implying a low similarity, hence complementarity, between the two maps<ref:2509.03242#pg20>.

Furthermore, in mutation analysis, aggregations gathered by TOPOMAP are generally 9% denser of contributors than DeepHyperion<ref:2509.03242#pg17>. The average gain across subjects when using our approach against random input selection is 35% on killable mutants and 74% on non-killable ones<ref:2509.03242#pg19>.

Human Assessment Correlation

The human study suggests that participants’ judgments are consistent with those of TOPOMAP when separating inputs into map regions, particularly in cases where the dataset images are easy to interpret<ref:2509.03242#pg18>. For the MN data scenario, empirical evidence suggests a strong correlation for both hA,Bacc and hA,Bdiff with all the DNN-based metrics<ref:2509.03242#pg18>.

In conclusion, TOPOMAP is a black-box model-agnostic approach that generates a topographical map of the DNN input feature space to identify regions sharing features likely to expose faults in a DL model<ref:2509.03242#pg19>. It also proposes an automated technique to configure the dimensionality reduction and clustering methods used in our approach, to optimize the discriminative power of the resulting map regions<ref:2509.03242#pg8>.

The research addresses four main questions:

RQ1 (Discrimination): How useful is the proposed approach to select the optimal clustering configuration, i.e., the clustering configuration that better discriminates the test input instances?

The results show that for each subject, TOPOMAP can generate a map with meaningful and distinguishable clusters<ref:2509.03242#pg18>.

1. Automated Topographical Map Generation: The system can generate a black-box and model-agnostic topographical map of input feature space by first computing an embedding using dimensionality reduction (e.g., PCA, UMAP) followed by unsupervised clustering (e.g., K-means, HDBSCAN).

2. Optimal Configuration Selection: A Deep Neural Network can be used to automatically select the optimal topographical map configuration among all possible combinations of embedding and clustering techniques based on maximizing DNN prediction accuracy on the test set.

3. Feature Space Exploration: The system explores all regions that may potentially trigger a misbehaviour of a DL model by creating explicit topographical maps, ensuring no region in the feature space is neglected, unlike existing approaches that focus only on specific failure-inducing features.

4. Mutation Analysis for Fault Localization: The system performs mutation analysis to assess whether clusters in the topographical map allow for an effective selection of mutation-killing inputs, potentially achieving a killing probability of 100% with just 9% of the inputs.

5. Discriminative Region Identification: The system identifies fault-revealing features by aggregating clusters based on their ability to kill mutants, allowing developers to focus testing effort on map regions that expose misbehaviours of the model under test.

6. Configuration Optimization: The system can dynamically select the optimal number of clusters k for parametric clustering algorithms by simulating a human assessment process using a DNN and stopping when accuracy gains plateau, rather than relying on arbitrary metrics like Silhouette score or elbow rule.

7. Human-DNN Alignment Assessment: The system can assess whether the topographical regions selected by the DNN assessor are distinguishable according to human assessors by computing Pearson correlation coefficients between human accuracy and DNN-based metrics, revealing which features are most interpretable.

Testing Deep Learning (DL)-based systems is an open challenge. Although it is relatively easy to find inputs that cause a DL model to misbehave, the grouping of such inputs by features that make them problematic for the DL model under test is largely unexplored. In this work, we propose TopoMap, a model-agnostic approach that relies solely on features that characterise inputs to construct a partitioning of the input space where each region represents a specific set of distinguishable input features. Such a partitioning gives users a possibility to identify and analyse areas that are under-represented, show higher failure rates, or exhibit prediction changes after a model update. The identified regions of interest can then be used for additional data generation or can help prioritise already generated inputs that require labelling efforts. To discriminate inputs according to the specific features they share, we first apply dimensionality reduction to obtain input embeddings that are then subjected to clustering. Each DL model might require specific embedding computations and clustering algorithms, and specific configurations of the two, to achieve a meaningful separation of inputs into discriminative groups. We propose a novel way to evaluate alternative configurations of embedding and clustering techniques. We use a deep neural network (DNN) as an approximation of a human evaluator who could tell whether a pair of clusters can be discriminated based on the features of the included elements. The evaluation results show that the maps generated by TopoMap consist of distinguishable and meaningful regions. In addition, we evaluate the effectiveness of TopoMap using mutation analysis. In particular, we assess whether the clusters in our topographical map allow for an effective selection of mutation-killing inputs.