Generative Expansion of Small Datasets: An Expansive Graph Approach

The gist: The Expansive Synthesis model introduces a framework to create largescale, information-rich datasets from minimal samples by using expander graph mappings and feature interpolation to preserve data distribution and feature relationships

How it works

The Expansive Synthesis approach addresses the limitation of traditional augmentation methods and GANs by providing a framework to create extensive datasets from limited data. This model leverages expander graph mappings and feature interpolation to preserve data distribution and feature relationships. The core idea is to use neural networks’ non-linear latent space, captured by a Koopman operator, to create a linear feature space for dataset expansion.

The methodology involves several key components that work in concert to achieve the expansion of a minimal dataset X into a larger and feature-rich data set X′. The process is detailed in Algorithm 1, which outlines the training procedure.

(a) Autoencoder:Koopman Feature Space

The goal is initiated by pretraining a convolutional Autoencoder (AE) uses a dataset X′′ analogous to (and distinct from) the target smaller dataset X. This phase equips the AE with understanding of general features in a typically sufficient dataset. The AE’s architecture has multiple convolutional layers, each with a set of cout−number filters and associated parameters W1, W2,., optimized to capture and encode essential data characteristics. With Wi as the i th filter parameters, a corresponding convolutional transformation is defined as conv(X′′; Wi). Intrinsically meant for dimension reduction, the AE encodes X′′ to Y′′, to retain key features as a compact and informative representation by minimizing a reconstruction loss. In adapting the pre-trained AE to a Koopman-Autoencoder (KAE), we optimize the pre-trained weights W to capture X data characteristics. Our model introduces scaling (γ) and shifting (β) parameters to finetune the KAE’ performance, to yield the following convolution operation during fine-tuning, conv(X; W) · γ + β = convX; 1 γW1 + β1,..., coutWcout + βcout. Here, γi adjusts each filter’s activation strength, controlling input feature sensitivity. Higher γi enhances filter response to specific features; βi modifies activation threshold. These filter-specific parameters enable precise adaptation to the smaller dataset X. Fine-tuning maintains generalization from the larger similar dataset while adapting to X’s distinct features.

(b) Feature Refinement: A Self-Attention Mechanism

The model addresses the incompatibility of longitudinal learning of attention maps across token sequences with images. Our proposed spatial multi-head SA mechanism to improve the resulting capture of spatial dependencies entails the partitioning of encoded data into patches, and the proximal positioning of flattened features in a stacked representation (with no positional encoding). This in fact enhances the spatial self-attention across all dimensions to focus on interacting features throughout a given image.

(c) Expander Graph Mapping

The Expansive Synthesis (ES) model follows a systematic selection of a proper feature set in the encoded space Y which are structurally combined to generate a new and larger dataset Y′ with new datapoints as row vectors y′new. The dimension expansion can be expressed as: (EG ◦ A)(Y) = EG(A(Y)) = Y′ ∈ R n’×d, ∀Y ∈ R n×d (2). The first map, A, represents the application of self-attention to extract L important features from each datapoint. The second map, EG, represents the Expander Graph Mapping that uses these features to generate new datapoints. Each size-standardized graph Gc yi ∈ Gc (set of all graphs for class c) is trained to maximize its Laplacian L spectral gap λ2 −λ1 to yield a well connected representative class code-graph Gˆc [4, 3]. An expander graph is a well connected sparse graph, with every not-too-large subset of vertices has a large boundary (hence with a capacity of diverse connections for new generation). This step corresponds to A: R n×d → R n×d×L. To avoid including nuisance and non-characteristic background information f c i, j, we proceed to first crop: f c i’, j = Att. Crop(f c i, j) prior to transferring them into Y′ using ConvT (Conversion Transformation), guided by Gˆc = (VGˆc, EGˆc). The dimension expansion constructively generates a new datapoint y′new by selecting one component from each column of the matrix of features F = [f ijc′] ∈ R n/c×L as vertices of Gˆc.

How it works

The generative ES flow is sought by regularizing global losses with other local constraints. The total loss function Ltotal is composed of four terms working in concert: Ltotal = α0Lre + α1Lce + α2LW + α3Lcov (5).

The four components of the total loss are defined as follows:

1. Reconstruction Loss (Lre): This loss ensures accuracy of the expanded dataset, defined by Lre(ϕ, θ; X) = EY ∼qϕ(Y X)[− log pθ(X Y)] + KL[qϕ(Y X)∥p(Y)]. To selectively exploit the most relevant features and further refine the intrinsic linear evolution of the nonlinear dynamics, we induce a composition map of a multi-head attention transformation on Y followed by an expander graph mapping to yield Y′ ∈ R n’×d [18]. The expansion phase entails crafting Y′ assume a higher dimension (n′) while reflecting the original dataset distribution.

2. Distributional Consistency (LW): Wasserstein distance, a key expression in Optimal Transport Loss [19], measures the cost to align the distribution of encoded data Y with expanded representation Y′. Minimizing LW ensures Y′ statistically resemble Y and preserve its geometric and topological properties, crucial to maintaining data distributional fidelity for subsequent learning tasks dependent on the data’ manifold structure [19].

3. Classification Loss (Lce): The Cross-Entropy (CE) loss is used: Lce(Cl, X, l ˜) = −X i li log(Cl(X˜ i)). This loss component ensures the expanded dataset encapsulate the original data’s structural attributes and label characteristics, while preserving essential discriminative features and preventing categorical information dilution in the course of expansion.

4. Expansion Diversity Loss (Lcov): Including covariance constraint term aims at promoting diverse and representative expanded datasets. The mathematical definition of Cov. Loss is: Lcov(Y′) = ∥Cov(Y′) − I∥2F, (9).

Results and Comparisons

The effectiveness in generating large-scale, high-performing datasets from minimal samples is demonstrated by comparing classifiers trained on these synthesized datasets against those trained on original and also on classically augmented datasets. Tables I compare classification accuracies across MNIST [3], CIFAR-10, and CIFAR100 [12]. Classifiers trained on 10 original images achieve the lowest accuracies. Expansive synthesis with row-wise self-attention improves accuracy by an average 20% over the baseline. Spatial attention further enhances performance by 6%, while adding skip connections yield an additional 8% improvement. Multi-head spatial selfattention boosts results by 4%, and incorporating Expander Graph Mapping (EGM) achieves the best results with a 6% enhancement over the previous configuration. The ablation study demonstrates that each module incrementally improves performance, with multi-head self-attention and EGM optimizing feature representation. Table II compares classification accuracy for models initialized with 10 or 100 images per class across MNIST, CIFAR-10, and CIFAR100 [3]. The expansive synthesis model consistently outperforms traditional augmentation. On MNIST, accuracy improves from 52.3% to 54.7% with 10 images per class and from 64.9% to 66.8% with 100 images per class. Similar improvements are seen for CIFAR-10 and CIFAR-100. These results demonstrate our model’s effectiveness in enhancing dataset quality and classifier performance.

Conclusion

In this paper, we introduced the Expansive Synthesis model for generating large-scale, high-performing datasets from minimal samples. It uses a Koopman feature space within an AE to transform non-linear latent space features into a linear space. Composing a multi-head spatial self-attention mechanism for improving feature extraction, with Expander graph mappings to secure feature connectivity during expansion, together with proper regularization such as Optimal transport preserving distribution integrity, and other yields a principled, effective enrichment of small datasets destined for training to learn.

1. Optimizing feature extraction through Multi-head Spatial Self-Attention enhances classification accuracy by an average of 4% over baseline configurations. This mechanism focuses on interacting features throughout a given image by partitioning encoded data into patches and proximal positioning of flattened features in a stacked representation with no positional encoding.

2. Implementing Expander Graph Mapping (EGM) ensures that the expanded dataset maintains structural and training-valued robustness properties by maintaining robust connectivity with minimal edges. This process generates new datapoints by selecting one component from each column of the feature matrix F, resulting in n L minus n unique combinations, which represents the total possible combinations excluding n original samples.

3. Integrating Optimal Transport (OT) regularization via Wasserstein distance guarantees distributional consistency for synthesized data by minimizing the cost to align the distribution of encoded data Y with expanded representation Y prime. This ensures that X prime statistically resembles X and preserves its geometric and topological properties, crucial for maintaining data distributional fidelity.

4. Utilizing a Koopman feature space within an Autoencoder transforms non-linear latent space features into a linear space, enabling flexible application in data-driven problems. This is achieved by using the AE to encode X to Y prime, then optimizing the pre-trained weights W with scaling (gamma) and shifting (beta) parameters to adapt the KAE's performance.

5. Employing a combined loss function Ltotal ensures that synthesized samples are diverse, persistent, and maintain a good cover of the original data distribution while preserving essential discriminative features. This loss is composed of reconstruction loss Lre, classification loss Lce, distributional consistency LW (Wasserstein distance), and expansion diversity loss Lcov (covariance constraint).

Limited data availability in machine learning significantly impacts performance and generalization. Traditional augmentation methods enhance moderately sufficient datasets. GANs struggle with convergence when generating diverse samples. Diffusion models, while effective, have high computational costs. We introduce an Expansive Synthesis model generating large-scale, information-rich datasets from minimal samples. It uses expander graph mappings and feature interpolation to preserve data distribution and feature relationships. The model leverages neural networks' non-linear latent space, captured by a Koopman operator, to create a linear feature space for dataset expansion. An autoencoder with self-attention layers and optimal transport refines distributional consistency. We validate by comparing classifiers trained on generated data to those trained on original datasets. Results show comparable performance, demonstrating the model's potential to augment training data effectively. This work advances data generation, addressing scarcity in machine learning applications.