Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget

arXiv:2510.21286 · cs.LG · Submitted 2025-10-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Adaptive Data Selection for Text Classification".

Jane: The gist: DVC, a novel budget-aware method for evaluating and selecting data for MLP training, decomposes data contribution into Layer Value Contribution (LVC) and Global Value Contribution (GVC),

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper right now titled "Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget". Basically, it tackles the hard problem of picking the best training data from tons of different sources when you have a limited budget.

Jane: That sounds really practical. I mean, when you're dealing with text classification and maybe you've got millions of samples scattered across different domains, how do you know which ones are actually worth using?

Lu: The core idea here is that we need to evaluate the potential contribution of data in the dynamic context of MLP training. It has to account for how representation learning works, where shallow layers capture basic features and deeper layers learn more abstract concepts.

Meng: So it's not just about looking at the final output accuracy, right? It has to look inside the hidden layers too, because that's where a lot of the signal lives.

Lalam: I see this paper is proposing DVC, Data Value Contribution. It claims they decompose data value into layer-wise and global contributions using six specific metrics.

Tom: Exactly. They use this formula: DVC(x, y) equals the sum over layers L of a lambda L times the Layer Value Contribution plus a global term mu times the Global Value Contribution, with weights that adapt based on normalization constraints.

Jane: That sounds complicated to calculate, but what does that mean for someone just trying to train an MLP? Are we talking about something super slow?

Lu: They are actually proposing scalable selection algorithms. They include things like gradient caching, locality-sensitive hashing for similarity assessment, and an online estimation approach.

Meng: Because when you're shipping models to production, speed matters a lot. If this method is too slow to evaluate samples, it won't help us at scale. What is the computational overhead they claim?

Lalam: The paper states that the total computational complexity ends up being O(budget times n rho log n times d), where rho is less than one and d is the dimension of the data. This means it scales sublinearly with dataset size, which is better than linear scaling.

Paper summary: Tom: That sublinear scaling is a big deal. It means doubling your data doesn't double your evaluation cost, which helps immensely with massive datasets. But they also mention they established convergence and generalization bounds that formally link higher data value to improved model performance.

Jane: So, what's the actual problem they are solving? What did the existing methods miss that this DVC approach manages to fix?

Lu: Existing methods, like coreset construction or data Shapley values, they often oversimplify nonlinear transformations or ignore informative intermediate representations in hidden layers. That’s a key limitation they are addressing.

Meng: And what about the practical side? They have these six metrics: Quality Assessment Ql for stability, Relevance Assessment Rl for alignment with learning direction, and Distributional Diversity Assessment Dl to find novel information. Does that mean we need to track all that data every time?

Lalam: It’s a lot of components. For example, the Conditional Uncertainty Assessment CU looks at both output-level uncertainty and intermediate representation uncertainty. And Training Stability Assessment TS checks the temporal consistency of a sample's loss over a sliding window.

Tom: I like how they set up these metrics to capture quality, relevance, and distributional diversity across different granularities. It’s a way to balance those competing needs without just picking one simple score.

Jane: Okay, so if we look at the experimental validation described in the paper, what kind of performance gains are we actually seeing?

Lu: They show that at twenty percent source retention, DVC achieves a mean macro F1 of seventy-five point four across eight different classification and domain-transfer tasks. That’s a substantial jump over baselines.

Meng: Seven fifty-four across eight tasks is pretty impressive, especially when you have to worry about noisy or imbalanced settings where real traffic data often lives. It says it works well in those difficult scenarios.

Tom: And they compare it against shared final training, showing a gain of zero point seven over that baseline and a gain of four point three over random selection. Plus, an equal-budget tuning study showed a zero point six gain over calibrated quality-only selection.

Paper summary: Jane: So, for someone just trying to get good results without spending all their time tuning hyperparameters, this method seems to offer a principled way to select data. What's the big picture implication here?

Lu: The implication is that by being budget-aware and using these hierarchical metrics, we can systematically evaluate data contribution in a way that links it directly to improved model performance through theoretical guarantees.

Meng: For deployment, this means if we only have a certain amount of data space, we can be much more confident in the quality of what goes into the training set without having to run an exhaustive search or wait for perfect label availability.

Lalam: The paper also reports that Distribution D and Training Stability TS are the most critical metrics for performance degradation when they are removed during ablation studies. That tells you where the real weaknesses in selection lie.

Tom: So, looking at the title again, "Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget". It sounds like they've built a system that knows exactly how much value to extract from each piece of data based on what the network is currently learning.

Jane: What does this actually mean for how we think about training models in the future? Does this change how we approach data curation across different AI tasks?

Lu: It suggests that as MLPs get deeper and more complex, we can't just rely on simple metrics; the selection method itself needs to adapt based on what the layers are doing at that moment.

Meng: From a production standpoint, it means we can build data pipelines that are smarter about what they feed into the training loop. It reduces waste by not using low-value samples or overly noisy ones.

Tom: So, DVC gives you a framework where you don't have to choose between getting more data and getting better quality; you manage that trade-off with this utility budget approach.

Jane: That sounds like it moves data selection from a guessing game toward something much more systematic and quantifiable for complex models.

Conclusion: Tom: So we're wrapping up this deep dive on "Adaptive Data Selection for Text Classification: Hierarchical Value with a Local Utility Budget". This paper is about how to smartly pick training data from messy, different sources when you have a limited budget.

Jane: It boils down to evaluating the actual value of each piece of text, not just picking things randomly. The authors are using this concept called Data Value Contribution, DVC.

Lu: They decompose that value into layer-wise contributions and a global contribution, using six different metrics to look at quality, relevance, and diversity across the network layers.

Meng: I'm interested in how they handle the budget constraint. If we're training a model on millions of samples, we need something fast that doesn't explode when we scale up.

Lalam: The selection algorithms they propose are designed to be scalable. They use things like Layer Gradient Caching and Fast Similarity Computation using LSH to keep the math manageable, scaling sublinearly with dataset size.

Tom: Sublinear scaling is good for production, but what does this mean for a developer who just wants to get a solid model running quickly?

Jane: It means we move away from just picking data based on labels or simple metrics and start picking data based on what the model actually learns at different stages. It’s about making informed choices that save training time and improve results without needing massive compute.

Lu: They show that when they use this method, like retaining twenty percent of sources, they see a significant jump in performance across eight different classification tasks compared to simpler selection methods.

Meng: That's a solid result if it holds up on real-world traffic data. But what are the practical trade-offs? It sounds like there's more work to do on those complex metrics.

Lalam: The authors note that Distributional Diversity and Training Stability are the most important factors when evaluating performance drop, which tells you where a selection strategy is weakest. They also provide theoretical guarantees on how this method learns the best sources over time.

Tom: So it’s a framework for data curation that adapts to the training process itself rather than being static before training even starts. It’s about making the selection process iterative and intelligent.

Jane: Exactly, it shifts our thinking from just 'more data is better' to 'this specific type of data contributes this much value at this moment in training.' This level of detail is what makes it useful for complex AI tasks.

Lu: The implication for future work is that we need to keep exploring how these layer-wise and global contributions interact as the model gets deeper and more complex. It’s an ongoing area of research.

Harbin Institute of Technology

cs.LG

Submitted: 2025-10-24

Updated: 2026-10-19

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: DVC, a novel budget-aware method for evaluating and selecting data for MLP training, decomposes data contribution into Layer Value Contribution (LVC) and Global Value Contribution (GVC),

Key concepts

Data Value Contribution (DVC): A novel method that decomposes data value into layer-wise (quality, relevance, diversity) and global assessments (gradient impact, uncertainty). It uses six metrics to quantify how much each data sample contributes to the overall model training process within a set budget.
Layer Value Contribution (LVC): Measures how useful a specific data sample is at different layers of the neural network. This includes assessing quality (stable activations), relevance (gradient alignment with learning direction), and distributional diversity (novelty in feature space).
Global Value Contribution (GVC): Evaluates the sample's impact on the entire network, looking beyond single layers. It assesses overall gradient magnitude, alignment with average gradient direction, predictive uncertainty at the output level, and training stability over time.
Upper Confidence Bound (UCB) Strategy: The selection algorithm that uses UCB to prioritize data sources. This strategy balances exploration (trying less-known data) and exploitation (selecting high-value data based on current knowledge), ensuring efficient learning of source quality.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Data Value Contribution — 1× in this paper
F1 Score — 1× in this paper · explained in 4 episodes
Influence Functions — 1× in this paper · explained in 2 episodes
Shapley values — 1× in this paper · explained in 2 episodes
Theoretical Guarantees — 1× in this paper · explained in 2 episodes

Summary

The gist: DVC, a novel budget-aware method for evaluating and selecting data for MLP training, decomposes data contribution into Layer Value Contribution (LVC) and Global Value Contribution (GVC), employing six carefully designed metrics to capture quality, relevance, and distributional diversity across different granularities.

Problem Definition

The objective is to select an optimal subset of heterogeneous data sources under a fixed budget constraint such that the MLP trained on this subset achieves minimal expected risk on the target test distribution Consider a multi-source learning scenario where we have K heterogeneous data sources S = S1, S2,..., SK, each containing labeled samples from potentially different distributions with varying quality characteristics The optimal data selection problem is formally defined as D∗ = arg min D: D ≤ B E(x,y)∼Ptest [l(fD (x), y)] where fD denotes the MLP trained on dataset D, Ptest represents the target test distribution, and l is the task-specific loss function. The complexity of this optimization problem stems from the discrete nature of subset selection combined with the non-convex training dynamics of neural networks.

Data Value Contribution Evaluation Metrics

The DVC method decomposes sample value into layer-wise and global contributions using the formula DVC(x, y) = ∑L l=1 λl · LVCl (x, y) + μ · GVC(x, y) where adaptive weights satisfy normalization constraints. The layer-wise decomposition is further broken down into three dimensions:

Quality Assessment Ql (x), which evaluates whether a sample produces stable and informative activations at layer l. This metric compares the activation norm of a candidate sample against the median activation norm of a reference set.
Relevance Assessment Rl (x, y), which measures how well a sample’s gradient at layer l aligns with the current learning direction. It computes the cosine similarity between the sample’s layer-wise gradient and the expected gradient direction over a mini-batch.
Distributional Diversity Assessment Dl (x), which identifies samples that provide novel information by occupying previously underexplored regions in the learned feature space at layer l. This metric models the feature space density using a Gaussian kernel density estimator.

The global assessment evaluates three network-wide metrics:

Gradient Impact Assessment GI(x, y), which quantifies how significantly a sample contributes to the overall parameter updates across the entire network. It computes both the magnitude of the sample’s gradient and its alignment with the current average gradient direction.
Conditional Uncertainty Assessment CU(x, y), which identifies samples where the model exhibits high predictive uncertainty, combining output-level uncertainty with intermediate representation uncertainty.
Training Stability Assessment TS(x, y), which evaluates the temporal consistency of a sample’s contribution to the training process by computing the variance of the sample’s loss over a sliding window.

Scalable Selection Algorithms

To manage computational bottlenecks, DVC employs five coordinated algorithms:

Adaptive Weight Learning using Bayesian optimization, which automatically discovers optimal metric combinations without manual hyperparameter tuning. It uses a Gaussian Process (GP) as a surrogate model to approximate the performance function and uses an Expected Improvement acquisition function to balance exploitation and exploration.
Layer Gradient Caching Mechanism, which exploits temporal locality by mapping samples to hash keys and returning stored gradients upon cache hit. This avoids expensive forward and backward propagation when gradients are requested again.
Fast Similarity Computation using LSH, which reduces quadratic complexity diversity assessment to sublinear operations by mapping similar items to the same hash buckets. This reduces complexity from O(N squared · d) to O(n ρ logn · d) where ρ < 1.
Online Data Value Estimator, which maintains incremental statistics using Welford’s algorithm for activation statistics and exponential moving averages for gradient momentum estimation. This enables real-time calibration without expensive batch recomputation.
Adaptive Data Selection Algorithm, which integrates all previous modules into a coherent strategy, employing the Upper Confidence Bound (UCB) strategy to provide principled source prioritization.

Theoretical Guarantees and Scalability

The DVC method achieves sublinear scaling with dataset size, with total computational complexity being O(budget · n ρ logn · d) where ρ < 1. This sublinear scaling means that doubling the dataset size increases computational cost by less than a factor of two. The UCB-based source selection achieves logarithmic regret, ensuring efficient learning of source quality without getting trapped in suboptimal choices. The theoretical analysis shows that DVC’s selection cost remains manageable regardless of dataset growth, providing a fundamental advantage over traditional methods that scale linearly with dataset size.

Experimental Validation

Extensive experiments across eight datasets and eight baselines demonstrate that DVC consistently outperforms existing approaches under various budget constraints, achieving superior accuracy and F1 scores. DVC maintains competitive performance across vision, tabular, and text modalities, suggesting the six-dimensional value assessment captures fundamental principles of sample informativeness that transcend domain-specific characteristics. Ablation studies show Distribution (D) and Training Stability (TS) as most critical metrics for performance degradation when removed. Furthermore, DVC consistently achieves 3.77-6.61× computational speedups while maintaining 96.4-99.4% of full training accuracy across various scales.

Conclusion

DVC offers a scalable and adaptive solution for data selection in MLP training scenarios, combining rigorous theory with practical applicability. The method uniquely decomposes value into layer-wise and global components via six metrics spanning quality, relevance, and diversity. Efficient algorithms ensure that selection cost grows with budget rather than dataset size. By automating data selection with minimal tuning and broad adaptability, DVC lowers deployment barriers across tasks.

Improvements for AI systems

Data selection for Multi-Layer Perceptron (MLP) training can be improved by deploying DVC to select optimal training subsets under budget constraints, achieving superior accuracy and F1 scores across six datasets and eight baselines. This allows practitioners to select a dataset that maximizes model performance by accounting for the dynamic evolution of network parameters during training.
The system can achieve scalable selection complexity, specifically reaching an efficiency of "O(budget · n ρ logn · d) where ρ < 1, which is a fundamental breakthrough in scalable data selection" compared to traditional methods that scale linearly with dataset size. This sublinear scaling ensures that the computational cost remains manageable even as dataset volumes grow exponentially.
The improved AI system can perform layer-aware feature analysis by decomposing sample value into Layer Value Contribution (LVC) and Global Value Contribution (GVC), capturing both local and holistic impacts on MLP learning. This enables better understanding of how samples influence different levels of representation, recognizing that different layers capture distinct levels of abstraction and require specialized evaluation criteria.
The system can integrate principled source prioritization by employing a multi-armed bandit approach guided by the Upper Confidence Bound (UCB) algorithm, which provides principled source prioritization with theoretical regret guarantees. This ensures efficient exploration and exploitation of heterogeneous data sources with unknown quality characteristics.
The selection process can be made robust against noise and distribution shift by incorporating six metrics: Quality (Ql), Relevance (Rl), and Distributional Diversity (Dl). The system assesses these through metrics like Quality Assessment, which evaluates whether a sample produces stable and informative activations at layer l, and Distributional Diversity, which identifies samples that provide novel information by occupying previously underexplored regions in the learned feature space.