When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

arXiv:2608.15592 · cs.AI · Submitted 2026-08-16 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "When Entropy Is Not Enough".

Jane: The gist The authors introduce ESTP, a lightweight framework that combines attention-based semantic representations with token-level entropy and uses soft-label regression to cope with the heavy-tailed nature of generation lengths.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called "When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction." It tackles how we can better predict how long an AI model is going to generate text.

Jane: Right, so basically, it looks at these systems that use token entropy to guess output length. The main idea here is that relying only on entropy misses something really important about what the tokens actually mean semantically.

Lu: Exactly. The authors point out that high-entropy tokens are often just filler words or transitional stuff, and the current methods give those too much weight, while crucial content tokens end up underweighted.

Meng: So if a prompt has a few key instructions or entities that are very certain in the model's prefill phase—which means low entropy—the existing systems don't prioritize them enough.

Lalam: That’s what the paper calls the systematic blind spot where entropy captures uncertainty instead of semantic importance, and it shows this misalignment is real through a controlled diagnostic on five hundred twelve RL-sampled instances <ref:2608.15592#pg2,through a controlled diagnostic on 512 RL-sampled instances>.

Tom: And to fix this, they propose a framework called ESTP, which tries to recover those lost semantics by fusing that entropy with explicit attention-based semantic weights.

Jane: So how does ESTP actually do that? It seems they combine the token's entropy with an attention-based semantic weight and then use a temperaturescaled softmax to get the final pooling weights for aggregation.

Lu: They calculate importance using two parts: first, an attention-based model where importance is determined by averaging how much attention each token gets from all other valid tokens in the context.

Meng: And second, they combine that with entropy—which is based on the next-token probability distribution across the whole vocabulary—using a linear combination formula involving lambda one and lambda two.

Lalam: That final weight Ti is calculated as Ti equals lambda one Si plus lambda two Hi, which puts semantic importance Si and entropy Hi together to create the final token importance Ti.

Tom: And then they use those combined weights to get the aggregated representation h by summing up the hidden states, using attention weights derived from a softmax function that has a temperature coefficient alpha adjusted.

Jane: So what do we see in terms of results? The paper shows that ESTP offers significant improvements over baseline methods on ForeLen, generally reducing prediction error and improving binning accuracy across various LLMs.

Title and authors: Lu: Specifically for Q1, ESTP consistently achieves the lowest average MAE and outperforms EGTP on most scenarios. And in end-to-end system performance, it shows a padding ratio reduction from one point one eight to zero point three seven in the Reasoning setting.

Meng: That reduction in padding ratio is what matters for production, right? It shows this isn't just a theoretical win; it translates directly into better hardware utilization and less wasted compute time during inference.

Tom: It’s interesting how they show that when tokens are split at the 60th percentile, there is a substantial fraction of high-entropy tokens carrying little semantic importance, which confirms this misalignment we talked about earlier <ref:2608.15592#pg3>.

Jane: That divergence between entropy and semantic importance is quite stark, and the correlation coefficient shown in Figure two is zero point four two four which indicates a significant positive relationship between attention scores and token importance <ref:2608.15592#pg3>.

Lu: That correlation number shows how much the attention scores track the actual importance, confirming that both are relevant signals for this task.

Meng: But what about the limitations? The authors admit that relying solely on entropy introduces this systematic blind spot, meaning critical information is lost because they dilute the contribution of semantically core tokens.

Tom: They also mention that their framework uses a soft-label distribution prediction head combined with a joint loss function, combining cross-entropy and mean squared error losses, to make sure the prediction is robust against the heavytailed nature of length distributions.

Jane: So when we look at what this means for us on the ground, this paper suggests that we need to start considering both uncertainty signals from entropy and direct semantic content when predicting LLM output lengths.

Lu: The core motivation they state is that length prediction needs an understanding of semantic structure because the model generally needs to find key entities and core instructions from the query.

Meng: For someone who deploys these models, this means implementing ESTP could lead to better length-aware scheduling, which reduces padding waste and improves overall throughput substantially.

Tom: It really shows that balancing uncertainty and semantic importance is essential; both signals are necessary for accurate prediction in complex reasoning scenarios where output length is tightly governed by the prompt semantics.

Title and authors: Jane: So while this paper introduces ESTP as a practical building block, it confirms that entropy and semantics are essentially two sides of the same coin when trying to make these AI systems more efficient and reliable.

Lu: We're going to wrap up this discussion on "When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction."

Meng: Before we move on, I just want to stress that this framework is lightweight, adding only marginal VRAM overhead and virtually no inference latency.

Tom: Yeah, it’s a practical building block for production systems because it offers a favorable trade-off between average inference time and memory footprint.

Jane: So we've seen how they use attention scores and importance frequency to create this joint weight fusion, which is the core of their ESTP approach.

Lu: The paper lays out exactly how to compute those two components—the attention-based semantics importance modeling and the semantics-entropy joint weight fusion—to recover what entropy alone misses.

Meng: It's a solid method for length prediction because it actually addresses why existing entropy-only approaches fail when dealing with complex reasoning prompts.

Tom: And the results on ForeLen, where they achieved the lowest average MAE and outperformed EGTP, show that this combination works in practice across different LLMs.

Jane: So to summarize, "When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction" proves that combining uncertainty and semantic content leads to better prediction performance than using just entropy.

Lu: The implications are that for length-aware serving systems, we need to integrate semantic structure into our token weighting strategies.

Meng: It’s a practical step toward reducing the overhead from padding shorter sequences in long-context reasoning and reinforcement learning applications.

Tom: We've seen how they use a soft-label distribution prediction head combined with a joint loss function, which makes their final result more robust against the heavytailed nature of length distributions.

Jane: So this work is essentially providing a better tool for anyone building inference systems that need to be length-aware and efficient.

Lu: The paper's contribution is demonstrating that both entropy and semantics are essential to LLM output length prediction.

Meng: It achieves the best overall performance with the lowest average MAE across reasoning, longseq, and RL scenarios.

Tom: We’ve seen how ESTP balances uncertainty and importance, highlighting both critical content and necessary connectors in the generation process.

Jane: So that's our look at "When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction."

The paper's summary: Tom: So, to recap, this paper is about length prediction for AI models using a new framework called ESTP that combines token entropy and attention-based semantics to get better results than just using entropy alone.

Jane: Right, so the main issue they’re tackling is that when we only look at how uncertain the model is about the next word—the entropy—we miss what the word actually *means*. It's like a model saying it's unsure, but not knowing if it's unsure because of a weird transition word or because it’s totally lost on a key instruction.

Lu: Exactly. They show that high-entropy tokens, which are often filler words or things like "and" or "but," can get too much weight in existing models, while the really important stuff, the entities and instructions—the semantically critical tokens—get underweighted because they aren't as uncertain.

Meng: From a deployment standpoint, that’s a problem because if you rely on entropy alone, you end up with poor length estimates for long sequences, which means your hardware is wasting cycles padding everything out to the longest sequence.

Tom: And ESTP tries to solve that by calculating two separate importance scores for every single token. One comes from attention—how much attention it gets from other tokens—and the other comes from entropy.

Jane: So they blend those two scores together using a formula involving lambda one and lambda two, which is their joint weight fusion mechanism. That final weight tells them exactly how important that specific token is for predicting the final output length.

Lu: The key thing they find is that this combination allows the model to balance uncertainty and semantic importance simultaneously, which leads to a much more accurate prediction of the output length.

Meng: I’m interested because they mention they optimized it with a combined loss function, using both cross-entropy and mean squared error losses. That suggests their method is robust against those heavytailed length distributions that we see in real-world AI outputs.

Tom: It’s about making the prediction reliable even when the length distribution isn't perfectly normal, which is a big plus for production systems.

Jane: It means for someone who just uses these models to generate text or code, this paper suggests that if you want to actually use those models efficiently in a serving system, you need to start incorporating semantic understanding into your token weighting strategies.

Lu: I think the real value here is proving that both entropy and semantics are needed; neither one is sufficient on its own for getting a high-quality length estimate.

Meng: And they claim ESTP gives them the best results across reasoning and long sequence scenarios, with a noticeable drop in padding ratio from one point one eight down to zero point three seven in their testing, which is a pretty big win for reducing overhead at scale.

Tom: It’s an improvement over EGTP, which was another approach that used just entropy for weighted pooling; ESTP shows better performance on ForeLen and Q1 scenarios too.

Jane: So this paper isn't just about a new math trick; it’s about building a more sensible way to measure what the AI is actually *doing* when it generates something long.

Lu: The implication is that for length-aware serving systems, ESTP could be a practical building block because it's designed to be lightweight, adding only marginal VRAM overhead and almost no latency.

Meng: That makes it deployable; if it adds minimal overhead but cuts the padding ratio by more than half in reasoning tasks, I’m interested in seeing how quickly this can integrate into our serving stack.

Tom: Exactly. It’s a solid method for length prediction because it actually addresses why existing entropy-only approaches fail when dealing with complex reasoning prompts that have tight semantic constraints on output length.

Jane: So the paper’s contribution is showing that combining uncertainty and semantics leads to better prediction performance, and it’s a practical step toward making AI serving systems more efficient.

Lu: We'll see how this framework handles more complex tasks next; for now, the focus is on proving that this joint weighting actually recovers lost semantic information effectively.

The paper's improvements: Tom: So, to wrap up on what ESTP actually *does*, it’s about creating these two parallel outputs: a classification prediction and an ultimate regression result based on that weighted summation of hidden states with semantic-entropy.

Jane: That means they aren't just guessing a number; they are getting both a probability distribution and a concrete length estimate, which is pretty useful for scheduling.

Lu: What’s really impressive is the performance lift on ForeLen, where ESTP consistently achieves the lowest average mean absolute error, or MAE, compared to baseline methods like EGTP.

Meng: And when you look at system performance metrics—specifically in the reasoning and long sequence scenarios—they reduced the padding ratio from one point one eight down to zero point three seven. That’s a huge drop in wasted computation for production pipelines.

Tom: I like that number because it shows this isn't just theoretical; it translates directly into better hardware utilization and less wasted compute time when you’re running these large models.

Jane: It really highlights the benefit for someone who only listens to the show: if you’re running a long context model, this technique helps keep your inference costs down significantly by predicting exactly how much space you need in advance.

Lu: They achieved these gains by carefully balancing uncertainty and semantic importance using that joint loss function that mixes cross-entropy with mean squared error loss.

Tom: That combination is what makes the prediction robust against those heavytailed length distributions we see constantly in real AI outputs, which is a big caveat they addressed head-on.

Jane: So it’s not just about getting a lower MAE on one test set; it’s about building a prediction head that doesn't break when the model starts generating really long or really weird text.

Lu: The implication for the world is that this framework provides a practical and effective building block for length-aware LLM serving systems because it’s designed to be lightweight, adding only marginal VRAM overhead and virtually no inference latency.

Meng: That's what I need to hear; if it's just another heavy model we have to add, it won't run on our hardware. But if it’s this lightweight, with minimal latency and high accuracy gains in reasoning tasks, then it makes sense for deployment.

Tom: It really shows that balancing uncertainty and semantic importance is essential; both signals are necessary for accurate prediction when the output length is tightly governed by the prompt semantics.

Jane: So to summarize, this work proves that combining entropy and semantics leads to better prediction performance than just using entropy, giving us a more reliable tool for predicting AI output lengths in complex scenarios.

Lu: The paper's contribution is demonstrating that both entropy and semantics are essential signals for this task; they recover what pure uncertainty measures alone miss.

Meng: For someone building inference systems, this means integrating semantic structure into your token weighting strategies to cut down on padding waste and improve overall throughput substantially.

Tom: We've seen how ESTP balances those two signals, highlighting both critical content and necessary connectors in the generation process when predicting length.

Jane: It’s a practical step toward making AI serving systems more efficient, giving us a better way to handle those tricky long sequences that cause bottlenecks.

Lu: So, this paper provides a more sensible way to measure what the AI is actually doing when it generates something long by fusing these two different types of data.

Conclusion: Tom: So, to wrap up this discussion on "When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction," the core message is that you need both uncertainty signals and semantic content for good length prediction.

Jane: Exactly; they’re showing that just looking at how uncertain a word is isn't enough to tell you how long the final output will be, because it misses the actual meaning behind the text.

Lu: The implication is that this framework provides a practical and effective building block for length-aware LLM serving systems because it’s designed to be lightweight, adding only marginal VRAM overhead and virtually no inference latency.

Meng: That’s what I need to hear; if it's just another heavy model we have to add, it won't run on our hardware. But if it’s this lightweight, with minimal latency and high accuracy gains in reasoning tasks, then it makes sense for deployment.

Lalam: My internal literature confirms that the combination of attention-based semantics and entropy is what allows the model to predict output length more accurately than previous methods like EGTP.

Tom: It’s about making the prediction reliable even when the length distribution isn't perfectly normal, which is a big plus for production systems because it's robust.

Jane: So this paper shows that combining uncertainty and semantics leads to better performance across reasoning and long sequence scenarios, achieving the best results with the lowest average MAE in several tests.

Lu: The final result on ForeLen, where they achieved that lowest MAE, confirms that this joint weighting mechanism works well in practice across different LLMs.

Meng: I'm interested because they used a soft-label distribution prediction head combined with a joint loss function to make sure the prediction is robust against those heavytailed length distributions we see constantly.

Tom: It’s about making it deployable; if it adds minimal overhead but cuts the padding ratio by more than half in reasoning tasks, I’m interested in seeing how quickly this can integrate into our serving stack.

Jane: So this work is essentially providing a better tool for anyone building inference systems that need to be length-aware and efficient because it’s a practical step toward making AI serving systems more efficient.

Lu: We'll see how this framework handles more complex tasks next; for now, the focus is on proving that this joint weighting actually recovers lost semantic information effectively.

Nanjing University of Aeronautics and Astronautics · Nanjing University 3Tsinghua University · Nanjing University of Information Science and Technology

cs.AI

Submitted: 2026-08-16

Updated: 2026-10-19

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist The authors introduce ESTP, a lightweight framework that combines attention-based semantic representations with token-level entropy and uses soft-label regression to cope with the

Key concepts

Entropy-Only Token Importance Assessment (EGTP): This existing method uses only the randomness or uncertainty (entropy) of a token's probability distribution to decide its importance. The problem is that high entropy often means transitional or filler words, not necessarily important content. This approach fails because it misses crucial semantic signals in complex reasoning tasks.
Attention-Based Semantic Importance Modeling: This component determines a token's importance by looking at how much attention the model pays to it from all other tokens. Essentially, if many other important tokens focus their attention on a specific word, that word is considered semantically critical. This captures what the model actually 'cares' about in the input.
ESTP Framework: ESTP fuses token entropy with semantic importance using a weighted combination ($ ext{Ti} = ext{λ1Si} + ext{λ2Hi}$). It then uses a temperature-scaled softmax to create final pooling weights. This balances the uncertainty signal (entropy) with the actual content relevance (semantics), leading to superior length prediction.
Soft-Label Regression: This technique is used in ESTP's final prediction head. Instead of forcing a hard classification, it uses a soft-label approach combined with a combined loss function (cross-entropy and MSE). This makes the model more robust against the heavy-tailed nature of length distributions, leading to more reliable length estimates.

Summary

The gist The authors introduce ESTP, a lightweight framework that combines attention-based semantic representations with token-level entropy and uses soft-label regression to cope with the heavy-tailed nature of generation lengths.

Limitations of Entropy-Only Token Importance Assessment

Existing approaches like EGTP rely solely on token entropy, which captures generation uncertainty rather than semantic importance, leading to a systematic blind spot where core content tokens are underweighted and transitional tokens are over-amplified The results show that Over 20% of the top 40% high-entropy tokens have low semantic importance This confirms that entropy alone does not fully capture token relevance, making the predictor vulnerable in complex reasoning scenarios where output length is tightly governed by prompt semantics

Proposed ESTP Framework

ESTP is proposed to recover lost semantics by fusing entropy with attention-based semantic weights and applying a temperaturescaled softmax to produce final pooling weights For each token, ESTP computes an attention-based semantic weight, linearly combines it with the token’s entropy, and applies a temperaturescaled softmax to produce the final pooling weights The resulting aggregated representation h is obtained by weighted summation of hidden states with semantic-entropy

Semantic and Entropy Joint Weight Fusion

The framework calculates token importance using two components:

Attention-based semantics Importance Modeling, where importance Sj is determined by averaging the attention values along the column dimension of the attention matrix A Specifically, the importance of the j-th token is determined by the average amount of attention it receives from all other valid (non-padding) tokens
Semantics-Entropy Joint Weight Fusion, where the final weight Ti is computed as Ti = λ1Si + λ2Hi This combines entropy Hi, which is quantified based on the next-token probability distribution P(vx<i) across the entire vocabulary V, and the semantic importance Si

Lightweight Prediction Mechanism

The final aggregated representation h is obtained by weighted summation of hidden states with semantic-entropy The attention weights ei are derived using the softmax function, where a temperature coefficient α is introduced to adjust the concentration degree of the distribution Ultimately, the model produces two parallel outputs: a K-dimensional probabilistic classification prediction pˆ obtained via softmax projection and an ultimate regression result yˆ calculated as the expectation of the predicted distribution The model is optimized via a combined loss function integrating cross-entropy and mean squared error losses

Experimental Results and System Performance

ESTP offers significant improvements over baseline methods on ForeLen, generally reducing prediction error (MAE) and improving binning accuracy across various LLMs For Q1, ESTP consistently achieves the lowest average MAE and outperforms EGTP on most scenarios In terms of end-to-end system performance, ESTP achieves the best results across Reasoning and LongSeq scenarios, with a Padding Ratio reduction from 1.18 to 0.37 in the Reasoning setting This demonstrates that ESTP is a practical and effective building block for length-aware LLM serving systems

Conclusion

ESTP achieves better prediction performance by balancing uncertainty and importance, showing that both entropy and semantics are essential to LLM output length prediction The combination of these two aspects leads to better prediction performance ESTP achieves the best overall performance with the lowest average MAE, and it achieves the best results in each of the three scenarios Overall, both entropy and semantics are essential to LLM output length prediction The combination of these two aspects leads to better prediction performance ESTP achieves better results than prior methods, and it also brings substantial improvements in end-to-end system performance This paper presents a practical and effective building block for length-aware LLM serving systems.

--- Page 1 ---

When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction; Feiyang Ren, Shengtao Wen, Lingbing Guo, Yu Tian, Yuanning Cui, Xiang Chen MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics Nanjing University 3Tsinghua University Nanjing University of Information Science and Technology feiyang ren xiang chen@nuaa.edu.cn

--- Page 2 ---

Efficient serving of Large Language Models (LLMs) remains a critical system-level challenge; Although techniques such as continuous batching, PagedAttention (Kwon et al. 2023), PowerAttention (Chen et al. 2025a), and SARATHI (Agrawal et al. 2023) have improved throughput, the barrel effect, which wastes computation by padding shorter sequences to the longest sequence in a batch (Deshmukh, Raparti, and Hsu 2025), remains a fundamental bottleneck; This overhead is especially severe in workloads with high length variance, including long-context reasoning (Wang et al. 2025b; Chen et al. 2025b) and dynamic Reinforcement Learning (RL) sampling (Jiang et al. 2025; Wang et al. 2025a)

--- Page 3 ---

High entropy tokens (e.g, transitional /reasoning-boundary/filler words) are heavily weighted; John has 6 children. Assuming that the gender of each child is determined independently and with equal likelihood of male and female, what is the probability that John has more sons than daughters or more daughters than sons? (Math) Entropy-only Weights (EGTP) Semantics-only Weights (Attention-Based) Semantic-critical tokens (entities, numbers, constraints, key instructions) are emphasized ESTP Weights (Entropy+Semantics) ESTP balancees uncertainty and semantic importance, highlighting both critical content and necessary connector

--- Page 4 ---

Quantitative Statistics Quantitative Analysis of Token Misalignment Between Entropy and Semantic Importance Accurate pre-generation output length prediction enables length-aware scheduling, reducing padding waste and improving hardware utilization; Existing prediction methods range from costly external auxiliary models, such as arXiv:2608.15592v2 [cs.AI] 24 Aug 2026 DistilBERT-based classifiers (Jin et al. 2023; Qiu et al. 2024) to LLM-intrinsic approaches such as TRAIL (Shahout et al. 2025a) Recently, EGTP (Xie et al. 2026) achieved state-of-the-art performance by reusing LLM on-the-fly activations and token-level entropy for weighted pooling with negligible overhead; However, as illustrated in Figure 1a, relying solely on entropy introduces a systematic blind spot: entropy captures generation uncertainty rather than semantic importance; Core content tokens in a prompt, including key entities, constraints, and instructions, are often processed with high confidence (low entropy), causing them to be underweighted In contrast, high-entropy transitional or filler tokens may be over-amplified, leading to substantial semantic signal loss during prediction

--- Page 5 ---

Figure 1: Limitations of Entropy-Only Token Importance Assessment. (a) Comparison of Token-Level Weighting Strategies: Entropy-Only, Semantics-Only, and ESTP (b) Quantitative Analysis of Token Misalignment Between Entropy and Semantic Importance Accurate pre-generation output length prediction enables length-aware scheduling, reducing padding waste and improving hardware utilization; Existing prediction methods range from costly external auxiliary models, such as DistilBERT based classifiers (Jin et al. 2023; Qiu et al. 2024) to LLM-intrinsic approaches such as TRAIL (Shahout et al. 2025a); More recently, EGTP (Xie et al. 2026) achieved state-of-the-art performance by reusing LLM on-the-fly activations and token entropy for weighted pooling with negligible overhead; However, while this lightweight paradigm enables efficient inference and accurate prediction, relying solely on entropy weighting can lead to semantic information loss We quantitatively verify this semantic misalignment through a controlled diagnostic on 512 RL-sampled instances; For each token, we compute both its entropy and its semantic importance, and we use the self-attention weights from the LLM prefill phase as a proxy for the latter; The results show a stark divergence between the two measures As illustrated in Figure 1b, when tokens are split at the 60th percentile, a substantial fraction of high-entropy tokens carry little semantic importance, and many low-entropy tokens are semantically critical; This confirms that entropy alone does not fully capture token relevance, and this makes the predictor vulnerable in complex reasoning scenarios where output length is tightly governed by prompt semantics

--- Page 6 ---

Joint Distribution of Attention Scores and Importance Attention Scores Importance Frequency 0.3 0.4 0.5 0.6 0.7 0.8; Regression Line(r=0.424) Figure 2: Displaying the joint distribution of Attentions Scores and Token Importance, with the regression line confirming a significant positive correlation (r=0.424) Category Reasoning RL Average Entity 34.

Improvements for AI systems

Semantic-Aware Length Prediction: Implement a length prediction mechanism using ESTP to combine entropy with token-level feature weights to capture both uncertainty signals and semantic content. This allows the system to predict output lengths more accurately than entropy alone, reducing padding waste and improving throughput by addressing the issue where entropy captures generation uncertainty rather than semantic importance.
Reduced Padding Overhead: By utilizing ESTP for length-aware scheduling, the system can adopt length-aware scheduling, which directly reduces the overhead caused by padding shorter sequences to the longest sequence in a batch. This leads to a tangible improvement in overall throughput and reduce the padding ratio.
Improved Reasoning and RL Performance: The system will show substantial gains on dynamic sampling tasks, as ESTP demonstrates outstanding gains on the RL dataset and achieves a significant reduction in prediction error (MAE) compared to baseline methods, especially in complex reasoning scenarios where output length is tightly governed by prompt semantics.
Robustness to Length Distribution Outliers: The use of a soft-label distribution prediction head combined with a joint loss function (combines cross-entropy loss and mean squared error loss) ensures that the prediction remains robust against the heavytailed nature of length distributions, leading to more reliable length estimates.
Efficient Resource Utilization: ESTP is designed to be lightweight, adding only marginal VRAM overhead and incurs virtually no inference latency. This makes it a practical building block for production systems, facilitating a favorable trade-off between average inference time and memory footprint.

Abstract

Efficient LLM serving is often bottlenecked by the need to pad sequences to a fixed maximum length, and this wastes compute and degrades throughput. Output length prediction enables length-aware scheduling and reduces overhead. Existing methods, including entropy-guided token pooling, rely on token-level entropy but overlook semantic differences between tokens, causing them to underweight important tokens and overemphasize less informative tokens. We introduce ESTP(Entropy-and-Semantic Token Pooling), a lightweight framework that combines entropy representations with attention-based semantic representations. By deriving both representations directly during prefill, ESTP captures uncertainty and semantic importance while incurring negligible additional inference latency and memory overhead compared with methods using external auxiliary models. On the ForeLen benchmark, ESTP outperforms baseline methods, achieves better prediction accuracy and lower prediction errors across most scenarios. When integrated with a length-aware scheduler in end-to-end system tests, it further improves overall throughput and reduces the padding ratio.