TabAgent: Distilling LLM Agent Components into Cheap Classifiers

summary

The gist

The gist: TABAGENT proposes a framework for replacing agentic generative components in closed-set selection tasks with a compact textual-tabular classifier trained on execution traces, enabling

In short

TABAGENT replaces slow, expensive large language model calls used for agent decision-making with a compact textual-tabular classifier trained on execution traces. It extracts structured features like schema and state from agent actions, generates synthetic data to improve training coverage, and uses these features to score candidates. This method maintains high task success while drastically reducing latency by about 95% and inference costs by 85-91%.

Key concepts

TabSchema: This module extracts structured features like schema, state, and dependencies from agent execution traces. It uses a hierarchical orchestration to analyze raw data, identifying signals directly from traces (e.g., thought snippets) and computed signals derived from the trace structure (e.g., loop index). These features are validated by multiple LLM judges for reliability.
TabSynth: This component generates synthetic training data to fill gaps where real trajectories are rare but critical. It takes the features extracted by TabSchema and creates new, schema-aligned training rows for candidate pairs. This synthesis is constrained by the feature card schema, observed execution patterns, and tool catalog information to ensure the synthetic data remains realistic.
TabHead: This is a lightweight textual-tabular classifier that scores potential candidates. It takes two inputs: a tabular view of the extracted TabSchema features and a textual representation of each candidate (name plus description/hints). Trained on real and synthetic data, it predicts the probability of success p(y|x) to rank candidates efficiently.
Agentic Decision Head Replacement: This is the core idea: replacing generative decision components in agent workflows with a compact classifier. Instead of asking an LLM to decide which tool to use (a generative task), TABAGENT trains a small, fast model on historical execution traces. This allows the agent to make decisions quickly and cheaply by learning from past successful actions.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Quantum Fisher Information (QFI) — explained in 21 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Flow Matching — explained in 20 episodes
Sample Complexity — explained in 20 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

TabAgent: Distilling LLM Agent Components into Cheap Classifiers · Read on arXiv

Ido Levy, Eilam Shapira, Yinon Goldshtein, Avi Yaeli, Nir Mashkif, Segev Shlomov

IBM Research

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "TabAgent: Distilling LLM Agent Components into Cheap Classifiers".

Jane: Here's the gist of the paper. The gist: TABAGENT proposes a framework for replacing agentic generative components in closed-set selection tasks with a compact textual-tabular classifier trained on execution traces, enabling agents to optimize their own decision-making by learning from successful executions.

Tom: How it works TABAGENT is a general framework that replaces generative decision components in closedset selection tasks with a compact textual-tabular classifier trained on signals extracted from execution traces, by (i) automatically distilling schema/state/dependency features from trajectories TabSchema, (ii) augmenting coverage with schemaaligned synthetic supervision TabSynth, and (iii) scoring candidates with a compact textual tabular classifier TabHead.

Jane: This approach enables agentic systems to automatically optimize their own decision-making by learning from successful executions, allowing for seamless replacement of expensive generative components.

Tom: TabSchema: Agentic Feature Extraction TABSchema employs a hierarchical agentic orchestration to extract behavior-grounded features from agent trajectories, ensuring both comprehensive coverage and explainable feature selection through multi-stage validation and judicial review.

Paper summary: Jane: So, to recap, this paper "TabAgent: Distilling LLM Agent Components into Cheap Classifiers" proposes a framework that replaces the repeated calls to large language models for closed-set decisions with a compact textual tabular classifier trained on execution traces.

Tom: The thesis is that this design makes agent deployments slow and expensive because of the cumulative latency and token usage from those repeated decisions, so TabAgent tackles this by having it automatically distill schema, state, and dependency features from trajectories using TabSchema.

Lu: And then they add synthetic supervision through TabSynth to augment coverage with schema-aligned data, and finally score candidates with a compact textual tabular classifier called TabHead. This whole approach is designed to allow agentic systems to automatically optimize their own decision-making by learning from successful executions (TabAgent: Distilling LLM Agent Components into Cheap Classifiers <ref:2602.16429#pg1>).

Meng: From a deployment standpoint, the key takeaway for me is that they’re replacing an expensive LLM call with something much smaller and faster, reducing latency by about ninety-five percent and inference cost by eighty-five to ninety-one percent on the AppWorld benchmark.

Lalam: They also showed that this works even when you move across different models; they found that TabSchema extracts genuine decision-relevant structure from execution traces rather than model-specific artifacts (TabAgent: Distilling LLM Agent Components into Cheap Classifiers <ref:2602.16429#pg1>).

Tom: That’s a solid foundation. So the next part is to see what the actual authors are saying about this work in their conclusion.

Conclusion: Jane: Okay, so we’re wrapping up with a look at the title and authors of "TabAgent: Distilling LLM Agent Components into Cheap Classifiers." The paper is authored by Ido Levy, Eilam Shapira, Yinon Goldshtein, Avi Yaeli, Nir Mashkif, and Segev Shlomov.

Tom: What this means for us is that they’ve developed a framework to systematically replace generative modules in agentic decision heads with compact classifiers trained on execution traces rather than relying on the massive input of an LLM for every single step.

Lu: The implication is that we can shift the focus toward learning from execution patterns instead of just trying to prompt massive models for every tactical choice.

Meng: For production, this means we have a way to get substantial latency and cost reductions without having to overhaul our entire agent architecture immediately.

Lalam: It establishes a general paradigm for learned discriminative replacements of generative bottlenecks in production agent architectures (TabAgent: Distilling LLM Agent Components into Cheap Classifiers <ref:2602.16429#pg1>).

More episodes

← Home