Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation

arXiv:2610.10812 · cs.RO · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation".

Dev: The gist Skill-SLM proposes a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem, which substantially outperforms distillation-oriented baselines,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at Skill-SLM today, the paper is titled "Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation," and it’s written by Wenhao Wang and Yanyan Li. It boils down to taking robot operation away from just teaching models through examples, which is what a lot of the current work has been doing.

Dev: Exactly, this isn't about just making bigger language models; it's about how we structure the whole process for an SLM to actually do something reliable on a physical robot. It’s moving from just pattern matching to actual planning and execution based on defined skills.

Taro: What I find interesting is that they are framing this as a task decomposition and skill composition problem, which means instead of one giant model trying to figure out everything at once, you break the job down into smaller, manageable steps first.

Rosa: That’s right, it’s about breaking things down. They propose a framework where an SLM takes a big instruction and breaks it into subtasks and then picks skills from a library to do those subtasks, and finally an orchestration agent puts those skills together into actual robot code.

Dev: It seems like the main idea is creating these two specialized agents, one for breaking things down and one for putting the pieces together, which addresses how you actually get from a natural language command to physical movement.

Taro: It’s interesting because it tackles generalization too; they mention this approach is supposed to help with unseen tasks that require generalizing capabilities beyond what was seen in the training data.

Rosa: Right, so we're talking about moving toward a system that can handle more varied robot jobs without needing a completely new massive dataset every single time.

The paper's summary: Rosa: The paper dives into how they built this skill library first, which is the foundation of the whole thing. They propose a novel robot operational skill aware context-free grammar, or CFG, to automatically extract all the skills a robot needs for specific tasks.

Dev: So they use an LLM, specifically configured with GPT-five point five as mentioned in their work, to look at a set of task instructions—about fifty of them—and generate a CFG from that corpus. This CFG then identifies things like object selection, movements, and inspections.

Taro: That part where they use the LLM to induce the CFG is key because it’s how they get an initial inventory of what skills are even possible for a robot to do in this context.

Rosa: Once those skills are found, they build the skill library by grounding each one in actual robot APIs with a human expert. Each skill gets a structured document, which includes instructions on purpose, inputs, preconditions, and the implementation procedure using those APIs.

Dev: The document structure they use is pretty specific: it’s defined as d(s) = m(s), b(s), where the body part b(s) gives the SLM all the necessary details to implement that specific skill.

Taro: And then you have these two agents, right? A decomposition agent that breaks tasks down using that library, and an orchestration agent that takes those chosen skills and turns them into executable robot operation code segments.

Rosa: Exactly. The whole point of this summary is showing how they structure the entire workflow as a chain: instruction to decomposition to selection to orchestration to execution.

The paper's improvements: Dev: One major improvement they focus on is this progressive skill orchestration strategy, which they introduce for reliability. Instead of just picking one skill and running it, the SLM generates the code segment for the current subtask while conditioning that generation on what happened in all previous steps.

Rosa: That means if you’re doing a sequence of moves, the model can refer back to variables or targets that were set earlier in the plan. It helps preserve those cross-skill dependencies during execution, which is crucial when skills rely on each other.

Taro: I see how that addresses a problem where simple one-shot orchestration might get an intermediate step right but then lose track of the overall plan coherence later on.

Dev: It’s about keeping the context alive as you build the final sequence of operations, which improves the reliability of what ends up being executed on the robot.

Rosa: And they synthesized training data using LLM teachers to create two separate datasets: one for decomposition and one for orchestration. This distillation process is used to teach SLMs how to perform this compositional reasoning effectively.

Taro: So, they are essentially using the powerful reasoning of an LLM teacher to distill its skill-based planning knowledge into the smaller SLMs, which is a smart way to bridge that gap in capability.

Conclusion: Rosa: So, looking at everything we've covered in Skill-SLM, it’s clear they’ve moved away from just hoping an SLM can learn by seeing examples. They’ve proposed a concrete structure where the SLM decomposes a task into subtasks, selects skills from a library, and then orchestrates them progressively for reliable execution.

Dev: The results show that this framework achieves the highest average TSR across all three task sets they tested. And importantly, on new-combination tasks or when new skills are introduced without extra distillation, it maintains performance close to what you’d see on seen-pattern tasks.

Taro: That suggests their compositional reasoning isn't just stuck in the training set; it can actually be extended through this skill augmentation, which is a pretty significant finding for generalization.

Rosa: It really shows that for onboard robot operation, having a framework focused on task decomposition and skill composition can lead to much more reliable and generalizable systems across different robot platforms.

Dev: They also highlighted some limitations. They pointed out that direct skill extraction, where you just pull skills straight from the task corpus, doesn't do well on new-capability tasks, meaning we can’t just rely on the model finding everything automatically.

Taro: And they also found that removing the distillation process causes a big drop in performance, showing that just having prompts and skill documents isn't enough for base models to get this skill-driven reasoning right by themselves.

Rosa: So, in summary, Skill-SLM provides a robust approach to task planning and robot operations by creating a formal skill library, using agent decomposition and orchestration with progressive context awareness for the final code generation.

Dev: It’s a promising direction for reliable onboard SLM-driven robot operation because it provides that structure where we know exactly what's supposed to happen at every step.

Department of Computer & Information Science, University of Massachusetts Dartmouth · Department of Computer Science & Engineering, California State University San Marcos

cs.RO

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 86/100

The gist: The gist Skill-SLM proposes a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem, which substantially outperforms distillation-oriented

Key concepts

Skill Library Construction: This involves creating a structured database of robot skills. A novel context-free grammar is used with an LLM to discover necessary operational skills from task instructions. These discovered skills are then grounded in real robot APIs and documented with detailed instructions for the SLM.
Decomposition and Orchestration Agents: Skill-SLM uses two specialized agents: one decomposes a complex task instruction into smaller subtasks and selects the right skills from the library. A second agent takes these selected skills and sequences them to generate executable robot operation code segments, forming the final plan.
Progressive Skill Orchestration: This strategy improves reliability by generating robot code step-by-step for each subtask. The generation of the current step's code is conditioned on the results of all previous steps, allowing the SLM to maintain context and reference intermediate results, ensuring smooth transitions between skills.

Terminology

Distillation — 14× in this paper · explained in 2 episodes
Generalization — 8× in this paper · explained in 13 episodes
Task Decomposition — 6× in this paper · explained in 2 episodes
Agent Skills — 5× in this paper
Task Success Rate — 5× in this paper
Ablation — 4× in this paper · explained in 1 episode
Task Planning — 3× in this paper · explained in 2 episodes
Ablation study — 2× in this paper · explained in 3 episodes
Adaptation — 2× in this paper · explained in 1 episode
Benchmark — 2× in this paper · explained in 2 episodes
Operation Success Rate — 2× in this paper
Redundancy — 2× in this paper · explained in 2 episodes
Verification — 2× in this paper · explained in 2 episodes
Abstraction — 1× in this paper · explained in 1 episode
Accuracy — 1× in this paper · explained in 1 episode
Conference on Automation Science and Engineering — 1× in this paper
Core — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Generalizability — 1× in this paper · explained in 6 episodes
IEEE International Conference on Robotics and Automation — 1× in this paper
Knowledge Distillation — 1× in this paper · explained in 14 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
LoRA Adapters — 1× in this paper · explained in 3 episodes
Low-Rank Adaptation (LoRA) — 1× in this paper · explained in 14 episodes
Procedural Knowledge — 1× in this paper · explained in 3 episodes
Quantization — 1× in this paper · explained in 14 episodes
Semantic Understanding — 1× in this paper · explained in 2 episodes
Transferability — 1× in this paper · explained in 5 episodes

Summary

The gist Skill-SLM proposes a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem, which substantially outperforms distillation-oriented baselines, especially on unseen tasks that require generalization of capabilities.

Skill Library Construction

To support the skill-driven workflow, the authors propose a novel robot operational skill aware context-free grammar (CFG) to extract the skills required to accomplish tasks and build the skill library accordingly <ref:2610.10812#pg5>.

The LLM CFG constructor is configured with OpenAI GPT-5.5 [30] to induce a CFG from a corpus of 50 task instructions, each requiring the robot to execute a sequence of operations <ref:2610.10812#pg5>.

The resulting CFG identifies a set of skills, such as object selection, movements, and inspections <ref:2610.10812#pg5>.

Based on the discovered operational skills in G, the skill library is constructed where each skill is grounded in the robot APIs by a human expert and encoded as a structured document d(s) = m(s), b(s) <ref:2610.10812#pg5>.

The body b(s) provides the SLM with instructions for a skill’s implementation, specifying the skill purpose, conditions of selection, required inputs and preconditions, implementation procedure with relevant robot APIs, expected output pattern, and operational constraints <ref:2610.10812#pg5>.

Decomposition and Orchestration Agents

Skill-SLM consists of two agents: a decomposition SLM agent decomposes a task instruction into subtasks and selects appropriate skills from the skill library; an orchestration SLM agent then orchestrates the selected skills into executable robot operations <ref:2610.10812#pg3>.

The decomposition dataset Ddec is constructed to enable the SLM to decompose a task into subtasks and select appropriate skills, consisting of (T n train, M(S), pn) <ref:2610.10812#pg6>.

The orchestration dataset Dorch is constructed so that the orchestration SLM agent can accomplish each subtask by orchestrating the selected skills into executable robot operation code segments, consisting of (x n i, cn i) <ref:2610.10812#pg6>.

Progressive Skill Orchestration

To improve the reliability of skill implementation and overall robot operation, Skill-SLM introduces a progressive skill orchestration strategy <ref:2610.10812#pg5>.

For the i-th step in the plan p, the SLM takes xi = (T, ui, b(sji), A, C1:i−1) as input and generates the code segment ci for the current subtask <ref:2610.10812#pg5>.

Conditioning the generation of ci on C1:i−1 enables the SLM to reference previously defined variables, targets, and intermediate results when implementing the current skill <ref:2610.10812#pg5>.

Altogether, by progressively extending the code while retaining both global and local context, the progressive skill orchestration helps preserve cross-skill dependencies and thereby improves the coherence and reliability of skill composition <ref:2610.10812#pg5>.

Dataset Synthesis and SLM Training

To enable SLMs to perform compositional reasoning, LLM teachers are used to generate the decomposition and orchestration datasets to distill the LLMs’ strong compositional reasoning into SLMs <ref:2610.10812#pg6>.

The decomposition dataset Ddec is constructed so that for each training instruction T n train, an LLM decomposes the task based on the metadata of all available skills M(S) and generates a plan p n <ref:2610.10812#pg6>.

The orchestration dataset Dorch is constructed such that for every p n, the LLM generates the corresponding code segment c n i <ref:2610.10812#pg6>.

Two separate LoRA adapters are trained for the decomposition and orchestration SLMs because their input/output spaces and context requirements are different <ref:2610.10812#pg6>.

Experimental Results

Skill-SLM achieves the highest average TSR on all three task sets, demonstrating the effectiveness of the skill-based task-solving framework in diverse scenarios <ref:2610.10812#pg7>.

On New-Combination tasks, both PRISM and Ro-SLM exhibit substantial performance degradation, whereas SkillSLM maintains performance close to that on Seen-Pattern tasks <ref:2610.10812#pg8>.

On New-Capability tasks, Skill-SLM also achieves the highest TSR after new skills are introduced without additional distillation, demonstrating that its compositional reasoning can be extended through skill augmentation <ref:2610.10812#pg8>.

The results demonstrate that Skill-SLM is promising for reliable onboard SLM-driven robot operation across different robot platforms <ref:2610.10812#pg6>.

Qwen3-8B achieves the strongest overall performance in the experiments <ref:2610.10812#pg8>.

Llama-3.2-3B-Instruct achieves the highest output throughput at 18.3 tokens/s and requires only 4.3 GB of peak memory <ref:2610.10812#pg9>.

The two 8B models exhibit similar throughput, with Llama-3.1-8B-Instruct and Qwen3-8B generating 10.6 and 10.7 tokens/s while requiring 9.5 and 10.4 GB of peak memory, respectively <ref:2610.10812#pg9>.

Skill-SLM provides a promising approach to reliable and generalizable onboard task planning and robot operations.

Ablation Study Findings

Direct Skill Extraction significantly falls behind on New-Capability tasks, indicating that directly extracting skills from the task corpus does not reliably identify consistent and complete operational skills <ref:2610.10812#pg8>.

No Distillation removes distillation causes the largest degradation, indicating that system prompts and skill documents alone are insufficient for base SLMs to perform skill-driven reasoning reliably <ref:2610.10812#pg8>.

One-Shot Orchestration often produces correct intermediate operations but fails to maintain coherence for successful task execution <ref:2610.10812#pg8>.

The progressive orchestration instead generates code subtask-by-subtask while preserving prior context, improving the coherence of the generated code and therefore enhancing the success of task execution <ref:2610.10812#pg8>.

Qwen3-4B provides a favorable balance between task-solving performance and deployment efficiency, achieving performance comparable to the 8B models while requiring less memory and providing higher throughput <ref:2610.10812#pg9>.

The results demonstrate that Skill-SLM can generalize its skill-based compositional reasoning across different robot platforms and scenarios.

Conclusion

Skill-SLM reformulates robot operation as a task decomposition and skill composition problem, constructs the skill library with an operational skill aware CFG, distills LLM knowledge and capabilities into SLM agents, and progressively orchestrates selected skills into coherent executable robot operations.

This work provides a promising approach to reliable and generalizable onboard task planning and robot operations.

Improvements for AI systems

Operational Skill Aware CFG for Skill Discovery: Instead of relying on direct LLM-induced CFG construction which can produce redundant skill definitions, this system utilizes an LLM configured with a system prompt to consider both task coverage of the CFG and robot operational semantics during the verification step, leading to a more accurate and less redundant skill library.
Teacher-Synthesized Training Datasets: The framework employs LLM teachers to synthesize two distillation datasets that capture different stages of the reasoning process, specifically a decomposition dataset like task instructions, subtask descriptions, and selected skills and an orchestration dataset containing subtask descriptions, skill specifications, and their corresponding robot operation code.
Progressive Skill Orchestration for Code Coherence: The system implements a strategy where the LLM generates code segments sequentially (code segment ci) conditioning the generation on previous code (C1:i-1) to preserve cross-skill dependencies, which improves the coherence and reliability of skill composition by preventing the SLM from forgetting intermediate operations.
Task Decomposition Agent for Skill Selection: A dedicated decomposition agent is trained using the decomposition dataset, enabling it to perform task decomposition and select appropriate skills from the skill library by generating a plan where each subtask is mapped to a specific skill based on context provided in M(S).
Orchestration Agent for Skill Composition: A separate orchestration agent is trained using the orchestration dataset, tasked with solving skill composition by orchestrating the selected skills into executable robot operation code, ensuring that the final output is a coherent sequence of operations rather than a monolithic, potentially erroneous command.

Abstract

Small language models (SLMs) have been increasingly adopted for onboard robot operation because they enable intelligent decision-making. However, existing approaches are mainly distillation-oriented and rely on enumerating representative task-solution pairs. This makes dataset construction difficult and limits generalization to diverse robot tasks whose possible forms grow rapidly. This paper proposes Skill-SLM, a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem. Given a natural language task instruction, Skill-SLM decomposes the task into subtasks, selects appropriate skills from the skill library, and orchestrates the selected skills into executable robot operations. First, to support the skill-driven workflow, we propose a novel robot operational skill aware context-free grammar (CFG) to extract the skills required to accomplish tasks and build the skill library accordingly. Then, we configure LLM teachers to induce and synthesize training datasets for the SLMs, enabling SLMs to decompose tasks and orchestrate skills reliably. Additionally, we employ a progressive skill orchestration strategy to improve the reliability of skill implementation and overall robot operation. Experiments on UAV operation tasks indicate that Skill-SLM substantially outperforms distillation-oriented baselines, especially on unseen tasks that require generalization of capabilities. Additional experiments on ground vehicle tasks further demonstrate that Skill-SLM can be applied to different robot platforms.