LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Search

Detailed Research Summary of LLMasTool: A Hybrid NAS Framework for Open-Ended Architecture Search

As a diligent researcher, I have meticulously analyzed the provided text snippets from the arXiv paper concerning LLMasTool, which introduces a novel Neural Architecture Search (NAS) framework designed to bridge the gap between reliable algorithmic search and the generative power of Large Language Models (LLMs). This summary integrates the core concepts, contributions, methodology, and empirical results presented in both sections to provide a comprehensive understanding of this state-of-the-art approach.

Core Problem and Motivation

The fundamental challenge addressed by LLMasTool lies in the limitations of existing NAS methods:

1. Conventional Algorithm-Driven NAS: These methods rely on hand-crafted search spaces, which inherently restrict open-ended exploration and manual design effort.

2. Agentic LLM Approaches (e.g., LLMatic, LeMo-NADe): While leveraging LLMs to reduce manual design, current implementations struggle to reliably generate complex, valid architectures. Furthermore, their proposals are often biased toward narrow patterns observed in the training data of the underlying LLM.

The authors aim to create a framework that harnesses the strengths of both paradigms: the reliability and structure of algorithmic search and the flexibility and knowledge incorporation capabilities of LLMs, without allowing the LLM to become an autonomous, potentially unreliable, architecture generator.

The LLMasTool Framework: A Hybrid Architecture

LLMasTool is introduced as a hybrid NAS framework characterized by a fundamental shift in how architectural evolution is conceptualized. Instead of relying on direct code generation by the LLM, it defines the evolution step as a tree transformation. This representation provides three critical advantages:

1. Precise and Executable Edits: Representing architectures as hierarchical trees allows for precise, rule-based transformations that ensure the resulting architecture is executable and valid.

2. Seamless Knowledge Reuse: Existing research knowledge can be directly incorporated as pre-defined sub-trees (modules), enabling the seamless integration of well-established components into the search space.

3. Structured Construction: Complex architectures can be constructed in a concise and highly structured manner through these tree operations.

The framework is built upon three synergistic, core components:

1. Algorithm-Based Module Mining (Rule-Based Component):

This component constructs a reliable and diverse search space by automatically extracting reusable modules from arbitrary source code. This provides the necessary constraint and diversity foundation for the search process, moving beyond purely LLM-driven suggestions.

2. Deploy-Friendly Tree Representation:

This is the structural backbone of LLMasTool, enabling precise, executable transformations on DNN architectures represented as hierarchical trees. This structure is key to allowing coarse-grained decisions to be managed by the algorithm while ensuring fine-grained edits are valid and efficient.

3. Coarse-to-Fine Evolution Strategy (Hybrid Planning):

This strategy governs the search process and is the mechanism for balancing algorithmic control with LLM flexibility:

• Coarse-Level Planning (Algorithm Guided): High-level exploration is guided by a diversity-guided algorithm, which leverages Bayesian modeling to improve exploration efficiency. This ensures broad coverage and prevents premature convergence.

• Fine-Grained Edits (LLM Assisted): The LLM is employed not as an autonomous generator, but as a tool to resolve the remaining degrees of freedom within the tree transformation steps, ensuring that the evolved architecture maintains validity and achieves a meaningful evolutionary trajectory.

Key Methodological Innovations

The synergy between these components is what drives the method's superior performance. The authors explicitly state that the performance gain does not originate from the LLM alone, but from this interplay:

Performance Gain proportional to Synergy between (1) Module Mining, (2) Tree Representation, and (3) Coarse-to-Fine Planning with Bayesian Sampling.

Furthermore, they introduce a sophisticated mechanism to enhance exploration efficiency:

• Hierarchical Bayesian-Thompson Sampling: By observing that the coarse-to-fine decision making naturally induces a hierarchical history of decisions within the tree structure, the authors leverage this structure to introduce Hierarchical Bayesian-Thompson Sampling. This technique significantly improves the efficiency of open-ended exploration by intelligently guiding subsequent steps based on past transformations.

The framework achieves an efficient exploration–exploitation trade-off without being rigidly committed to any single heuristic, balancing rule-based algorithmic rigor with LLM flexibility.

Empirical Results and Conclusion

LLMasTool has demonstrated significant efficacy in open-ended NAS:

• Performance Gains: The method achieved state-of-the-art results, yielding performance improvements of 0.69 points on CIFAR-10, 1.83 points on CIFAR-100, and 2.68 points on ImageNet16-120.

• State-of-the-Art Achievement: The framework achieves state-of-the art results, significantly outperforming previous methods in openended NAS.

• Validation of Coarse Planning: The results confirm that the coarse-to-fine decision making remains crucial even under standard training settings, underscoring the necessity of exploration that is not constrained by inherent LLM biases.

**In conclusion, LLMasTool represents a fundamental departure from direct architecture generation methods (like LLMatic or LeMo-NADe). By reformulating the search process as a sequence of tree transformations guided by rule-based algorithms and augmented by an LLM acting as a precise editing tool, it successfully balances algorithmic control with generative flexibility to achieve superior, state-of-the-art performance in open-ended NAS.

1. Bold header: Integration of LLMs as tools in a hierarchical tree-based NAS framework enables stable and open-ended model evolution. This allows architectures to be evolved through reliable tree transformations rather than code generation, which circumvents the current limitation where current LLMs struggle to reliably generate complex, valid architectures.

2. Bold header: The coarse-to-fine decision framework utilizes a hybrid approach where a diversity-guided algorithm leverages Bayesian modeling for coarse planning, while the LLM resolves the remaining degrees of freedom to ensure a meaningful evolutionary trajectory and an executable generated architecture. This prevents exploration from being entirely biased toward a narrow set of patterns observed in their training data.

3. Bold header: Automated module mining from arbitrary source code constructs a reusable module database by inspecting constructor arguments, which allows the search process to manipulate hyperparameters such as channel width, kernel sizes, and internal modules. This directly facilitates the direct reuse of existing implementations and knowledge assets without relying on LLM memorization.

4. Bold header: The architecture is represented as a deploy-friendly hierarchical tree, which simplifies the integration of complex human-written code blocks by allowing for a top-down design process, making it easy to start with a rough architecture and progressively refine its details.

5. Bold header: A novel hierarchical Bayesian-Thompson sampling scheme is introduced, which balances exploration and exploitation by accumulating historical outcomes at upper layers to guide decision-making. This prevents the search from becoming too sparse to enable anything beyond random sampling, leading to a more efficient exploration–exploitation trade-off.

6. Bold header: The system can perform state-of-the-art results, improving performance by 0.69, 1.83, and 2.68 points on CIFAR-10, CIFAR-100, and ImageNet16-120, demonstrating effectiveness across various benchmark datasets under open-ended NAS settings.

Neural Architecture Search (NAS) aims to automatically discover high-performing deep neural network (DNN) architectures. However, conventional algorithm-driven NAS relies on carefully hand-crafted search spaces to ensure executability, which restricts open-ended exploration. Recent coding-based agentic approaches using large language models (LLMs) reduce manual design, but current LLMs struggle to reliably generate complex, valid architectures, and their proposals are often biased toward a narrow set of patterns observed in their training data. To bridge reliable algorithmic search with powerful LLM assistance, we propose LLMasTool, a hierarchical tree-based NAS framework for stable and open-ended model evolution. Our method automatically extracts reusable modules from arbitrary source code and represents full architectures as hierarchical trees, enabling evolution through reliable tree transformations rather than code generation. At each evolution step, coarse-level planning is governed by a diversity-guided algorithm that leverages Bayesian modeling to improve exploration efficiency, while the LLM resolves the remaining degrees of freedom to ensure a meaningful evolutionary trajectory and an executable generated architecture. With this formulation, instead of fully agentic LLM approaches, our method explores diverse directions beyond the inherent biases in the LLM. Our method improves over existing NAS methods by 0.69, 1.83, and 2.68 points on CIFAR-10, CIFAR-100, and ImageNet16-120, demonstrating its effectiveness.