AutoMind: Adaptive Knowledgeable Agent for Automated Data Science

arXiv:2506.10974 · cs.CL, cs.AI, cs.HC, cs.LG, cs.MA · Submitted 2025-06-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "AutoMind: Adaptive Knowledgeable Agent for Automated Data Science".

Jane: Large Language Model (LLM) agents have shown great potential in addressing real-world data science problems, yet their effectiveness remains limited because existing frameworks depend on rigid,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at the paper titled AutoMind: Adaptive Knowledgeable Agent for Automated Data Science. Jane, can you give us the quick rundown on what this whole thing is about?

Jane: Well, it's about how current large language model agents struggle with real-world data science problems because they stick to really rigid workflows and inflexible coding strategies, which means they can only handle simple tasks and miss the actual expertise that human practitioners bring to complex things.

Tom: Right. So what does this AutoMind framework claim it does to fix those problems? What's the main idea?

Jane: They introduce AutoMind, an adaptive, knowledgeable LLM-agent framework that tackles those issues by using three main things: first, a curated expert knowledge base to ground the agent in domain expert knowledge.

Tom: Okay, so you're feeding it specific data science expertise. What's the second advance they have?

Jane: Second is an agentic knowledgeable tree search algorithm that strategically explores different possible solutions to find the best one.

Tom: And what's the third piece of the puzzle that makes it adaptive? The coding strategy, I bet that's important for real work.

Jane: Exactly, they have a self-adaptive coding strategy which dynamically tailors code generation based on how complex the task is.

Tom: That sounds pretty ambitious for an agent framework. So what do the results show when they tested this AutoMind system?

Jane: They evaluated it on two automated data science benchmarks using two different families of foundation models, and AutoMind delivered superior performance compared to the state-of-the-art baselines.

Tom: Superior how much? They mentioned an improvement of eight point zero percent over the prior state-of-the-art on the official MLE-Bench leaderboard. That's a solid number.

Jane: And they also confirmed that AutoMind was more efficient and used lower token costs compared to those prior state-of-the-art systems, which is important for actually deploying these things.

Tom: It sounds like they've done a fair job showing both better performance and better efficiency. So what does this mean for the folks who use data science agents every day?

Jane: It suggests that we might finally have tools that can handle the complexity of modern data science tasks much better than just relying on pre-defined workflows.

Tom: I'm curious about the structure behind it. How does this AutoMind actually work under the hood? What's the core mechanism?

Jane: They model it like an optimization problem where a solution is a tuple with a textual plan, Python code, and a validation metric to assess the results. The objective is to find the optimal solution by maximizing that validation metric.

Tom: That's interesting because it formalizes the whole process into something quantifiable. So they have these three key components you mentioned earlier—the knowledge base, the tree search, and the coding strategy—how do they interact?

Jane: The framework introduces an Expert Knowledge Base to ground the agent in practice, then that feeds into the Agentic Knowledgeable Tree Search to explore pathways, and finally that whole process uses a Self-Adaptive Coding Strategy during implementation to match code complexity with what the LLM can actually write.

Tom: So it's not just one big thing; it's a system where each part supports the next. Lu, from your perspective on model training and representation learning, what do you think is the most interesting part about this approach?

Lu: I find the Expert Knowledge Base construction really compelling. They built it using domain-specific resources like three thousand two hundred thirty-seven public forum posts for four hundred fifty-five Kaggle competitions and papers from top conferences like KDD, ICLR, NeurIPS, ICML and domain journals. That's grounding the agent in what actually works empirically rather than just general knowledge <ref:2506.10974#pg3>.

Meng: From a production standpoint, the self-adaptive coding strategy is what catches my eye because it addresses the problem of code quality at scale. They have this five-point scale based on professional rubrics to decide whether to use a one-pass generation or switch to a stepwise strategy with execution feedback and AST checks when complexity gets high <ref:2506.10974#pg2>.

Lalam: I'm confident in the direction this is going because it systematically addresses the limitations of static knowledge by integrating dynamic search and context-aware coding, which is exactly what's needed for complex problem-solving tasks like data science <ref:2506.10974#pg3>.

Tom: So we’ve seen how they build this system and how it performed against the state-of-the-art. Now, let's look at what the authors say about its title and who they are. What's the big picture implication here?

Jane: The title AutoMind: Adaptive Knowledgeable Agent for Automated Data Science suggests a move away from simple, rigid agent designs toward systems that can learn and adapt their methods based on specialized knowledge.

Tom: And it does that through integrating human expertise directly into the agent's search and implementation process rather than treating it as just another black box.

Jane: It means for the future of AI agents in data science, we should expect them to be much more capable of handling innovative tasks that require creativity and tailoring their approach dynamically.

Tom: So, AutoMind is showing us a path toward more robust and flexible autonomous systems in this field.

Conclusion: Tom: So we’re wrapping up on AutoMind, and we’re looking at what it actually means for us in the long run.

Jane: It comes down to taking those rigid workflows you see in most data science agents and replacing them with something that can learn and adapt its own approach.

Tom: Yeah, the title itself says "Adaptive Knowledgeable Agent." It’s not just another tool following a fixed recipe; it’s a system that knows how to change its recipe based on what it’s doing.

Lu: What this means is that we move away from these systems where you just optimize on top of a workflow, and toward agents that can actually think about the problem space and choose the best path.

Meng: From my side, it suggests a way to build something more robust at scale because it’s not tied to one specific way of coding or planning.

Lalam: I'm fairly sure this is significant because it puts human expertise—that empirical knowledge from people who have actually done the work—directly into the agent’s decision-making process.

Jane: Exactly, so instead of just being trained on code and data, these agents are getting a better handle on what real practitioners know how to do when things get messy.

Tom: It changes how we even think about training these models because we’re not just teaching them tasks; we’re teaching them how to learn and adapt their learning strategy itself.

Lu: And the goal there is a fully autonomous ecosystem where the AI doesn't just execute steps, it figures out the best sequence of those steps for a new kind of problem.

Jane: It sets up a future where data science isn't just about following pre-set pipelines, but about having an agent that can innovate its way to the solution.

Tom: And we’re going to keep looking at how they did it—the expert knowledge base, the tree search—because that’s where the real mechanics of making this work are hiding.

Lu: Right.

Zhejiang University · Ant Group

cs.CL, cs.AI, cs.HC, cs.LG, cs.MA

Submitted: 2025-06-12

Updated: 2026-10-19

Comments: DAI 2026

Code: https://github.com/zjunlp/AutoMind

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Large Language Model (LLM) agents have shown great potential in addressing real-world data science problems, yet their effectiveness remains limited because existing frameworks depend on rigid,

Key concepts

Expert Knowledge Base: This component grounds the LLM agent in domain expert knowledge by archiving 3,237 public forum posts and top conference papers. It uses a hierarchical labeling system to retrieve relevant technical approaches based on task descriptions, ensuring the agent leverages real-world best practices instead of just general LLM training.
Agentic Knowledgeable Tree Search: This algorithm models the solution space as a tree where each node represents a potential solution (plan, code, metric). The agent iteratively selects parent nodes based on a search policy to strategically explore and refine possible solutions for complex data science tasks.
Self-Adaptive Coding Strategy: This mechanism dynamically adjusts how the agent writes code based on the task's complexity, using a five-point scale. For simple plans, it generates code in one pass; for complex ones, it uses a stepwise approach with AST checks and execution feedback to ensure high-quality implementation.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes
Large Language Model — 1× in this paper
Large Language Model (LLM) — 1× in this paper · explained in 3 episodes

Summary

Large Language Model (LLM) agents have shown great potential in addressing real-world data science problems, yet their effectiveness remains limited because existing frameworks depend on rigid, pre-defined workflows and inflexible coding strategies that fail to capture human empirical expertise in complex tasks. This work introduces AUTOMIND, an adaptive, knowledgeable LLM-agent framework that overcomes these deficiencies through three key advances: a curated expert knowledge base grounding the agent in domain expert knowledge, an agentic knowledgeable tree search algorithm strategically exploring possible solutions, and a self-adaptive coding strategy dynamically tailoring code generation to task complexity. Evaluations on two automated data science benchmarks demonstrate that AUTOMIND delivers superior performance versus state-of-the-art baselines, achieving an improvement of 8.0% over the prior state-of-the-art on the official MLE-Bench leaderboard.

How it works

AUTOMIND is modeled as an optimization problem where a possible solution s is defined as a tuple s = (p, σ, η), where p denotes a textual plan of the proposed approach, σ is the Python code snippet, and η is the validation metric used to assess the execution results. The objective is to find the optimal solution: s∗ = arg max s∈S η, where η∗ = max(η (p, σ, η) ∈ S). The framework introduces three major innovations illustrated in Figure 1: an Expert Knowledge Base for data science (§3.1), an Agentic Knowledgeable Tree Search Algorithm (§3.2), and a Self-Adaptive Coding Strategy (§3.3).

Expert Knowledge for Data Science

The expert knowledge base is constructed using domain-specific resources to ground the agent in empirical best practices, overcoming LLMs’ inherent lack of human practitioner experience. This involves two main components:

Knowledge Base Construction: This includes archiving 3,237 public forum posts offering valid solutions for 455 Kaggle competitions and collecting papers from top-tier conferences in recent years like KDD, ICLR, NeurIPS, ICML, and domain-specific journals.
Knowledge Retrieval: To address the weak correlation between task descriptions and technical approaches, a hierarchical labeling system is proposed to facilitate retrieval. This involves constructing a hierarchy label set based on all collected data science tasks with 11 top-level categories and subcategories, then using LLMs to generate summaries for papers from various perspectives like data type and technique, followed by similarity search and re-ranking.

Agentic Knowledgeable Tree Search

The agentic knowledge tree search algorithm models the search space as a solution tree T = (N, E), where each node N ∈ N ⊂ T corresponds to a unique solution s = (p, σ, η). The search is iterative: at the start of each iteration, it selects a parent node Nparent according to a search policy π and specifies an action that generates a child node Nchild.

Self-Adaptive Coding Strategy

The self-adaptive coding strategy is invoked during the code implementation stage of each action to reconcile solution complexity with the inherent coding capabilities of LLMs. The mechanism scales with task complexity based on a five-point scale derived from professional rubrics.

For plans scoring below a preset threshold, the agent implements the entire code for the plan in one pass to maximize efficiency.
If the score exceeds this threshold, it switches to a stepwise strategy by decomposing the plan into sequential substeps and incorporating execution feedback at each substep. This involves performing an Abstract Syntax Tree (AST) check and executing code in a terminal session for each substep, regenerating implementation if tests fail.

Evaluation and Results

AUTOMIND was evaluated on two automated data science benchmarks with two different families of foundation models, showing superior performance compared to baselines. Specifically, on the official MLE-Bench leaderboard, AUTOMIND (o3-mini) surpasses 39.9% of human participants on average, representing an improvement of 8.0% over the prior state-of-the-art (SOTA). Furthermore, AUTOMIND achieves higher efficiency and lower token costs compared to prior SOTA baselines. Ablation studies confirm that expert knowledge provides additional effective supervision for agentic tree search, leading to performance gains of 11.8% and 5.6% in win rate and valid rate metrics when the expert knowledge base is included. Conversely, ablating the self-adaptive coding mechanism by replacing it with a one-pass strategy leads to declines of 13.2% and 27.6% in win rate and valid rate metrics, highlighting its limitations for complex tasks.

Conclusion

AUTOMIND is an adaptive, knowledgeable LLM-agent framework designed for automated data science challenges, delivering superior performance through the integration of expert knowledge, agentic tree search, and self-adaptive coding. The future vision involves extending this framework to create a fully autonomous, continuously evolving knowledge ecosystem where LLM agents generate novel insights.

--- Page 1 ---

AUTOMIND: Adaptive Knowledgeable Agent for Automated Data Science. Yixin Ou, Yujie Luo, Jingsheng Zheng, Lanning Wei, Zhuoyun Yu, Shuofei Qiao, Jintian Zhang †, Yuren Mao, Yunjun Gao, Huajun Chen, Ningyu Zhang† Zhejiang University Ant Group. https://github.com/innovatingAI/AutoMind.

--- Page 2 ---

Abstract Large Language Model (LLM) agents have shown great potential in addressing real-world data science problems, yet their real-world effectiveness remains limited because existing frameworks depend on rigid, pre-defined workflows and inflexible coding strategies that fail to capture human empirical expertise in complex tasks. In this work, we introduce AUTOMIND, an adaptive, knowledgeable LLM-agent framework that overcomes these deficiencies through three key advances: a curated expert knowledge base grounding the agent in domain expert knowledge, an agentic knowledgeable tree search algorithm strategically exploring possible solutions, and a self-adaptive coding strategy dynamically tailoring code generation to task complexity. Evaluations on two automated data science benchmarks demonstrate that AUTOMIND delivers superior performance versus state-of-the-art baselines, achieving an improvement of 8.0% over the prior state-of-the-art on the official MLE-Bench leaderboard. Additional analyses confirm favorable effectiveness, efficiency, and qualitative solution quality.

--- Page 3 ---

1 Introduction Data science agents aim to leverage LLM agents to automate data-centric machine learning tasks that begin with task comprehension, data exploration and analysis, advance through feature engineering, and culminate in model selection, training, and evaluation (Sun et al., 2024; Zheng et al., 2025a; Liu et al., 2025), serving as a critical component for future AI agents to achieve autonomous scientific discovery. Because of the great complexity of these tasks, most existing data science agent frameworks rely on pre-defined workflows and optimize on top of the specific workflow through search and refinement (Jiang et al., 2025; Hong et al., 2024a; Trirat et al., 2024) or extend to a multi-agent framework to better stimulate the performance of each workflow node. However, current data science agents all overlook the fundamental limitations in model capabilities: despite being trained on a massive code-based corpus, the agents inherently lack the rich empirical expertise accumulated by human practitioners in data science tasks (Zheng et al., 2025a). Moreover, existing data science agents largely employ an inflexible coding strategy, and tend to implement code only for relatively simple and classic tasks in practice. Yet the diversity and complexity of real-world problems require a dynamic, context-aware coding strategy. Indeed, addressing truly complex or even cuttingedge tasks that require high levels of creativity and innovation poses significant challenges for data science agents in generating high-quality code appropriately tailored to such complex tasks. To tackle these issues, we propose AUTOMIND, an adaptive knowledgeable LLM agent framework designed for automated data science challenges.

Improvements for AI systems

Expert Knowledge Base Grounding: Implement a curated expert knowledge base that grounds the agent in domain expert knowledge, which allows agents to overcome their inherent lack of human empirical expertise by incorporating papers from top-tier conferences and journals, as well as expert-curated insights from top-ranked competition solutions.
Agentic Knowledgeable Tree Search: Enhance solution exploration by utilizing an agentic knowledgeable tree search algorithm that strategically explores possible solutions, enabling the agent to leverage the knowledge base to dynamically explore multiple solution pathways and thereby enhance its performance on complex problem-solving tasks.
Self-Adaptive Coding Strategy: Introduce a self-adaptive coding strategy that dynamically tailors code generation to task complexity, which replaces rigid workflows with context-aware implementations and thereby providing flexible, efficient solutions across varying levels of difficulty by switching between one-pass generation for simple tasks and stepwise decomposition for complex ones.