Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures

arXiv:2610.10261 · cs.SE, cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures".

Jane: The gist The study evaluates whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address limitations in extracting Machine Learning pipeline structures,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called "Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures." It’s about using these smaller language models to figure out how data scientists actually build their machine learning pipelines by looking at the code.

Jane: That sounds really practical, Tom. For someone listening right now, it means they're trying to understand the actual steps in a complex data science workflow instead of just guessing based on a description. What’s the big goal here?

Tom: The main goal is to see if these small language models can actually do this extraction job better than existing methods, like those based on rules or other classifiers. They set out to test whether these models can handle the diversity of different ML pipelines we see in the real world.

Lu: What's interesting about their approach is that they aren't just testing one model; they are comparing different small language models trained for coding tasks, and then unifying the taxonomies from two existing reference works before running those comparisons. This shows they’re trying to create a more robust test of the SLMs' ability to generalize across different pipeline structures.

Meng: I’m curious about the actual performance numbers here. Is this just theoretical, or are we talking about something that actually runs on real code? Because if it takes forever, it doesn't matter if it’s good on paper.

Jane: That’s a fair question, Meng. The paper does conduct a study to find the best performing SLM among open-weight models trained for coding tasks, and they selected Qwen2 point 5-Coder-7B Instruct as the best performer in their comparison because it had a weighted average F1 score of zero point seven five four on the DASWOW test dataset >

Tom: That's the number they got on that specific test, which was better than the second best model, Gemma three 4B, by a margin of plus zero point one one three > It shows that even in this task, there’s a clear winner among these models.

Lu: But the paper also looked at how much the wording of those pipeline stages matters. They did this by taking their unified taxonomy and mutating the headwords using a synonym substitution method, and they found that this wording change had a highly significant impact on classification results, with a Cochran’s Q test showing a q value of four thousand one hundred twelve point zero one, p less than zero point zero zero one >

Title and authors: Jane: So even tiny changes in how you describe the steps in the pipeline can seriously mess up how well the AI classifies them, which suggests these models are really sensitive to the exact terminology used.

Tom: Exactly. And when they found that best performing taxonomy mutation, it led to an increase of up to zero point one zero eight for the weighted average F1 score and zero point one five four for the weighted average Matthews Correlation Coefficient > That MCC metric is important because it handles cases where some classes are much larger than others, which is a common problem in classification tasks.

Meng: Okay, so performance varies wildly depending on the words they use to describe things. But what about how this applies when we try to actually deploy this for a real data scientist? Is it usable?

Jane: That brings us to the next part of their investigation, which looks at how the SLM’s output compares to actual data scientist insights. They compared the classification results with those from reference studies and even manually labeled ML pipelines >

Tom: And what they found was some pretty big differences in reported proportions. For example, when using SLMbest, it showed a twenty percent difference in the proportion for Data Preparation compared to the ground truth >

Lu: That’s a notable finding because it means the model isn't just getting the general idea right; it’s getting specific parts of the process wrong in terms of how much of each stage exists > They also saw that when classifying with DS-Pipelines, there was a thirty-five point two three percent under-representation for Data Preparation stages compared to the ground truth >

Jane: It sounds like they’re telling us that even when these models are performing well on a test set, the way they interpret and report the actual pipeline structure is different from what we see in reality.

Tom: And they also found that analyzing transition probabilities showed DASWOW appeared more representative of the transition probabilities from the ground truth, even though it had a greater difference in number of yielded transitions compared to SLMbest > It’s a bit confusing, but it points toward where the model might be struggling most.

Title and authors: Meng: So what does this mean for us in terms of building tools? If we use an AI to audit code, we need to know that its interpretation isn't just slightly off on a few steps, but fundamentally misrepresenting parts of the process >

Jane: That’s right. And they pointed out that while SLMbest showed good classification performance without needing prior fine-tuning, it still didn't outperform the reference studies when it came to things like Data Preparation because of that high false positive rate on Data Preparation stages >

Lu: The paper ends by saying the inference duration for SLMbest was much larger than reference studies, taking over twenty minutes to classify the entire DASWOW test dataset and consuming one hundred seven Watt hours of energy > So, while it shows promise in understanding practices, it’s not ready for massive scale deployment right now due to those resource requirements.

Tom: It highlights that we still need to focus on making things more efficient. They suggest that traditional supervised classifiers paired with LLM embeddings might be a better way forward if you want to cut down on the cost and get something deployable >

Jane: So, in summary, this paper on "Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures" shows that SLMs can indeed identify different practices than existing studies, but they are highly sensitive to how the taxonomy is written and still face big issues with inference cost and accuracy on certain pipeline stages >

Lu: It confirms that there’s significant work left in developing a comprehensive understanding of these practices, especially since the observed stage distribution and transition probabilities were found to be statistically significant for all classification methods >

Meng: I think if we want to use these tools, we have to focus on those efficiency constraints. A tool that takes twenty minutes to run isn't going into a production environment.

Jane: Exactly. So, the path forward seems to be designing an efficient classification method that can properly analyze and understand the practices of data scientists, rather than just relying on the raw output of a model >

The paper's summary: Tom: So we’re looking at this paper titled "Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures." The core idea is testing whether small language models can actually figure out the structure of a machine learning pipeline just by looking at the code that defines it.

Jane: That sounds really useful for understanding how data scientists actually work in practice. So, what’s the main takeaway if they could do this extraction job?

Tom: The main thing is that while these small language models show promise, they aren't perfect yet. They’re very sensitive to the exact wording used to describe the pipeline stages.

Lu: I think that sensitivity is key because it shows how much linguistic detail matters when an AI tries to interpret a process. It’s not just about the code itself; it’s about the labels attached to that code.

Meng: So, if I'm trying to build a tool for production, does this mean I can just throw any LLM at my pipeline code and expect accurate results?

Jane: Not exactly. The paper shows they compared their model against existing methods and found that even with the best model on the test set, it still didn't fully match the performance of those traditional classifiers in certain areas.

Tom: Yeah, specifically around data preparation stages. They saw a significant difference there where one approach reported twenty percent more of that stage than what was actually there in their ground truth data.

Lu: That suggests that the way an AI interprets how much work is done on a specific part of the pipeline can be quite misleading, and that's a big gap in our understanding.

Jane: And it’s not just about accuracy; they also looked at how much computing power these models need to do this extraction. The inference time was quite long, taking over twenty minutes for one test set classification.

Meng: That’s what keeps me up at night. If a model takes twenty minutes to analyze a pipeline structure, it's not practical for any real-time auditing or large-scale deployment.

Tom: Exactly. So the paper proves that while these models can get close, they still have significant limitations when we push them toward real-world application where speed and cost matter more than just the raw classification score.

Jane: It’s a big signal that we need to focus not just on making models smarter at reading code, but on developing much more efficient ways to analyze that information for practical use.

Lu: And they pointed out that the observed differences in stage distribution were statistically significant across all methods, which means whatever structure you find is definitely different from the real thing.

Tom: So while it’s interesting for understanding how these models are learning pipeline structures, the paper warns us that we still have a lot of work to do before we can trust them to analyze complex data science workflows reliably.

The paper's improvements: Tom: So the paper doesn't just stop there; they actually lay out some suggestions for how to make this process better in the future. They point out that using a single model isn't enough if you want robust results.

Jane: What kind of improvements are we talking about here? Are they suggesting we just use bigger models, or something more structural?

Lu: They suggest incorporating more techniques into the prompting and reasoning itself. Specifically, they discuss using Chain-of-Thought to force the model to think sequentially through the stages rather than just looking at everything at once.

Meng: Thinking sequentially would help with accuracy, but what about the cost? If we add more steps in the AI's thinking process, does that just make it run even slower and use more energy?

Tom: That’s a valid concern. The authors also emphasize that we should be looking at using metrics like Matthews Correlation Coefficient alongside the F1 score because those metrics give us a better picture when some parts of our classification task are much bigger than others.

Jane: That makes sense for unbalanced tasks, but what about how they handle the ambiguity in the pipeline description? They found that if you change just a few words in the taxonomy, the entire classification can shift dramatically.

Lu: Exactly. It suggests we need to build systems that are more resilient to small changes in input vocabulary, maybe by using expert consensus to unify those stage names before feeding them into an AI.

Tom: And they also touched on making sure you select your model carefully from the start, not just picking the biggest one available. They suggest benchmarking against other coding benchmarks like BigCodeBench and EvalPlus to ensure you pick a model that’s both smart and actually runs well on limited hardware.

Jane: So, it’s about a more careful selection process than just looking at one test score. It’s about finding the right balance between performance and practical deployment constraints.

Meng: I agree with that part. If we can get a model that runs fast and is reliable, then the pipeline structure extraction becomes something we can actually deploy in production systems.

Lu: And they also flagged the inference cost again, stressing that for large-scale work, you have to be extremely mindful of the resources required to run these models, which is why optimizing models like Qwen2 point five-Coder-7B Instruct is so important.

Tom: So the paper’s direction is clear: use small language models for reverse engineering, but do it intelligently by refining how we prompt them and how we evaluate their performance against real-world constraints.

Conclusion: Tom: So to wrap up, we looked at the paper "Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures." It shows that small language models can identify different practices than previous studies, but they struggle with accuracy on certain pipeline stages and face big inference costs.

Jane: It really highlights that as we build AI tools for data science, we can’t just rely on the model's raw output; we have to be careful about how it interprets those steps.

Lu: The paper confirms that there’s still a lot of work needed to develop a comprehensive understanding of these practices because the observed stage distribution and transition probabilities were statistically significant across all classification methods.

Meng: And from an engineering standpoint, the major hurdle remains making this process efficient enough for real-world deployment without those long run times.

Tom: Yeah, so while they’ve shown promise in identifying structure, we still need better ways to make these analyses practical and cost-effective.

Jane: It points toward a future where we design classification methods that are inherently more efficient for analyzing data scientist practices.

Lu: I think the focus should be on those efficiency constraints, making sure the analysis is fast and cheap enough for actual use.

Meng: Yeah, I'm curious to see what other papers come out next that actually tackle these deployment hurdles head-on.

Nicolas Lacroix, Frédéric Precioso, Mireille Blay-Fornarino, Sěastien Mosser

Universit d Cote d'Azur · inria · CNRS · McSCert, McMaster University

cs.SE, cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Code: https://github.com/sumonbis/DS-Pipeline

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 76/100

The gist: The gist The study evaluates whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address limitations in extracting Machine Learning pipeline

Key concepts

Small Language Models (SLMs): These are relatively smaller language models designed for specific tasks, like understanding code. The researchers tested if these models could use their coding knowledge to identify the different stages of an ML pipeline, such as data preparation or training steps.
Taxonomy Wording: This refers to the specific vocabulary and phrasing used when describing different parts of a machine learning pipeline. The study found that small changes in how these terms were written significantly affected how well the SLM could classify the pipeline stages correctly.
Inference Costs: This is the computational expense incurred when a model processes an input to produce an output. The researchers noted that using the best SLM still resulted in long processing times and high energy consumption, indicating a practical limitation for widespread use.

Summary

The gist The study evaluates whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address limitations in extracting Machine Learning pipeline structures, finding that while SLMs show promise, they are sensitive to taxonomy wording and still exhibit common limitations like high inference costs.

How it works

The research conducted a confirmatory study based on two relevant reference works representing current limitations in the state-of-the-art The paper addresses the following research questions: RQ1 In ML pipeline’s structure extraction, are SLMs significantly more accurate in identifying stages than existing model-based or rule-based classifiers? RQ2 Does the taxonomy “wording” significantly influence the classification results of SLMs? RQ3 Does the use of an SLM in the reverse-engineering process provide significantly different exploratory insights than those produced by reference studies compared to the ground truth?

To answer RQ1, researchers selected two reference studies and replicated their classification methods using provided reproduction packages, benchmarking and selecting the best performing SLM among open-weight SLMs trained for coding tasks. They then created a mapping between different reference studies’ taxonomies to obtain a unified taxonomy. To answer RQ2, they mutated the unified taxonomy’s headwords using the “synonym substitution” method and evaluated each mutation impact using the same benchmarking methodology. Finally, to answer RQ3, they compared key insights emerging from classification results with those from prior studies and manually labeled ML pipelines to assess consistency.

Taxonomy Unification and Model Selection

The study involved unifying taxonomies from two reference studies by identifying equivalent terms and defining a mapping function w that assigns a unique headword to each group. This process was assisted by three expert data scientists who helped elaborate the unified taxonomy, achieving an overall agreement of almost perfect (κ = 0.89). The best performing SLM selected for comparison was Qwen2.5-Coder-7B Instruct, which significantly surpassed other models across critical performance metrics. This model achieved a weighted average F1 score of 0.754 on the DASWOW test dataset, surpassing the second-best model (Gemma 3 4B) by a margin of +0.113.

Impact of Taxonomy Wording and Performance Comparison

The research found that taxonomy wording significantly impacts classification performance, as demonstrated by a Cochran’s Q test revealing a highly significant difference among the 18 mutated taxonomies (Q = 4112.01, p < 0.001). Practically, the best performing taxonomy mutation led to an increase of up to +0.108 for the weighted average F1 score and +0.154 for the weighted average Matthews Correlation Coefficient (MCC). When comparing SLMbest with reference studies, it achieved a weighted average F1 score of 0.912 on DS-Pipelines and 0.748 on DASWOW, proving competitive but not outperforming the reference classifiers due to a high False Positive Rate on Data Preparation.

Insights into ML Practices

The comparison of insights from SLMbest classification results with those produced by reference studies and ground truth revealed significant differences in reported proportions of stages. Specifically, SLMbest presented the largest difference in the proportion for Data Preparation (+20% compared to the ground truth), while classifying with DS-Pipelines led to an under-representation of Data Preparation stages (−35.23% compared to the ground truth). Furthermore, analyzing transition probabilities showed that DASWOW appeared more representative of the transition probabilities from the ground truth, despite having a greater difference in the number of yielded transitions compared to SLMbest.

Limitations and Future Directions

The study concluded that while SLMbest demonstrated good classification performance without prior fine-tuning, it did not outperform reference studies, especially concerning Data Preparation with a high False Positive Rate. The inference duration of SLMbest remained much larger than reference studies, taking over 20 minutes to classify the entire DASWOW test dataset and consuming 107W h of energy. Future work should investigate more traditional supervised classifiers with LLM embeddings to mitigate cost while benefiting from extensive training datasets. The findings highlight the need for defining an efficient classification method to properly analyze and understand data scientists’ practices.

The study concludes that using SLMs leads to the identification of significantly different practices compared to existing studies, demonstrating that there remains a significant amount of work to be done to develop a comprehensive understanding of these practices. The results highlight the importance of defining an efficient classification method to properly analyse and understand the practices of data scientists.

--- Page 1 ---

Improvements for AI systems

textbfImprove SLM selection criteria based on recent benchmarks and resource constraints. The current selection process can be refined by incorporating a more dynamic evaluation of SLMs based on their performance on the HumanEval benchmark alongside new leaderboards like "BigCodeBench [46] and EvalPlus [47], to ensure the selected models are both high-performing and compatible with practical inference requirements, as noted in Section 4.2. The paper notes that this selection strategy ensures each benchmark has the same weight in our evaluation (same amount of models per benchmark).
textbfImplement a robust taxonomy refinement process guided by expert consensus to reduce linguistic sensitivity. Instead of relying solely on automated model generation for unification, the system should incorporate the findings that we worked with 3 expert data scientists (E1, E2, E3) from academia to resolve disagreements and create the final Tunified taxonomy, mitigating risks associated with potential hallucinations during unification as discussed in Section 4.3.
textbfEmploy a context-aware prompting strategy that prioritizes code structure over static textual cues. The system should adopt the finding that placing the notebook code in the system prompt as context and moving all other elements (guidance, taxonomy, code instruction) to the user prompt led to an improvement, especially for the weighted average MCC metric, to enhance classification accuracy.
textbfIntegrate conditional reasoning into prompts via Chain-of-Thought for sequential stage identification. By following Chain-of-Thought (CoT) and taxonomy ordering techniques, the system can be designed to process each element of the taxonomy sequentially rather than as a whole, allowing it to evaluate and mitigate potential positional biases by testing different branch orderings, which is crucial for identifying patterns like pipeline jungles.
textbfDevelop a multi-metric evaluation framework that prioritizes balanced performance over raw accuracy. The system should utilize metrics like the Matthews Correlation Coefficient (MCC) alongside F1 score because MCC supports well unbalanced classes in classification tasks, especially given the observed issue where SLMbest showed a high False Positive Rate on Data Preparation, which has a large support, and strongly impact[s] the weighted average MCC score.
textbfAssess inference cost and latency as primary constraints for large-scale deployment. The system must incorporate the finding that the resources required to classify code instructions interrogates the applicability of SLMs for this task at larger scale, given that classifying the entire DASWOW test dataset took more than 20 minutes and incurred significant energy consumption, necessitating a focus on optimizing models like Qwen2.5-Coder-7B Instruct for practical application.
textbfProvide actionable insights into structural discrepancies between classifiers and ground truth. The system should be designed to explicitly report the findings that the observed stage distribution and transition probabilities were found to be statistically significant for all classification methods, allowing users to interpret results with the caveat that the magnitude of this significance (M = 6.61, SD = 2.78) is very high compared to the thresholds of 2 or 3.

Abstract

Context: Once defined a taxonomy of stages structuring Machine Learning (ML) pipelines (e.g. Data Preprocessing, Modeling...), extracting these stages from source code is key for better understanding ML practices. However, the diversity caused by the constant evolution of ML (e.g., algorithms, datasets) makes this task challenging. Existing approaches either rely on non-scalable manual labeling or on classifiers that do not properly support domain's diversity. These limitations call for more reliable solutions. Objective: We evaluate whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address these limitations, and enhance our understanding of practices in ML. Method: We conduct a confirmatory study based on two relevant reference works representing current limitations in the state-of-the-art. We first compare several SLMs using Cochran's Q test, then evaluate the best-performing model against reference studies via two McNemar's tests. An additional Cochran's Q test examines how taxonomy definition variations affect the SLM performance. Finally, goodness-of-fit tests compare ML practice insights from SLM classification with those from prior studies. Results: First, we found that the taxonomy wording significantly impacts classification performance. Second, the best performing SLM yielded good results, yet, without outperforming other classifiers. Third, the three classification methods led to significantly different insights, with varying effect sizes, when exploring practices of data scientists. Conclusions: Limitations of existing classification methods bias our understanding of ML practices. While current SLMs show promising results without prior fine-tuning, they still exhibit common limitations, in addition to inference high costs challenging their applicability in large-scale studies.