Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'

arXiv:2410.21647 · cs.SE, cs.CL · Submitted 2024-10-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'".

Jane: Here's the gist of the paper. The gist The authors create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects to evaluate whether language models can replace programmers for coding.

Tom: Existing repository-level code generation benchmarks are insufficient because they often consist of short completions or focus on limited scale repositories, failing to represent real-world coding tasks <ref:2410.21647#pg2>.

Jane: To address this, REPOCOD includes "980 whole-function generation tasks from 11 popular projects, 50.8 % of which require repository-level context" <ref:2410.21647#pg4>.

Tom: The benchmark meets four criteria: tasks should be real-world coding tasks such as implementing complete functions based on specifications <ref:2410.21647#pg4>, both tasks and source repositories need to be complex, and tasks should have repository-level dependencies <ref:2410.21647#pg4>.

Paper summary: Jane: So, looking at the whole discussion around Can Language Models Replace Programmers for Coding? REPOCOD Says ‘Not Yet’, what does this mean for the broader conversation about AI and programming?

Tom: Basically, the message is that LLMs aren't ready to replace programmers on their own yet when it comes to generating complete, complex code from scratch within a large project structure.

Lu: The authors are arguing that existing benchmarks are too narrow; we need things like REPOCOD which include repository-level context and test cases for better evaluation metrics.

Meng: From my perspective at the startup level, this means that if we’re building AI tools for developers, we can’t just focus on generating single functions in isolation anymore.

Tom: We have to design systems that can effectively retrieve and use context from the entire repository when generating new code.

Jane: It points toward a future where AI assistants are more like sophisticated coding partners who understand how all the pieces fit together, rather than just autocomplete tools for small files.

Lu: The research underscores that building models strong enough to handle these deep dependencies is the next big step in this field.

Meng: I think the focus needs to shift from just training bigger models to training them on data that accurately reflects how real software is structured and used.

Tom: And we need better ways to test them, which is exactly what REPOCOD tries to provide, by measuring performance on these complex, dependency-heavy tasks.

Jane: So the implication is clear: we're still in a phase where LLMs are powerful assistants, but they aren't yet fully autonomous programmers for massive software projects.

Conclusion: Jane: So, we’re wrapping up on this paper about REPOCOD, which basically asks if language models can actually replace programmers for coding yet.

Tom: It seems like the authors are pretty direct with their title: "Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'".

Lu: Yeah, they set up a benchmark called REPOCOD to test this idea against real-world complexity.

Meng: So what’s the main message from their testing? Did they find that the models actually passed much of it?

Lalam: The results show none of the ten language models achieved more than thirty percent pass at one on REPOCOD.

Jane: Thirty percent is a low number for this kind of complex task, isn't it? What does that mean for how capable these AI tools actually are right now?

Tom: It means they aren't ready to generate the whole thing from scratch in a real project structure yet. They need more training.

Lu: I think the core finding is that current models struggle with repository-level code generation, even when given context.

Meng: So it’s not just a matter of making the models bigger or trained on more general code; we need something specific for this level of dependency management.

Jane: The paper also points out that retrieval augmented generation works better than just feeding it the target function dependencies directly as context.

Tom: That’s an important detail because it shows *how* you feed the information to the model matters more than just dumping all the code in one massive prompt.

Lu: And they found that higher recall in retrieving those dependencies as context actually improved how well the models performed on REPOCOD.

Meng: For us at scale, that suggests our systems need smarter ways to pull relevant files from a repository before we even try to generate something new.

Jane: So, while the idea is exciting, the evidence right now points toward a gap between current AI capabilities and what actual software development requires.

Tom: Exactly. It’s a clear signal that we still need stronger models built specifically for those deep coding tasks in real-world environments.

Lu: This work confirms that building models that can handle those deep, complex dependencies is definitely the next research direction to focus on.

Meng: And it also tells us what kind of data we need to collect if we want any benchmark like REPOCOD to be useful for actually deploying things.

Jane: It leaves us with a clear picture: AI is a powerful tool, but it’s not yet the fully autonomous programmer we hoped it would be for massive projects.

Shanchao Liang, Nan Jiang, Yiran Hu, Lin Tan

Purdue University

cs.SE, cs.CL

Submitted: 2024-10-29

Updated: 2026-10-20

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 83/100

The gist: The gist The authors create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects to evaluate whether language models can

Key concepts

Repository-Level Code Generation: This refers to the complex task of having an AI write entire functions or code blocks that depend on understanding the structure and context of an entire large software project repository, not just isolated snippets.
REPOCOD Benchmark: This is a new testing suite designed to evaluate LLMs by using tasks that mimic real-world coding. It includes 980 whole-function generation tasks drawn from 11 popular projects, with over half requiring context from the entire repository.
Retrieval-Augmented Generation (RAG): This technique improves LLM performance by allowing the model to look up relevant information, such as dependencies of a target function, before generating its response. The study found that using this method yielded better results than simply using target function dependencies as context.
Pass@1 Metric: This is a standard metric used to evaluate generative models. It measures the percentage of generated solutions that are correct on the first attempt, indicating how often the model successfully solves a given coding problem without needing further refinement.

Terminology

Benchmark — 5× in this paper · explained in 2 episodes
Language Models (LLMs) — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The authors create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects to evaluate whether language models can replace programmers for coding.

Motivation and Benchmark Design

Existing repository-level code generation benchmarks are insufficient because they often consist of short completions or focus on limited scale repositories, failing to represent real-world coding tasks <ref:2410.21647#pg2>. To address this, REPOCOD includes 980 whole-function generation tasks from 11 popular projects, 50.8 % of which require repository-level context <ref:2410.21647#pg4>. The benchmark meets four criteria: tasks should be real-world coding tasks such as implementing complete functions based on specifications <ref:2410.21647#pg4>, both tasks and source repositories need to be complex, and tasks should have repository-level dependencies <ref:2410.21647#pg4>.

REPOCOD Structure and Scale

REPOCOD includes 314 developer-written test cases per instance for better evaluation <ref:2410.21647#pg4>. The dataset is characterized by Complex canonical solutions (Avg. 331.6 tokens per instance) <ref:2410.21647#pg4>, and the instances are collected from repositories with an average of 2,610 files and 290,110 lines of code per repository <ref:2410.21647#pg5>. The statistics show that REPOCOD has the highest ratio of repository-level dependencies (498) <ref:2410.21647#pg8>.

LLM Performance Evaluation

The evaluation involves ten LLMs on REPOCOD, and the key findings indicate that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development <ref:2410.21647#pg4>. Specifically, LLMs are ineffective on REPOCOD (only 28.6% pass@1 under oracle-retrieval) <ref:2410.21647#pg4>. Furthermore, Higher recall in retrieving a target function’s dependencies as context improves LLMs’ performance <ref:2410.21647#pg4>.

Impact of Complexity and Retrieval

The study shows that SOTA LLMs still struggle with repository-level code generation <ref:2410.21647#pg6>. Performance negatively correlates with complexity, as overall pass rate decreases as the complexity of the context level increases <ref:2410.21647#pg9>. The analysis of retrieval methods shows that Current-File retrieval setting leads to a higher syntax error ratio <ref:2410.21647#pg6>, and Models using the Current-File retrieval generally perform better for moderate dependency counts <ref:2410.21647#pg9>.

Conclusion on LLM Capability

The research concludes that the results suggest that existing LLMs fall short of generating realistic repository-level code <ref:2410.21647#pg7>. This work underscores the need for further research in repository-level code generation <ref:2410.21647#pg9>.

Limitations

The study acknowledges limitations, stating that First, we collect data from only 11 repositories, covering a small subset of those that could be included in this benchmark <ref:2410.21647#pg10>, and Second, we evaluate only ten models, representing a subset of popular LLMs <ref:2410.21647#pg10>.

Acknowledgements

The research was supported in part by NSF 1901242 and 2006688 and a CFI fund <ref:2410.21647#pg10>. The authors thank the anonymous reviewers for their feedback on this work <ref:2410.21647#pg10>.

References

The paper cites numerous existing works, including those by Achiam et al. 2023 <ref:2410.21647#pg9>, Bogomolov et al. 2024 <ref:2410.21647#pg9>, and Touvron et al. 2023. The references listed on page 9 include works by Kiela et al. 2021 and Ott et al. 2022. The paper also cites work by Ren et al. 2020 and Papineni et al. 2002. The references on page 9 include works by Evtikhiev et al. 2023 and Achiam et al. 2023. The paper also cites work by Guo et al. 2024 and Luo et al. 2024. The references on page 11 include works by Li et al. 2023a and Ouyang et al. 2023. The references on page 13 include works by Ding et al. 2023 and Li et al. 2024b. The references on page 14 include works by Ding et al. 2023 and Liu et al. 2025. The references on page 15 include works by Ren et al. 2020 and Papineni et al. 2002. The references on page 16 include works by Nijkamp et al. 2023 and Ott et al. 2022. The references on page 17 include works by Lex et al. 2014 and Upset. The references on page 18 include works by Li et al. 2023a and Jia Li et al. 2024a. The references on page 19 include works by Claude 3.5 Sonnet and DeepSeek-V2.5. The references on page 20 include works by Starcoder 2 and the stack v2 and Wizardcoder. The final references on page 20 include works by T.J. McCabe 1976 and Nijkamp et al. 2023. The paper also cites work by Ding et al. 2023 and Li et al. 2024b. The references on page 11 include works by Starcoder 2 and the stack v2 and Wizardcoder. The references on page 5 include works by Achiam et al. 2023 and Bogomolov et al. 2024. The references on page 9 include works by Kiela et al. 2021 and Ott et al. 2022. The references on page 13 include works by Ding et al. 2023 and Li et al. 2024b. The references on page 14 include works by Ding et al. 2023 and Liu et al. 2025. The references on page 15 include works by Ren et al. 2020 and Papineni et al. 2002. The references on page 16 include works by Nijkamp et al. 2023 and Ott et al. 2022. The references on page 17 include works by Lex et al. 2014 and Upset. The references on page 18 include works by Li et al. 2023a and Jia Li et al. 2024a. The references on page 19 include works by Claude 3.5 Sonnet and DeepSeek-V2.5.

Improvements for AI systems

Bold header: Improve LLM performance on repository-level tasks by optimizing retrieval strategies. This involves using Retrieval-Augmented Generation (RAG) with specific settings, as higher recall (in range (0.5,1]) in retrieved content consistently leads to better model performance.
Bold header: Enhance context utilization by leveraging the best retrieval method for the task complexity level. For instance, Models using the Current-File retrieval generally perform better for moderate dependency counts, suggesting this setting is beneficial before complexity overwhelms retrieval effectiveness.
Bold header: Develop more robust LLM architectures capable of handling high context complexity without performance degradation. The paper shows that higher context complexity does not necessarily result in reduced LLM performance, and GPT-4o achieves a higher pass@1 rate for repository-level instances compared to file-level ones, pointing toward model improvements that can better manage this challenge.
Bold header: Create specialized LLMs for domain-specific code generation tasks. The finding that LLMs exhibit diverse strengths, as each model has uniquely solved tasks suggests focusing on training models to excel in specific repository types (e.g., Claude excels on pylint, whereas DeepSeekCoder performs better on plotly.py).
Bold header: Improve code quality by addressing common LLM failure reasons in generation. Systems should be improved to mitigate issues like frequently lack sufficient input validation and parameter handling and incorrect or incomplete implementation of core logic.

Abstract

Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.