Large Language Models for Agent-Based Modelling: Current and possible uses across the modelling cycle

The gist: This study reflects on the opportunities and challenges of Large Language Models (LLMs) in Agent-Based Modelling (ABM) by following the modelling cycle from problem formulation to documentation and communication of model results, holding a critical stance.

Conceptual Background

Agent-based Modelling (ABM) provides a bottom-up framework for simulating complex social phenomena through the behaviors and interactions of individual agents in a given environment Social simulation is a methodology for investigating complex social phenomena through the construction and analysis of artificial societies A common approach used within social simulation is Agent-based Modelling (ABM), which involves modelling individual entities (agents) that represent social actors such as individuals, groups, or organizations ABM usually emphasizes bottom-up dynamics whereby macro-level patterns emerge from micro-level interactions of heterogeneous agents ABM is a multi-phase process involving abstracting from the world / phenomenon to the target system, understanding this system, abstracting concepts, modeling relationships, and creating simulation outputs A generic ABM cycle is presented in Figure 1 Each phase consists of several iterative steps and sub-steps This iterative process applies to the main phases (refinement through multiple cycles) and also within each phase by continuous improvement in feedback loops

Large Language Models (LLMs) are advanced neural networks pre-trained on vast textual datasets and used to process and generate human-like language They typically fall into two categories: generative models (e.g., GPT) and non-generative models (e.g., BERT) Generative models predict the next token in a sequence, enabling them to produce coherent, contextually appropriate text In contrast, non-generative models are commonly used for quantifying text into mathematical vectors These vectors have the property of being geometrically close when the original texts from which they are derived are semantically close, making them well-suited for tasks like classification and retrieval of similar information Although both capture deep linguistic patterns, only generative LLMs are designed for open-ended text generation and dialogue There are several main usage areas for LLMs [48], including natural language understanding (LLMs exhibit strong generalization capabilities), text and code generation (LLMs can produce coherent, contextually appropriate, and high-quality text and code, supporting applications such as content creation, programming assistance, and automated writing), knowledge-intensive tasks (LLMs can encode extensive and possibly implicit domain-specific knowledge, enabling them to perform tasks that require broad world knowledge or specialized expertise), translations and transcriptions, and reasoning and problem solving (LLMs can support complex tasks through pattern-based inference, analogical reasoning, and step-by-step problem decomposition) The use of LLMs has been associated with several ethical concerns [23], including bias and fairness, privacy and data security, misinformation and disinformation, transparency and accountability, intellectual property and plagiarism, access and inequality

How it works

The study addresses two research questions: 1. Where and how are LLMs used in the ABM cycle? 2. Where and how could LLMs be used in the ABM cycle The authors followed the modelling cycle, from problem formulation to documentation and communication of model results, and held a critical stance They identified that almost 80% of the papers (n=17) are published in computer science, three in engineering [28] [22] [50], and one in each mathematics [12] and social sciences [16] The most common use (n=20, 91%) involves implementation, particularly focusing on agents powered by LLMs for reasoning, deliberation /decision-making, or communication Code generation (in NetLogo) is the main focus in one study [31] Interpretation of model results is the main focus in one study using LLMs to generate narrative explanations that communicate agents’ perspectives on key life events [30] As such, it can be concluded that the inherent potential of LLMs has not yet been fully leveraged throughout the ABM cycle

Problem Formulation

LLMs can assist in problem formulation by helping with several tasks:

1. conducting literature reviews by summarizing research findings across disciplines

2. supporting communication with stakeholders through natural language dialogues to gather goals and constraints

3. generating and evaluating hypothetical scenarios to help refine problem definitions, objectives, and research questions for the simulation

4. translate text (e.g., literature, stakeholders input) into the working language of the modeler

Potential pitfalls include overreliance on LLM-generated content, which may introduce biases from the training data (e.g., implicitly attributing certain characteristics depending on the gender or ethnicity of the agent), hallucinate facts, reduce the breadth of possible models and research activities to pre-established norms To mitigate this, modelers should critically evaluate LLM outputs, combine them with expert knowledge, and validate findings through consultation with stakeholders and domain experts

System Analysis

LLMs can assist in system analysis by helping with several tasks:

1. extracting mechanisms (parsing large volumes of domain-specific literature to identify relevant theories and summarize relevant processes, feedback loops, mechanisms, and causal relationships) and generating candidate system structures, including agent typologies, interaction rules, and environmental features (based on descriptive text, such as field notes or stakeholder inputs)

2. conceptual mapping (generating or supporting the development of influence diagrams or interaction schematics based on textual descriptions)

3. hypothesis generation (proposing plausible agent rules or system mechanisms, including alternative explanations for observed behaviors or empirical patterns)

4. cross-domain integration (facilitating connections between theoretical frameworks or empirical findings from different disciplines to enrich system representations)

5. providing contextual argumentation (based on previous models or on primary data, by offering narratives explaining agent behaviors in given scenarios or by imitating the view points of various stakeholders)

Risks include misinterpreting or oversimplifying complex dynamics when relying on LLMs, as they might suggest mechanisms that sound plausible but lack empirical support or misrepresent causal directionality To mitigate this, LLM outputs should be treated as provisional aids, not definitive sources

Conceptualization

LLMs can assist in conceptualization by helping with several tasks:

1. agent and environment specification (translating textual system descriptions into structured outlines of agents, their goals, decision-making processes, and the properties of the environment they inhabit)

2. rule formulation (suggesting candidate behavioral rules for agent actions based on theoretical inputs, empirical cases, or stakeholder narratives)

3. model articulation (helping to draft readable and structured descriptions of the model using established templates such as inherent in the ODD protocol [19])

1. Agent Reasoning Integration: Integrate LLMs as cognitive components within agents to simulate natural language reasoning, narrative behavior, or adaptive decision-making in complex or poorly defined environments. This allows agents to perform tasks beyond traditional rule-based logic by reasoning based on received messages via an LLM component.

2. Problem Formulation Enhancement: Use LLMs to conduct literature reviews by summarizing research findings across disciplines and generate and evaluate hypothetical scenarios to help refine problem definitions, objectives, and research questions for the simulation. This assists modelers who are not sufficiently acquainted with disciplinary terminology in synthesizing large bodies of knowledge.

3. System Analysis Augmentation: Employ LLMs to extract mechanisms by parsing large volumes of domain-specific literature to identify relevant theories and summarize relevant processes, feedback loops, mechanisms, and causal relationships. This helps generate candidate system structures, including agent typologies and interaction rules based on descriptive text like field notes or stakeholder inputs.

4. Conceptualization Support: Utilize LLMs for agent and environment specification by translating textual system descriptions into structured outlines of agents, their goals, decision-making processes, and the properties of the environment they inhabit. This includes suggesting candidate behavioral rules for agent actions based on theoretical inputs or empirical cases.

5. Implementation Acceleration: Leverage LLM code generation to translate structured model descriptions into executable code in platforms like NetLogo or Repast at speeds orders of magnitude greater than achievable through human efforts. This significantly reduces iteration times and increases the number of iterations during model development.

6. Verification Support: Use LLMs for trace interpretation by analyzing simulation logs or stepwise outputs to detect anomalies or deviations from expected behavior, generating stories based on rules executed by agents that are afterwards checked for meaningfulness by the modeller.

7. Validation Assistance: Employ LLMs for narrative validation by generating or analyzing scenario-based narratives that describe plausible sequences of agent behavior and comparing these with stakeholder accounts or case studies. This assists in expert emulation to evaluate whether model outcomes are perceived as realistic or plausible.

8. Interpretation and Communication: Utilize LLMs for pattern discovery by identifying trends, clusters, or tipping points in outputs and for narrative synthesis by generating structured or story-based explanations of model behavior over time. This aids in stakeholder-tailored reporting by adapting result summaries to different audiences.

9. Documentation Automation: Employ LLMs for automated report generation by summarizing model structure, parameters, and outputs in clear language from code bases or ODD descriptions. This facilitates creating audience adapted documentation, such as executive summaries or policy briefs, while maintaining transparency and reproducibility.

The emergence of Large Language Models (LLMs) with increasingly sophisticated natural language understanding and generative capabilities has sparked interest in the Agent-based Modelling (ABM) community. With their ability to summarize, generate, analyze, categorize, transcribe and translate text, answer questions, propose explanations, sustain dialogue, extract information from unstructured text, and perform logical reasoning and problem-solving tasks, LLMs have a good potential to contribute to the modelling process. After reviewing the current use of LLMs in ABM, this study reflects on the opportunities and challenges of the potential use of LLMs in ABM. It does so by following the modelling cycle, from problem formulation to documentation and communication of model results, and holding a critical stance.