A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets".

Jane: Here's the gist of the paper. The gist: This work introduces a benchmark and framework for evaluating systems that observe sequences of user actions in spreadsheets and predict future actions, addressing challenges related to edit histories and the complex space of spreadsheet actions.

Tom: Benchmark Dataset The researchers manually curated 52 sequences of 12K actions that recreate spreadsheets from public corpora, seeded by parametrized heuristics and LLM refinement The dataset consists of 52 high-quality trajectories of workbooks being created from start to finish, each having between 35 and 821 steps with an average of 229 steps, for 11,907 steps in total Table 7 lists the supported actions.

Jane: The operations include actions such as input(d, v) Enter v (value or matrix) at location d and border(d, s, v) Set border on side s ∈

left, right, top, bottom, outside or all: to style v at d Online Evaluation Framework The paper proposes an online evaluation that expects a prediction after each user action and accepts or rejects that prediction This procedure is repeated until either there are no actions left to do (final spreadsheet obtained) or a threshold is reached.

Tom: The online evaluation has three advantages over a teacher-forced offline evaluation at fixed time steps.

Title and authors: Tom: So we're looking at this paper, "A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets." The authors are Tejas Agrawal, Vu Le, Sumit Gulwani, and Gust Verbruggen. Basically they are tackling the fact that spreadsheet systems don't have edit histories like some other software.

Jane: Right. So the core problem is that predicting what a user will do next in a spreadsheet is hard because you don't always know what they did before, and there are just so many different ways to change a sheet. What they introduce here is a way to test systems against that complexity, by creating their own dataset and an online evaluation method.

Tom: Exactly. They’ve built this benchmark dataset manually by taking fifty-two sequences of twelve thousand actions each, pulled from public spreadsheets, and then using LLMs and some heuristics to refine those seeds <ref:2606.13802#pg1>. That gives them a lot of real-world stuff to work with for training models.

Lu: The structure they use for the data creation is interesting because they have this three-stage pipeline involving symbolic cold-start generation, LLM assisted refinement, and then human annotation. That’s a smart way to get high quality data without needing every single annotator to write everything from scratch.

Meng: But how big is this dataset? I mean, for me, the real question is if this benchmark actually reflects what happens when you try to deploy something on real traffic. They're talking about eleven thousand nine hundred seven total steps across those fifty-two sequences.

Lalam: From my perspective, the scale of these trajectories is important because it allows us to see how predictions compound over a long sequence of actions and how the model handles that accumulation of errors or successes >

Jane: That leads us into their evaluation framework, which isn't just looking at a final guess. They propose an online evaluation where you expect a prediction after every single user action and you decide whether to accept or reject it.

Tom: And they say this is better than just looking at fixed time steps because when you accept or reject a suggestion, it actually changes the state that the model continues from, which means errors can build up and later predictions depend on earlier ones >

Jane: It’s about simulating how a user actually interacts with a system. So instead of asking the model to predict ten steps ahead all at once, you test it step by step in real time.

Lu: They define metrics at three different levels: property/action level, prediction level, and emulation level. This multi-level approach means they aren't just looking for a single right or wrong answer, but they are checking how well the model predicts specific cell changes against the target state >

Meng: Three levels sounds complicated to run in production. How do you actually use that property/action level comparison when you're trying to keep latency low? Does it slow things down too much?

Title and authors: Tom: They analyze different predictors, starting with zero-shot LLMs, fine-tuned SLMs, and even classical models. The results they show are pretty telling: bigger models save more actions than smaller ones. For example, GPT-five with reasoning saved thirty-three percent on some tasks compared to GPT-five mini at only eighteen percent >

Jane: That’s a clear signal that model size matters, but then they also found something about the acceptance heuristics themselves. They saw that low-precision acceptance rules actually resulted in negative savings, meaning abstaining based on net user benefit is actually crucial >

Lu: I think the paper points out that action prediction itself is a learnable task, which is a big conceptual move. The data shows that content-heavy operations get accepted much higher than presentational ones like formatting actions >

Meng: So what does this mean for deployment? If content operations are easier to predict, maybe we should focus our training on those types of edits instead of everything equally >

Tom: They also found that the context length has a clear effect on performance, but it starts diminishing returns pretty fast. It suggests that the most important predictive signals are actually in the very recent history >

Jane: That makes sense for a user interface. If you're trying to predict what they type next, the last few things they typed are probably more useful than everything from an hour ago >

Lu: And they also looked at how context length affects performance across different settings. They found that acceptance rates tend to increase as the trajectories progress through a sequence >

Meng: That’s interesting because it means a model has to get better at predicting things later in the process, which is harder than just getting the first few steps right >

Lalam: I think this whole setup is important because it provides actionable insights for improving predictors, like finding better stopping criteria or cheaper triggers for when to suggest something >

Tom: So to wrap up on what they actually delivered with "A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets," they created the first benchmark for this kind of task. They used synthetic generation and refinement to create fifty-two tasks spanning twelve thousand actions total > <ref:2606.13802#pg1>

Jane: And the key contribution is that their online evaluation allows them to evaluate how users actually perceive using a predictor, not just whether it gets the final answer right >

Lu: The implication for the field is that action prediction is definitely a learnable task. They gave us a concrete way to test and measure progress across different parts of the prediction system >

Meng: For deployment, I’m interested in those findings about content-heavy operations being preferred. That tells me where we might focus our fine-tuning efforts on real user data >

Lalam: It sounds like this benchmark helps us figure out what makes a good suggestion for a spreadsheet system, not just which one gets the final output correct >

The paper's summary: Tom: So we just talked about how they built this benchmark dataset for spreadsheet action prediction, and now we're going to look at what the paper actually says about its main summary and what that means for us as people building these kinds of AI systems.

Jane: Right. They created fifty-two sequences of actions, over twelve thousand steps total, using public data seeded with LLMs to create high-quality trajectories, so they have this massive testbed for predicting what a user will do next in a spreadsheet.

Tom: Exactly. The paper summarizes the whole approach: they've got this benchmark and an online evaluation framework that lets you test your prediction after every single user action, accepting or rejecting it as you go.

Lu: The core idea they push is that action prediction isn't just some static guessing game; it’s a learnable task, and this benchmark proves it by showing how different parts of the system—like content versus formatting—are predicted differently.

Meng: So, when they look at their results, what's the big finding they highlight? I mean, is it just that bigger models are better overall?

Tom: It’s more nuanced than just size. They found that content-heavy operations—like putting data in a cell—get accepted much higher than presentational ones, like changing the border on a cell. That's a big hint for where we should focus our training efforts later on.

Jane: And they also looked at the context length, and they say that while there's an effect from context, it really starts flattening out quickly. It suggests that most of what you need to predict is actually right in the immediate recent history of actions.

Lu: That’s a practical constraint for deploying these models; you don't need to feed it everything from the start of the spreadsheet if only the last few moves matter most. Plus, they looked at how acceptance rates change as the sequence progresses, which means your model needs to get better at predicting things later on in a long workflow.

Tom: And they’re using these three levels of metrics—property action level, prediction level, and emulation level—to give us a detailed view of where the model is actually succeeding or failing.

Jane: It really gives us tools to target improvements. For example, they found that low-precision acceptance rules actually hurt performance; you have to balance being perfectly accurate against getting the most useful suggestions for the user.

Meng: So what does this mean for me on the engineering side? It means we can stop guessing and start designing specific triggers or stopping criteria based on these findings, instead of just tweaking parameters blindly.

Lu: Basically, they show us that action prediction is a learnable task with clear signals in the data, and now we have the framework to measure those signals precisely across different prediction strategies.

The paper's improvements: Tom: So we’ve looked at how they set up their benchmark, and now let's talk about what improvements they suggest for this whole prediction framework itself.

Jane: They aren't just giving us a static test; they’re proposing ways to make the evaluation system dynamic, something that adapts after every single action prediction.

Tom: That’s right. They suggest implementing online evaluation where after every user action, you check their suggestion and then update the ground truth trajectories based on whether you accepted or rejected it.

Lu: It gets more complex because they propose prepending inverse operations if a prediction was a false positive, which means the model has to learn to undo mistakes as well as make good ones.

Meng: That sounds like it adds a lot of computational overhead for deployment; I mean, how much extra work does that add to the inference time we’re trying to keep low?

Tom: The authors are thinking about that trade-off, and they also suggest using multi-level metrics—property action level, prediction level, and emulation level—to get a more granular look at exactly what parts of the prediction system need fixing.

Jane: It moves beyond just getting a final score; it lets researchers pinpoint whether the model is messing up formatting or data entry specifically.

Lu: They also suggest testing solvers across different settings, like single-action re-prediction versus multi-action prediction, to figure out the best way to decide when to stop making predictions entirely.

Tom: And they’ve found some insights into context length; they’re suggesting that the performance gain from adding more history drops off really fast, which means we might be able to keep our input smaller and still get good results.

Jane: That makes sense for real-world systems; you don't need a massive memory of everything that ever happened in a spreadsheet if the last few moves are what truly matter.

Lu: They also tweaked their acceptance heuristics, suggesting we analyze the trade-off between how precise our prediction is versus how much cumulative benefit it gives the user, finding that low-precision rules don't always help.

Meng: So where does this leave us? Are these improvements practical for a production environment or just for research papers?

Tom: They are aiming for both, but they’re focusing on making the decision-making process more robust so that when you deploy a predictor, it makes decisions that actually serve the user's goal.

Conclusion: Tom: So to wrap up on "A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets," the researchers have essentially given us a comprehensive tool to test and measure how well AI systems can predict what a user will do next in a spreadsheet.

Jane: It’s about creating this benchmark dataset of fifty-two sequences, spanning twelve thousand actions, that lets us see how users actually interact with spreadsheets when they are making edits.

Lu: The main point is that action prediction is definitely a learnable task, and this framework gives us the metrics to track progress across different prediction strategies.

Meng: So what’s the big picture for deployment? Does this mean we can actually start building reliable spreadsheet assistants instead of just guessing?

Tom: It means we have actionable insights, like knowing whether to focus on content-heavy operations or better stopping criteria because the data shows those are easier to predict.

Jane: They’ve shown that online evaluation is a powerful way to test this because it lets you see how users react in real-time and adjust your system based on their actual decisions.

Lu: It provides a concrete way to evaluate the entire pipeline, not just the final output, which is a really important step for building robust AI agents.

Lalam: I’m fairly sure this benchmark is solid; it covers a wide range of spreadsheet behavior and gives us plenty of data to train on.

Tom: It definitely shows that model size matters, but also that we need smarter heuristics in how we decide to suggest something based on the user's benefit.

Jane: So, for anyone listening who isn't a deep machine learning researcher, this paper shows us that predicting spreadsheet actions is a tricky problem with clear rules we can start following.

Lu: This work gives us the foundation to build better planning models because it validates how these models handle long sequences of interdependent actions.

Meng: I still think we need to see this benchmark run on actual user traffic before anyone starts shipping production-level tools based on it.

Tom: True, that’s a fair point, but this paper gives us the map so we know what to build for when the time comes.