Scaling RNA Foundation Models through Iterative Refinement
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: Today's paper: "Scaling RNA Foundation Models through Iterative Refinement".
Jane: The gist: Eco-RNA introduces an 80Mparameter RNA foundation model that makes inference depth adjustable through learned iterative refinement,
Tom: First, who's behind it and why it matters.
Title and authors: Tom: So, what does the actual mechanism look like? They are using a shared, input-conditioned Transformer core and applying it repeatedly based on a learned iteration count.
Jane: That sounds like they’re reusing the same core but feeding it different information at each step, which is interesting because it suggests that the refinement isn't just adding more layers.
Lu: Exactly. They use a recurrent-depth framework where one shared CORE is reused K times, and they have a prelude representation that gets injected at every iteration to supply sequence information directly into the update process.
Tom: So, the math involves this recurrence relation s(k) equals N r alpha⊙s(k-one) plus delta⊙Be, k equal one through K <ref:2610.13106#pg3>. It's a learned way of deciding how much of the previous state to keep versus how much new input is added at each step.
Meng: That sounds like a complex update rule to manage for training, but if it works well, it means we can tune the depth without retraining the entire model structure from scratch for every use case.
Lalam: The paper also mentions their training objective involves masked reconstruction loss at sampled terminal depths, along with relative displacement penalty and a one-sided reconstruction anchor.
The paper's summary: Jane: That leads into the training objectives, right? They aren't just doing simple prediction tasks; they are optimizing the model to have these specific properties.
Tom: They use this one-sided reconstruction anchor loss, which is pretty novel. It basically trains the model to favor deeper endpoints whose reconstruction loss is higher than a shallower reference from the same path.
Lu: That anchor loss reinforces those deeper reconstruction points, effectively training iterative computation within a self-supervised backbone. It's like they are guiding the model to learn better representations at different levels of refinement simultaneously.
Meng: From an engineering standpoint, that sounds like it helps stabilize training because you’re giving the model a targeted way to improve its performance at different scales without just relying on one single loss signal.
Lalam: And they show results on the RNAGym fitness benchmark, where increasing depth from one iteration up to four iterations raises the aggregate absolute Spearman correlation from zero point two four two to zero point two eight six, and at four iterations, Eco-RNA surpasses Evo two 40B on all three aggregate fitness metrics despite being five hundred times fewer parameters <ref:2610.13106#pg2>.
The paper's improvements: Tom: So, to wrap this up, the main point from "Scaling RNA Foundation Models through Iterative Refinement" is that internal inference computation can be treated as a scaling dimension for these RNA foundation models.
Jane: It seems like they've established that a single recurrent checkpoint can give you several different operating points for compute and performance. They are showing how to use this iterative refinement to tune the depth based on what you need for a specific task.
Lu: The results connect this internal inference capability to both the way the model learns its representations and how well it performs downstream predictions, specifically showing gains in splicing and aptamer assays where deeper predictions matter most.
Meng: For deployment, that means we can have a fixed-size model that we can reconfigure dynamically for different tasks just by changing the iteration count at inference time, which is a huge win for efficiency.
Lalam: The paper suggests that this depth adjustment reduces held-out reconstruction loss at multiple stages of pretraining and those depth-dependent gains persist even after adapting to sequence labels or chemical reactivity.
Conclusion: ---: Conclusion ---
Tom: So, to wrap up "Scaling RNA Foundation Models through Iterative Refinement," the main point is that internal inference computation can be treated as a scaling dimension for these RNA foundation models.
Jane: It seems like they've established that a single recurrent checkpoint can give you several different operating points for compute and performance. They are showing how to use this iterative refinement to tune the depth based on what you need for a specific task.
Lu: The results connect this internal inference capability to both the way the model learns its representations and how well it performs downstream predictions, specifically showing gains in splicing and aptamer assays where deeper predictions matter most.
Meng: For deployment, that means we can have a fixed-size model that we can reconfigure dynamically for different tasks just by changing the iteration count at inference time, which is a huge win for efficiency.
Lalam: The paper suggests that this depth adjustment reduces held-out reconstruction loss at multiple stages of pretraining and those depth-dependent gains persist even after adapting to sequence labels or chemical reactivity.
Tom: Yeah, it’s about linking the recurrent state dynamics directly to a criterion in prediction space. It connects representation learning and prediction training in a way that was previously separate.
Jane: That's really interesting because it shows how you can use the model's own internal computation to guide its learning process during pretraining.
Lu: The separation of the task-level criterion for training additional recurrent computation within a shared sequence model is something that opens up a lot of possibilities for future architectures.
Meng: If we can tune the budget this way, it means we can optimize compute per prediction much more effectively on real traffic, not just on some idealized benchmark.
Lalam: I'm sure these findings about Eco-RNA will keep researchers looking at how to scale these foundation models for more complex biological sequences.
Tom: We'll be talking about that next week when we look at some of those agents and alignment papers from arXiv, but for now, this is how we use internal computation to tune the model’s depth.
Pengan Chen, Liang Hong, Xinyuan Liu, Han Huang Ziqian Lin, Jiayang Chen, Yimin Fan, Jiyue Jiang Yu Li
Department of Computer Science and Engineering, The Chinese University of Hong Kong
cs.LG
Submitted: 2026-10-09
Updated: 2026-10-09
Comments: Technical Report. 20 pages, 5 figures, 9 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 85/100
The gist: The gist: Eco-RNA introduces an 80Mparameter RNA foundation model that makes inference depth adjustable through learned iterative refinement, establishing internal inference computation as an
Key concepts
- Iterative Refinement
- Eco-RNA refines sequence representations by repeatedly applying a shared core transformation. The number of times this refinement happens (the iteration count) is learned during training and controls the effective depth of the model without increasing the total parameter count. This allows for adjustable inference depth.
- Recurrent Depth Framework
- The architecture reuses a single shared Transformer core multiple times, decoupling the executed inference depth from the fixed number of model parameters. A recurrent update equation dictates how information is retained and updated across these repeated applications, enabling internal scaling.
- One-Sided Reconstruction Anchor
- A training criterion that anchors deeper reconstructions to a shallower reference point within the same trajectory. This loss function encourages the model to learn meaningful, progressively refined representations during the iterative process, guiding the model toward better deep outputs.
Terminology
Summary
The gist: Eco-RNA introduces an 80Mparameter RNA foundation model that makes inference depth adjustable through learned iterative refinement, establishing internal inference computation as an effective scaling dimension for RNA foundation models.
How it works
Eco-RNA utilizes a shared, input-conditioned Transformer core that refines sequence representations through repeated application where the iteration count is selected after training and controls effective depth at fixed model parameters. The architecture employs a recurrent-depth framework where one shared CORE is reused K times, decoupling executed depth from parameter count. This allows for internal scaling by repeatedly applying a shared transformation to a latent representation.
The recurrent update mechanism is defined by the equation s(k) = Nr α ⊙ s(k-1) + δ ⊙ Be, k = 1,..., K. Here, the term α ⊙ s(k-1) retains a channelwise fraction of the preceding state, while δ⊙Be makes the input-conditioned prelude representation directly available at every iteration. The learned coefficients for the retention coefficient are shared across sequences, positions, and iterations.
Training Objective
The training objective couples masked reconstruction at sampled terminal depths with two criteria on the recurrent trajectory: relative displacement and a one-sided reconstruction anchor. The combined objective is L = LMLM + λfpLfp + λaLanchor.
The relative displacement penalty, Lfp, penalizes the final recurrent update relative to the scale of the preceding state. The one-sided reconstruction anchor reinforces deep endpoints whose reconstruction loss exceeds a shallower reference from the same trajectory. This formulation trains iterative computation within a self-supervised sequence backbone.
Performance and Scaling
Eco-RNA advances the RNAGym frontier through adjustable recurrence, showing that increasing depth from one to four iterations raises aggregate absolute Spearman correlation from 0.242 to 0.286. At four iterations, Eco-RNA surpasses Evo 2 40B on all three aggregate fitness metrics despite 500× fewer parameters.
The model also outperforms the evaluated baselines in non-coding RNA family classification, RNA secondary-structure prediction, and chemical probing profile prediction. The iteration configurations advance the compute–performance Pareto frontier among the evaluated models.
Task-Specific Benefits
Additional depth produces different benefits across assays, with the largest improvements occurring in splicing and aptamer assays. Finer-grained predictions retain benefits from refinement after sequence-level classification has largely saturated. For secondary structure prediction, increased recall at sustained precision accounts for most of the improvement.
Conclusion
Eco-RNA establishes internal inference computation as a scaling dimension for RNA foundation models. A single recurrent checkpoint provides several compute–performance operating points and advances the RNAGym Pareto frontier. The results connect this inference capability to both representation learning and downstream prediction. Additional depth reduces held-out reconstruction loss at multiple stages of pretraining, and depth-dependent gains persist after adaptation to sequence labels, nucleotide pairs, and chemical reactivity. The training formulation links recurrent state dynamics to a criterion in prediction space. The observed nucleotides specify the learning target, and the reference determines when to strengthen that supervision. This separation provides a task-level criterion for training additional recurrent computation within a shared sequence model.
How it works
The compute accounting estimates the single-forward count as FEco(K, L) = D(K)Fblk(L) + 2(K + 1)Ld2 + 2LdV for a sequence length L". For L = 512, the count is 90.011, 180.608, and 301.404 GFLOPs at K = 1, 4, and 8.
How it works
The training system samples the terminal depth Ki independently for every sequence using a Poisson–lognormal distribution. The backward window Wbwd controls recurrent activation storage, and gradients are retained through the final min(Ki, Wbwd) iterations.
How it works
Inference involves specifying K explicitly to control the computation budget while reusing a fixed initial-state table at every depth<ref:2610.13106p5, "At inference, K is specified explicitly.
Improvements for AI systems
-
textbfAdjustable Inference Depth for Computation Allocation and Performance Tuning: Eco-RNA allows users to select an
iteration count chosen at inference
to control effective depth while keeping model parameters fixed (K changes compute, not parameters
). This enables users toadvance the compute–performance Pareto frontier among the evaluated models
by choosing configurations like K=4 whichsurpasses the current SOTA Evo 2 40B on all three aggregate fitness metrics despite 500× fewer parameters.
-
textbfEnhanced Sequence Representation Learning via Recurrent Refinement: The system employs a
shared, input-conditioned Transformer core
that refines sequence representations throughrepeated application,
ensuring that the shared update retains access to sequence information while learning aninput-conditioned shared update.
-
textbfImproved Self-Supervised Pretraining Convergence: Training objectives include a novel one-sided reconstruction anchor (
Lanchor
) which is defined as reinforcement when a deeper prediction receives supervision because it exceeds a shallow reference loss, allowing the system to selectively reinforcedeeper endpoints whose reconstruction loss exceeds a shallower reference from the same trajectory.
-
textbfTask-Specific Scaling for Prediction Granularity: The model demonstrates that
finer-grained predictions retain benefits from refinement after sequence-level classification has largely saturated,
suggesting that additional iterations are useful for tasks like per-nucleotide chemical reactivity wheresmaller gains persist to K=7, where all three means reach their best observed values.
-
textbfGuaranteed Control over Prediction Budget: Users gain the ability to
select depth according to the assay and the target metric, using validation to identify where additional iterations remain useful,
allowing them to choose an inference budget that optimally balances computation and prediction quality for specific biological questions.
Sources
- Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems
- Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions
- Scalable Deep Learning for RNA Secondary Structure Prediction
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- DeepLoop: Depth Scaling for Looped Transformers
- Parcae: Scaling Laws For Stable Looped Language Models
- ncRNA Classification with Graph Convolutional Networks
- GLU Variants Improve Transformer
- Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
- SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks