RIBOSPAN Brings Native Long-Context Modeling to Full-Length RNA

Introduction

RNA foundation models have often been trained on windows that are shorter than many complete transcripts, especially messenger RNAs. When a transcript is split into fragments, a model may lose relationships between coding and non-coding regions or fail to use information that lies far apart in the sequence. RIBOSPAN addresses this limitation by making long-context modeling part of pretraining rather than treating it as an inference-time extension.

What the paper introduces

• Native 10K context. RIBOSPAN has 1.61 billion parameters and is pretrained with sequence lengths of up to 10,240 nucleotides. The model is bidirectional, allowing each position to use information from both sides of the RNA sequence.
• Single-nucleotide resolution. Instead of relying on coarse units, the model tokenizes RNA at the nucleotide level. This preserves fine-grained information while allowing the system to process full-length transcripts in one context.
• Isolated sequence packing. Training can pack multiple sequences into a batch, but attention is isolated between them. This is intended to improve utilization without allowing unrelated examples to contaminate one another’s representations.
• Evaluation beyond reconstruction. The authors examine nucleotide reconstruction, a controlled long-context representation benchmark, and frozen representations across RNA types. Native 10K models retain strong reconstruction behavior at the longest tested context and show useful contextual responsiveness and separation between context-specific representations.
• Robustness under corruption. Continued pretraining with a 40% masking rate improves recovery when the input is heavily corrupted, while preserving representation quality according to the reported evaluations.
• A gap between scaling and pretraining. YaRN-based inference-time scaling recovers much of the contextual organization lost when short-context models are directly extrapolated. However, it also causes considerably more distal representation diffusion. The comparison supports the case for learning long contexts natively when localization of information matters.
• From embeddings to sequence design. Using the same backbone, the work develops a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign. It also explores synonymous-codon diffusion, which aims to optimize coding sequences while preserving the encoded protein sequence.

Why it matters

The main contribution is not simply a larger context number. RIBOSPAN treats the complete transcript as the modeling unit and evaluates whether information remains organized when the sequence becomes long. This distinction is important: a model that accepts a longer input is not necessarily a model that has learned stable long-range relationships. The reported localization results suggest that native long-context training can reduce the spread of representation changes caused by local perturbations.

The work also illustrates a practical path from foundation-model pretraining to RNA design. A model that represents an entire transcript can potentially provide a more coherent substrate for generation and optimization than one built only from short fragments. At the same time, the reported results concern reconstruction, representations, and model-based generation frameworks. They do not by themselves establish biological function in the laboratory. Experimental validation remains necessary for any designed mRNA or optimized sequence.

RIBOSPAN therefore represents a substantial step toward transferable, full-transcript RNA modeling: long-context capability is learned at the foundation-model stage and then connected to downstream generation and redesign tasks.

Source: Hugging Face Daily Papers