VAIOM: A Continuous-Input Transformer for FX Return Modeling

Lead

Financial time series do not naturally look like text. Prices, spreads, volatility measures, gaps, and asset identifiers form continuous, heterogeneous, and noisy observations, while decoder-only next-token models are usually built around discrete symbols. The arXiv paper “VAIOM: Continuous-Input, Discrete-Output Decoder-Only Financial Sequence Modeling” addresses that mismatch by changing the interface: it feeds continuous financial event vectors into a Transformer, but keeps the output as a discrete probability distribution over the next return bucket.

Core ideas

• Continuous input, discrete likelihood: VAIOM preserves numerical structure at the input by representing each financial event as a continuous multivariate vector. The target is a categorical distribution over the next volatility-normalized return bucket, enabling cross-entropy training and likelihood evaluation.
• Decoder-only sequence modeling: The model is used for probabilistic next-return modeling on one-hour foreign-exchange bars. It is not presented as a trading system; it estimates distributions over future return categories.
• Hybrid continuous-input design: The selected 0.9M-parameter Hybrid Continuous Input model combines continuous event features with categorical asset metadata.
• Mixture and auxiliary heads: The architecture includes a Mixture-of-Market-States return head, plus auxiliary objectives for gap, volatility regime, and ordinal structure. These are intended to shape internal representations rather than merely optimize a single next-step target.
• Full-sequence supervision: Instead of training only on the last position of a sequence, the paper uses supervision across the full sequence, which the validation experiments find beneficial.

Results and interpretation

The evaluation protocol is one of the more important parts of the paper. Models and preprocessing are fit only on pre-2024 training data. Checkpoints are selected using a 2024H2 validation split, then evaluated without refitting on two test periods in 2025. Under this setup, the authors report that every model, across three independent training seeds, beats a fixed single-bar LightGBM baseline in both test halves. For the canonical checkpoint, the paired gains over LightGBM are 0.029 and 0.043 bits per event.

Those gains should be read carefully. A better likelihood score is not the same as a deployable trading edge, and the paper does not claim that the model produces profitable strategies after costs, slippage, or risk constraints. What it does suggest is that a sequence model can extract probabilistic information beyond what a fixed one-bar gradient boosting baseline captures.

The ablation results are arguably more informative than the headline comparison. Under the same categorical return objective, continuous inputs outperform discrete-token inputs. Full-sequence supervision outperforms last-position training. Auxiliary representation shaping and the mixture-structured return head also improve return likelihood in controlled comparisons. A capacity study further finds that the smallest evaluated complete architecture rung achieves the strongest validation likelihood on this corpus, a useful reminder that financial sequence modeling does not automatically reward larger models.

Why it matters

VAIOM is interesting because it does not force market data to imitate language tokens. Instead, it separates the problem into two parts: keep continuous market observations continuous where numerical relationships matter, and use a discrete output distribution where likelihood-based training is convenient. That design could be relevant for other noisy, irregular, non-text domains where decoder-only Transformers are attractive but naive tokenization loses too much structure.

For quantitative researchers, the paper is less a ready-made trading recipe than a modeling template: representation, supervision placement, auxiliary targets, and time-aware validation all matter. Its contribution is to show a plausible way to adapt next-token-style architectures to financial event streams while maintaining a careful out-of-sample evaluation protocol.

Source: arXiv