Back to articles
Evaluation & Benchmarks

Why Do Multimodal Models Change Their Answers When a Spectrum Is Rewritten?

3 min read

Introduction

An optical spectrum can be written as a table in nanometers, micrometers, scientific notation, or electronvolts. It can also be shown as a plot whose horizontal axis uses wavelength or photon energy. These formats may encode the same physical evidence, so a reliable scientific model should reach the same decision regardless of presentation. A new study highlighted by Hugging Face Daily Papers shows that current multimodal systems are far from meeting that standard: changing only the representation moved accuracy by up to 41 percentage points within a single model.

The work examines a second capability that is just as important as getting an answer right. If the part of the spectrum needed to resolve a question is missing, the model should say that the result is undetermined rather than infer an answer from incomplete evidence. This distinction matters in scientific settings, where a confident guess can be more damaging than an explicit refusal.

Key findings

  • Equivalent evidence produces unequal results. The benchmark uses measured optical spectra from the MOCLIP silicon-metasurface collection and from gold nanoparticles. Labels are computed from the spectra themselves. Each example is rendered as four tables—nanometers, micrometers, scientific notation, and electronvolts—and as plots in wavelength and photon energy.
  • Coordinate choice can help or hurt. With the underlying evidence held constant, asking in photon energy rather than wavelength improved some model-view combinations by 16–50 percentage points, but reduced others by as much as 25 points. The largest instability appeared in photon-energy tables and plots.
  • Missing extrema are often ignored. The researchers created partial spectra by removing data inside or outside the queried range. When the relevant extremum was removed, unadapted systems rarely responded with “undetermined.” A gap in the middle of a range was especially difficult: remaining values on both sides could be mistaken for sufficient evidence.
  • Fine-tuning can teach the wrong rule. LoRA adaptation improved d3-nano’s comparison performance on the table formats used in training. However, part of its improvement on unanswerable cases came from a shortcut: it learned to abstain whenever an input looked truncated, without checking whether the removed segment actually contained the answer.
  • Counterexamples matter. Training with truncations that remained answerable eliminated this shortcut across all six seeds. On held-out structures, abstention rates after removing the answer-bearing region versus another region differed by 90–100 percentage points. Adding the same amount of ordinary task data left the shortcut in five of six seeds.

Why it matters

The results argue for evaluating scientific multimodal models on more than average accuracy. A robust system should be invariant to physically equivalent representations, should not let unit conventions determine its conclusion, and should recognize when the available evidence is insufficient.

The experiments reveal two opposite failure modes. Unadapted models tend to answer when they should abstain. A poorly designed fine-tuning recipe can produce the reverse behavior: the model abstains whenever it sees a truncated input, including cases where the answer remains recoverable. The first failure reflects weak evidence awareness; the second is format-based overfitting.

For scientific decision systems, training data should therefore combine unanswerable examples with truncated examples that remain answerable. Evaluation should also span units, notation styles, and plot formats rather than treating one rendering as representative. The authors release the tasks, per-response results, and analysis code, offering a basis for reproducing the findings and building more demanding benchmarks for scientific multimodal reasoning.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles

CCTest · Blog
TurtLES: A Large-Scale Benchmark for 3D Turbulent Neural PDE Surrogates
Evaluation & Benchmarks
cctest.ai

TurtLES: A Large-Scale Benchmark for 3D Turbulent Neural PDE Surrogates

TurtLES introduces a 14 TB benchmark of three-dimensional turbulent cylinder wakes for testing neural PDE surrogates under realistic geometry, irregular meshes, and long temporal rollouts. Its evaluations expose recurring weaknesses in stability, small-scale energy preservation, and large-domain learning.

Read more