TokaMark Robustness Benchmark Exposes Sensor-Failure Weaknesses in Fusion Diagnostic Models

Introduction

Plasma diagnostic models for tokamak fusion devices are often judged on clean and complete sensor streams. That is convenient for benchmarking, but it does not match the conditions under which a future fusion control system would have to operate. In real machines, acquisition systems may start late, individual diagnostics can fail, and dropouts may appear exactly when the plasma is approaching a disruption.

This paper introduces a systematic robustness benchmark for plasma diagnostic machine learning using TokaMark, a dataset containing 11,573 MAST tokamak shots. The author evaluates XGBoost, LSTM, Transformer, and the TokaMark CNN baseline across six physically motivated sensor-failure scenarios and three imputation strategies.

Key points

• Robustness becomes the central metric, not an afterthought. The work introduces a Robustness Score (RS) to compare architectures under sensor corruption in a standardized way. This moves evaluation beyond clean-data accuracy.
• Failures near the disruption are especially damaging. When corruption is injected into the final timesteps of the model window, sequence models degrade sharply. The abstract reports a +212% NRMSE increase for LSTM, compared with +37% for XGBoost.
• Imputation depends on the failure pattern. Forward-fill imputation nearly removes the impact of random dropout for sequence models, taking LSTM degradation from +57% to roughly 0%. But it provides little help when the end of the input window is corrupted.
• Regression error and alarm behavior can disagree. In shot-level alarm evaluation using ground-truth disruption timestamps, LSTM detection falls to TPR=0.00 under proximate sensor failure. Mean-fill imputation restores it to TPR=1.00, a reversal of what the NRMSE trend alone would suggest.
• Plasma current is the most important diagnostic. Removing it causes performance degradation of +73% to +140% across architectures, making it the single most critical signal in the benchmark.

Why it matters

The main contribution is not another larger architecture, but a more realistic way to ask whether plasma diagnostic ML can be trusted. If future commercial fusion reactors rely on learned models for disruption warning or operational support, those models must remain useful when sensors are incomplete, delayed, or corrupted at the worst possible time.

The findings also complicate a common assumption: sequence models are not automatically more reliable just because they model temporal structure. They may be highly sensitive to the latest timesteps, which are precisely the timesteps most likely to be compromised near a disruption. In contrast, a feature-based statistical model such as XGBoost can be more stable in some failure settings.

The study suggests that future benchmarks for fusion ML should report more than performance on clean data. They should include robustness under realistic sensor failures, the effect of imputation, and shot-level alarm metrics tied to actual disruption timestamps. With code, data, and trained checkpoints made available, this benchmark can also serve as a reproducible starting point for broader robustness studies in AI for Science.

Source: Hugging Face Daily Papers