Training Code Models to Write Faster Programs, Not Just Correct Ones

Introduction

Reinforcement learning for code has largely focused on a simple loop: generate a program, run hidden tests, and reward outputs that pass. The next step sounds obvious—also reward solutions that run faster. But the paper “Reinforcement Learning for Code Optimization” argues that this simple idea breaks down in practice. Once execution time becomes part of the reward, small timing fluctuations, sparse signals, and GRPO instability can dominate the learning process.

Key ideas

• Speed is a noisy reward signal. Runtime depends on the sandbox, input scale, measurement variance, and other system-level details. If these are not controlled, a model may chase noise rather than true algorithmic improvement.
• The authors introduce DMC-Optim. The benchmark is designed around larger optimization tests and a calibrated sandbox, making it more suitable for measuring whether generated programs are meaningfully faster.
• Correctness and speed must be composed carefully. The RL environment combines both objectives, while an offline simulator is used to identify promising reward configurations before expensive training runs.
• GRPO needs adaptation. Timed execution creates a sparser and noisier setting than pass/fail correctness. The paper modifies training and evaluation to make learning from runtime feedback more stable.

Results and impact

On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% for Qwen 2.5 7B, and from 30.7% to 50.4% for CWM 32B. Gains become even larger at stricter thresholds: at top-30%, CWM 32B shows a 125% relative improvement, while pure correctness scores are preserved. When the timing sandbox is degraded, robust optimization RL still reaches 100% to 200% improvement over standard RLVR depending on the evaluation criterion.

On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Compared with the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements, 14% versus 28%.

The broader message is that code intelligence is moving beyond “does it pass?” toward “does it run efficiently?” This matters for coding assistants, automated refactoring, and systems where generated code may be executed repeatedly at scale. The paper also serves as a warning: optimization-oriented RL cannot rely on naive runtime rewards. Without reliable tests, calibrated measurement, and robust reward design, reinforcement learning may optimize the benchmark noise instead of the program.

Source: Hugging Face Daily Papers