, 6 min read, llms, diffusion, reinforcement-learning
ICML 2026 Outstanding Paper: any-order decoding looks like a reasoning upgrade for diffusion LLMs, but in RL it lets them skip the tokens where reasoning forks.
I keep half an eye on diffusion language models because the pitch sounds like a cheat code. Instead of writing strictly left to right, a dLLM starts from a block of masked positions and fills them in over several steps, and it gets to choose which position to resolve next. My gut reaction was that this has to be a strict upgrade. More possible orders should mean more ways to reach an answer, not fewer.
"The Flexibility Trap," by Zanlin Ni and coauthors from Tsinghua and Alibaba, argues that for math and code this intuition is backwards. ICML just named it one of this year's two Outstanding Papers1, and after reading it I think the award is deserved, mostly because the fix it proposes is so unglamorous.
The case for arbitrary order is easy to state. An autoregressive model has one path through a sequence. A model that can fill tokens in any order has many paths, and the paper's own abstract puts the intuition fairly: that flexibility "implies a solution space that strictly supersets the fixed autoregressive trajectory." If you are doing reinforcement learning on top, a bigger reachable space should mean more to explore.
Then the abstract turns around. For "general reasoning tasks (e.g., mathematics and coding), arbitrary order generation may in fact limit the reasoning potential of dLLMs."
The evidence is Pass@k, which the authors use as a proxy for how high RL can push a model: sample k answers, count a problem as solved if any of them is right. On LLaDA-Instruct, Dream-Instruct and LLaDA 1.5, across GSM8K, MATH-500, HumanEval and MBPP, left-to-right order has the steeper Pass@k curve. Arbitrary order is competitive at k=1, often better, and then falls behind as k grows. The number that stuck with me: on HumanEval at k=1024, 21.3% of problems are solved only by left-to-right decoding, and 0.6% only by arbitrary order. The supposedly bigger solution space turns out to be, in practice, mostly a subset of the smaller one.
The paper's explanation is what it calls entropy degradation, and it is the part worth reading slowly.
Reasoning text is not uniformly hard. Most tokens are close to forced, but a few "forking tokens" decide which way the argument goes. Those are high-entropy: several continuations are reasonable. A confidence-based sampler, which is how dLLMs usually pick the next position, resolves the easy, high-certainty tokens first and keeps skipping the forks. The authors checked which tokens get deferred most, and it is mostly connectives and transition words: "Therefore", "Thus", "Since".
By the time the sampler comes back to a fork, the text around it is already written, so the fork is no longer a real choice. In the paper's words, "the high entropy originally inherent to the fork is suppressed." The model is filling in a word that bridges to a conclusion it already committed to. Across a batch of rollouts that gives you what the abstract calls "a premature collapse of solution coverage", and an RL method like GRPO, which learns by comparing many sampled attempts, has nothing new to learn from.
I expected the fix to be some clever sampler that steers toward fork tokens. It isn't. JustGRPO drops arbitrary order during RL. Rollouts are generated left to right, and because the model is then treated as an autoregressive policy over the same network (future positions stay masked, no causal attention mask is added), the sequence likelihood becomes "an exactly computable likelihood", a product of next-token probabilities. That removes the ELBO-style approximations that RL on dLLMs usually needs, and the loss is plain GRPO. The README says the core of it, grpo.py, "fits in ~60 lines".2
On LLaDA 8B Instruct, the paper reports 89.1 on GSM8K, 45.1 on MATH-500, 49.4 on HumanEval and 52.4 on MBPP at generation length 256. The repo later added LoRA results at the same length (89.6 on GSM8K, for example).
One detail that is easy to miss: those headline numbers are measured unmasking one token per step. The parallel decoding claim is a separate experiment, using a training-free sampler (EB-Sampler) at several tokens-per-step settings. There the JustGRPO model holds up better than base LLaDA-Instruct as parallelism goes up. So "keeps parallel decoding" is true, but it is a different chart from the 89.1.
I can't check any of this on a real 8B model, so I wrote a small toy instead. It is not a reproduction. It is about 80 lines of Python over a hand-made distribution, and its only job is to make the mechanism concrete.
There are 8 equally likely "solutions", each 8 tokens long. Position 0 is the fork: every solution has a different token there, so it carries 3 bits. Positions 1 to 7 are easy: solution i has a 1 at position i and everyone else has a 0, so each easy position is 0 with probability 7/8. Two decoders sample from the exact conditionals. One goes left to right. The other always resolves whichever masked position it is most confident about, which is the dLLM-style rule. Both sample at the same temperature.
def next_pos(rule, fixed):
masked = [p for p in range(M) if p not in fixed]
if rule == "left-to-right":
return masked[0]
# most confident masked position first
return max(masked, key=lambda p: max(cond(fixed, p).values()))The temperature 1.0 run is the interesting one. Both decoders produce all 8 solutions with probability 0.125 each, with the same expected coverage (7.06 distinct out of 8 in 16 samples). The fork still gets degraded in the confident-first decoder: its entropy at the moment it is decoded drops from 3.00 bits to 0.25. But with exact sampling, the uncertainty just moved into the easy tokens, so nothing was lost.
At temperature 0.6, the setting the paper uses for its Pass@k analysis, it changes. Left to right is untouched: the fork is sampled first, it is uniform so sharpening it changes nothing, and everything after it is forced. The confident-first decoder now leans toward 0 on every easy token, and the solutions that need a 1 early get squeezed:
left-to-right
P(each solution): [0.125, 0.125, 0.125, 0.125, 0.125, 0.125, 0.125, 0.125]
fork entropy at the moment it is decoded: 3.00 bits
expected distinct solutions in 16 samples: 7.06 / 8
any-order
P(each solution): [0.256, 0.038, 0.046, 0.059, 0.077, 0.108, 0.161, 0.256]
fork entropy at the moment it is decoded: 0.51 bits
expected distinct solutions in 16 samples: 6.09 / 8Solution 1 drops to 3.8%. Expected coverage falls by about one solution out of eight. Honestly, one seeded run of 16 samples found 7 of 8 for both decoders, so in a toy this small the gap only shows up in the exact probabilities.
What I take from it: order on its own did nothing here. The damage came from order plus sharpening. Committing to the confident tokens first, at a temperature below 1, keeps rounding toward the majority, and the fork's share of the uncertainty gets spent before the fork is reached. Real dLLMs aren't a clean joint distribution like my toy, and the paper's numbers come from real models, so take this as intuition and nothing more.
Split the claims
Is "flexible decoding" a speed claim or a reasoning claim? They need different evidence.
Look at the RL rollouts
If the recipe depends on arbitrary-order rollouts and ELBO-style likelihoods, this paper is a reason to expect weaker exploration on math and code.
Check the decoding setting behind each number
A headline accuracy at one token per step and a parallel decoding speedup can come from two different experiments, as they do here.
I'm not going to train a diffusion LLM to check this myself. But the paper and my toy point at the same place: what matters is the order the rollouts used while the model was learning, more than how it decodes afterwards.
Based in India. Works with teams in any time zone.
Comments