ByteDance finds why DeepSeek is sometimes strong, sometimes weak
字节找到了DeepSeek时强时弱的原因
C · Weak signal QbitAI news write-up of a ByteDance Seed paper; it reports the paper's numbers accurately but adds no data or analysis of its own, so the signal lies in the underlying paper.
Summary based on the full text·Original: about 8 minutes to read
In short
A QbitAI article (9 October 2026) summarising a ByteDance Seed paper (arXiv 2609.36322) on why DeepSeek-V4 models answer the same question correctly or wrongly depending on a few irrelevant tokens. The cause is chunked KV cache compression: performance cycles with the compression stride, every 4 tokens in V4 and every 2 in V4.1. The researchers call this "phase sensitivity". Post-training and newer versions shrink the effect but do not remove it.
Key numbers
- 40.2 pplargest retrieval-accuracy gap across positions for DeepSeek-V4-Flash-Base in a 128K-token test
- 6.1 ppremaining gap for the newer DeepSeek-V4.1-Flash-0910
- 71.3% vs 91.5%average probability of the wrong answer at some padding lengths versus the right answer at others, same code task
As reported by the source.
Chinese·Web page·opens qbitai.com
What it reports
- Code test on DeepSeek-V4-Flash-Base: completing an FP8 function where the answer is 8, the model sometimes said 32. Only the number of padding "=" signs before the code changed.
- Over 16 padding lengths: with length mod 4 equal to 0 or 1 the model leaned to 32; with 2 or 3, to 8. Average probabilities: 71.3% for 32 and 26.4% for 8 in one group; 91.5% for 8 and 7.2% for 32 in the other.
- Needle-in-a-haystack at 128K tokens with about 16,000 key-value pairs: maximum accuracy gaps of 40.2 pp (V4-Flash-Base) and 34.8 pp (V4-Pro-Base).
- After post-training: 19.1 pp (V4-Flash-0731), 14.8 pp (V4-Pro-0813); 6.1 pp for V4.1-Flash-0910.
- Models trained from scratch on a Qwen3-0.6B architecture showed periods matching stride 4, 6 and 8; a full-attention baseline did not. The effect persisted without RoPE and with simple averaging in place of learned compression.
- Different attention heads specialise in different phases ("phase specialization").
- The authors recommend testing the same fact at different phases, since average benchmark scores hide the dips.
How they got it
The article relays the paper: a controlled code-completion probe, a long-context retrieval test that shifts the target relative to compression window boundaries, small models trained with varied compression settings, attention-head ablations and a simplified theoretical model of training.
Limits
- Secondary reporting; numbers were not checked against the paper here.
- Tests are synthetic retrieval and completion tasks; impact on real documents is not measured.
Why it matters for companies
Models with compressed long context can miss a fact because of where it sits in the input, not because the fact is hard. For companies feeding long contracts, policies or case files to such models, an average accuracy score is not enough. Test the same question with the key passage moved a few characters, and treat long-context answers on critical facts as needing a check.
Key passages in translation
Chinese original
"同一道题，什么都没改，只在前面多塞几个无关紧要的字符，模型突然就不会了？？？"
"The same question, nothing changed, just a few irrelevant characters stuffed in front, and suddenly the model can't do it???"
Chinese original
"在128K长上下文检索测试中，研究人员发现，同一条信息只是换了个位置，DeepSeek-V4系列模型的检索准确率最高就能拉开40.2个百分点。"
"In a 128K long-context retrieval test, researchers found that just moving the same piece of information changed the DeepSeek-V4 series' retrieval accuracy by as much as 40.2 percentage points."
Chinese original
"研究人员发现，模型对不同相位的信息，检索能力存在系统性差异。"
"The researchers found systematic differences in the model's ability to retrieve information at different phases."
Chinese original
"分块压缩这种设计本身，就可能引入周期性的检索弱点。"
"The design of chunked compression itself can introduce periodic retrieval weak spots."
Chinese original
"评估采用分块KV Cache压缩的模型时，不能只看整体检索准确率，还得把同一条信息放到不同压缩相位上，分别测一遍。"
"When evaluating models that use chunked KV cache compression, you cannot look only at overall retrieval accuracy; you also have to place the same information at different compression phases and test each one."
Summary based on the full text. Read the original for the full argument and data.Read the original on qbitai.com