Follow Aqk[5,2] in this 16×16 tile as we switch between first-row and midpoint references.
Write at 2, decay, read at 5
The green path shows how token 2's contribution decays before query 5 reads it.
Split the decay for GEMM
Q̃5,d = q5,d 2G5,d − rd, K̃2,d = k2,d 2rd − G2,d
Aqk = mask(Q̃ K̃⊤)
Split the decay between query and key. Share one reference per channel across the block to control the factors' scale.
G₈ cancels exactly, but looks ahead of row 5
G₀ cancels exactly and never looks ahead
G8 − G2 = δ₃ + δ₄ + δ₅ + δ₆ + δ₇ + δ₈ ≤ 0
sum = δ₃ + δ₄ + δ₅ − δ₆ − δ₇ − δ₈ + δ₆ + δ₇ + δ₈ = G5 − G2 ✓
G0 − G2 = −(δ₁ + δ₂) ≥ 0
sum = δ₃ + δ₄ + δ₅ + δ₁ + δ₂ − δ₁ − δ₂ = G5 − G2 ✓
Reusing row 8 shortens the longest decay span versus row 0, but for rows 0–7, the reference is influenced by decay factors from future tokens.
Row 0 sits at or before every query in the block, so no future gate reaches the reference. The cost is range: row 15's factors span all 15 decay steps.
Does Aqk[5,2] change with r = G₈?
Does Aqk[5,2] change with r = G₀?
Tokens 0–5 stay fixed. Set the decay strength at tokens 6, 7, and 8. no change · sum changes · stored BF16 changes
Calculation details
| path for Aqk[5,2] | value | Δ vs start |
|---|
| r = G₈r = G₀, channel d | Q̃5,d | K̃2,d | Q̃·K̃ | Δ vs start |
|---|
Gate conversion δt,d = ht·wd·log₂e,
This browser calculation uses four channels and emulates BF16 round-to-nearest-even for the operands and final storage. The products and sum are exactly representable in FP32.