Skip to content

Commit 79da7db

Browse files
starpitclaude
andcommitted
metal: bring v2 f16 attention to MLX-faithful exp/sentinel pattern
Per `feedback_ferrite_metal_mlx_only.md` rule 7, both v2 attention variants must match MLX `sdpa_vector` bit-exact: - `exp` → `metal::fast::exp` - `-INFINITY` → `Limits<U>::finite_min` (= `-FLT_MAX` for f32 accum) The bf16 sibling (`attention_via_cache_v2_bf16_specialized`) was already updated; the f16 variant lagged with bare `exp` + `-INFINITY` at three sites (online softmax + per-simdgroup combine). Drift on the f16 path is small in steady-state usage but compounds across long sequences and degrades agreement with the bf16 sibling at mixed-dtype model swaps. Patching the f16 kernel restores the contract spelled out in the kernel header comment ("port of MLX sdpa_vector"). Goldens unchanged: 68/68 ferrite-forward metal lib tests pass, including the f16 numerical-correctness check `attention_via_cache_matches_cpu_golden`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent cc42495 commit 79da7db

1 file changed

Lines changed: 10 additions & 5 deletions

File tree

vllm-rs/crates/ferrite-metal-kernels/shaders/attention.metal

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -148,12 +148,16 @@ kernel void attention_via_cache_v2_f16_specialized(
148148
device half* o_row = output + (seq_idx * num_q + q_head_idx) * head_dim;
149149
device const uint* row_block_table = block_table + seq_idx * max_blocks;
150150

151+
// Pre-multiply Q by scale (MLX `sdpa_vector`: `q[i] = scale * queries[i]`).
151152
for (uint i = 0; i < qk_per_thread; ++i) {
152153
q_reg[i] = U(scale) * U(q_row[simd_lid * qk_per_thread + i]);
153154
o_reg[i] = 0;
154155
}
155156

156-
U max_score = -INFINITY;
157+
// Initialize per-thread max with finite minimum (MLX uses
158+
// `Limits<U>::finite_min`; -FLT_MAX is the f32 equivalent).
159+
// fast::exp doesn't handle -INFINITY safely so we avoid it.
160+
U max_score = -FLT_MAX;
157161
U sum_exp_score = 0;
158162

159163
// For each key, simdgroup `simd_gid` handles tokens at indices
@@ -185,10 +189,11 @@ kernel void attention_via_cache_v2_f16_specialized(
185189
}
186190
score = simd_sum(score);
187191

188-
// Online softmax update.
192+
// Online softmax update. Match MLX `sdpa_vector`: fast::exp
193+
// for both factor + exp_score.
189194
U new_max = max(max_score, score);
190-
U factor = exp(max_score - new_max);
191-
U exp_score = exp(score - new_max);
195+
U factor = metal::fast::exp(max_score - new_max);
196+
U exp_score = metal::fast::exp(score - new_max);
192197

193198
max_score = new_max;
194199
sum_exp_score = sum_exp_score * factor + exp_score;
@@ -214,7 +219,7 @@ kernel void attention_via_cache_v2_f16_specialized(
214219
// and (factor-rescaled) global sum_exp.
215220
U other_max = tg_max[simd_lid];
216221
U global_max = simd_max(other_max);
217-
U factor = exp(other_max - global_max);
222+
U factor = metal::fast::exp(other_max - global_max);
218223
U global_sum = simd_sum(tg_sum[simd_lid] * factor);
219224

220225
// Combine output partials. Each simdgroup wrote o_reg[j] for

0 commit comments

Comments
 (0)