[Klaud-Cold]

Goal: Update vLLM image from vllm/vllm-openai-rocm:v0.32.0@sha256:14f6a575e1fb0a580b0f633c09e06f648709d60405b85f6a7ff3f80d49f39905 to vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db.
Baseline: 2026-10-23 · vllm/vllm-openai-rocm:v0.32.0@sha256:14f6a575e1fb0a580b0f633c09e06f648709d60405b85f6a7ff3f80d49f39905
AgentX · TP8/EP1 · Mean latency · Sources: API 1, API 2, API 3, API 4

Concurrency	Total tok/s/GPU ↑	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
2	1,013.22	7.91	1,092.1	18.99
4	1,174.37	9.29	1,072.09	31.02
6	1,610.25	12.87	1,114.66	38.58
8	2,441.54	17.05	1,015.62	44.27
10	2,655.76	20.11	3,305.7	56.45
16	2,063.16	15.79	53,493.41	73.81

Eval	Score ↑	Samples
gsm8k/em_strict · c16	97.04%	1,319
minimax_m3_full/em_strict · c2	99%	102
minimax_m3_full/em_strict · c4	99%	102
minimax_m3_full/em_strict · c6	99%	102
minimax_m3_full/em_strict · c8	99%	102
minimax_m3_full/em_strict · c10	99%	102
minimax_m3_full/em_strict · c16	99%	102

Stop: Add the klaud-abort label at any time to stop this candidate now; Klaud cancels its runs, closes this PR and does not retry it. Remove the label to let the planner pick it up again.

中文

**目标：**将 vLLM 镜像从 vllm/vllm-openai-rocm:v0.32.0@sha256:14f6a575e1fb0a580b0f633c09e06f648709d60405b85f6a7ff3f80d49f39905 更新至 vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db。
**基线：**2026-10-23 · vllm/vllm-openai-rocm:v0.32.0@sha256:14f6a575e1fb0a580b0f633c09e06f648709d60405b85f6a7ff3f80d49f39905
AgentX · TP8/EP1 · 平均延迟 · 来源： API 1, API 2, API 3, API 4；数值及异常说明见上表。
**停止：**随时添加 klaud-abort 标签即可立即停止该候选：Klaud 会取消其运行、关闭此 PR 且不再重试。移除该标签后，规划器会在下次扫描时重新考虑它。

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[Klaud-Cold]

Initial attempt · Failed · Run 40071590817 / attempt 1 · 2026-10-24 07:37 UTC
vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db · 601e531e49a8 · AgentX · TP8/EP1 · Mean latency
Change: Measure branch SHA 8ce181d with vLLM main nightly c50aca14 (2026-10-24) in its -rocm72 variant pinned by digest, keeping the v0.32.0 ROCm 7.2.3, torch 733fca1 and Triton 669b31a stack (AITER v0.1.24.post1 to v0.1.26) because upstream made ROCm 10 the default after v0.32.0 (vllm#58761) and the LMCache v0.5.3 ROCm wheel targets ROCm 7.2; the v0.32.0...c50aca14 source diff keeps every recipe flag, env var and the unchanged minimax_m3 tool/reasoning parsers, MiniMax-M3 specialized decode stays opt-in (--monokernel, gfx950 only), and TRITON_ATTN now enables FP8 query quantization on gfx942 with the FP8 KV cache, which the evals cover.

Concurrency	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
2	N/A	N/A	N/A
4	N/A	N/A	N/A
6	N/A	N/A	N/A
8	N/A	N/A	N/A
10	N/A	N/A	N/A
16	N/A	N/A	N/A

Note: c2, c4, c6, c8, c10: cancelled.
Note: All rows: request errors unavailable.
Note: All rows: Δ N/A: point unavailable.
Note: c16: failed.

Eval	Score ↑	Samples
minimax_m3_full/em_strict · c2	99% (+0.00 pp)	102 each
minimax_m3_full/em_strict · c4	99% (+0.00 pp)	102 each
minimax_m3_full/em_strict · c6	98% (-1.00 pp)	102 each
minimax_m3_full/em_strict · c10	99% (+0.00 pp)	102 each

Next: Rerun the failed and cancelled jobs on the same head as infrastructure retry 1: the c16 LMCache benchmark hung at startup when TP rank 2 never published its AITER custom all-reduce IPC handles after CUDA graph capture (DistStoreError after 600000 ms), and its gsm8k eval died when LMCache retrievals stalled for 14 minutes behind two serialized store workers, LMCache went unhealthy and the vLLM scheduler asserted in _update_from_kv_xfer_finished, the same LMCache failure that broke baseline run 39808205896 attempt 1 on v0.32.0 before its rerun passed.

中文

初次尝试 · 失败 · Run 40071590817 / attempt 1 · 2026-10-24 07:37 UTC
vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db · 601e531e49a8 · AgentX · TP8/EP1 · 平均延迟
**变更：**在分支 SHA 8ce181d 上测量按摘要固定的 vLLM main nightly c50aca14（2026-10-24）的 -rocm72 版本，保留 v0.32.0 的 ROCm 7.2.3、torch 733fca1 和 Triton 669b31a 栈（AITER 由 v0.1.24.post1 升至 v0.1.26），因为上游在 v0.32.0 之后将 ROCm 10 设为默认（vllm#58761），而 LMCache v0.5.3 ROCm wheel 面向 ROCm 7.2；v0.32.0...c50aca14 源码差异保留所有配方参数、环境变量以及未改动的 minimax_m3 工具/推理解析器，MiniMax-M3 专用解码仍需显式开启（--monokernel，仅 gfx950），TRITON_ATTN 现在在 gfx942 上配合 FP8 KV cache 启用 FP8 query 量化，由评测覆盖。实测数值及异常说明见上表。
**下一步：**作为第 1 次基础设施重试，在同一 head 上重跑失败和被取消的作业：c16 LMCache 基准在启动时挂起，原因是 CUDA graph 捕获后 TP rank 2 未发布 AITER custom all-reduce IPC 句柄（600000 ms 后 DistStoreError）；其 gsm8k 评测在 LMCache 检索因两个串行存储 worker 积压而停滞 14 分钟、LMCache 变为不健康、vLLM 调度器在 _update_from_kv_xfer_finished 断言失败后中止，这与 v0.32.0 基线运行 39808205896 第 1 次尝试中出现的 LMCache 故障相同，该基线重跑后通过。

[Klaud-Cold]

Infrastructure retry 1/2 · Passed · Run 40071590817 / attempt 2 · 2026-10-24 09:28 UTC
vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db · 601e531e49a8 · AgentX · TP8/EP1 · Mean latency
Change: Rerun the failed and fail-fast-cancelled jobs of run 40071590817 on the same head and image after an intermittent c16 LMCache startup hang and LMCache stall, keeping the four passing evals from attempt 1.

Concurrency	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
2	16.15 (+104.2%)	937.88 (-14.1%)	6.28 (-66.9%)
4	20.94 (+125.4%)	985.93 (-8%)	7.84 (-74.7%)
6	27.58 (+114.2%)	872.64 (-21.7%)	8.82 (-77.1%)
8	36.28 (+112.8%)	1,196.19 (+17.8%)	12.95 (-70.7%)
10	39.14 (+94.6%)	2,197.5 (-33.5%)	17.75 (-68.6%)
16	42.52 (+169.2%)	8,916.04 (-83.3%)	26.27 (-64.4%)

Eval	Score ↑	Samples
minimax_m3_full/em_strict · c2	99% (+0.00 pp)	102 each
minimax_m3_full/em_strict · c4	99% (+0.00 pp)	102 each
minimax_m3_full/em_strict · c6	98% (-1.00 pp)	102 each
minimax_m3_full/em_strict · c8	99% (+0.00 pp)	102 each
minimax_m3_full/em_strict · c10	99% (+0.00 pp)	102 each
gsm8k/em_strict · c16	97.35% (+0.30 pp)	1,319 each
minimax_m3_full/em_strict · c16	99% (+0.00 pp)	102 each

Next: Append the perf-changelog entry, check the final matrix and start the final full sweep with full-sweep-fail-fast.

中文

基础设施重试 1/2 · 已通过 · Run 40071590817 / attempt 2 · 2026-10-24 09:28 UTC
vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db · 601e531e49a8 · AgentX · TP8/EP1 · 平均延迟
**变更：**在相同 head 和镜像上重跑运行 40071590817 中失败及被 fail-fast 取消的作业，原因是 c16 LMCache 间歇性启动挂起与 LMCache 停滞，保留第 1 次尝试中通过的 4 个评测。实测数值及异常说明见上表。
**下一步：**追加 perf-changelog 条目，检查最终矩阵，并以 full-sweep-fail-fast 启动最终全量扫描。

[Klaud-Cold]

Final full sweep · Running · Run 40082452212 / attempt 1 · 2026-10-24 09:34 UTC
vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db · 5fe560422fad · AgentX · TP8/EP1 · Mean latency
Change: Run the final full-sweep-fail-fast sweep of the complete family at head 5fe5604 after appending the perf-changelog entry.

Concurrency	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
2	N/A	N/A	N/A
4	N/A	N/A	N/A
6	N/A	N/A	N/A
8	N/A	N/A	N/A
10	N/A	N/A	N/A
16	N/A	N/A	N/A

Note: All rows: pending.
Note: All rows: request errors unavailable.
Note: All rows: Δ N/A: point unavailable.

Next: Wait for run-sweep run 40082452212 to finish all six benchmarks and seven evals.

中文

最终完整 sweep · 运行中 · Run 40082452212 / attempt 1 · 2026-10-24 09:34 UTC
vllm/vllm-openai-rocm:nightly-rocm72-c50aca14416ef29f6ed0ed256302f30646726dcd@sha256:ebda27e713fcd6146dbb55fd1a5efa29def2424d59f60b8153819485319c63db · 5fe560422fad · AgentX · TP8/EP1 · 平均延迟
**变更：**追加 perf-changelog 条目后，在 head 5fe5604 上对完整配置族运行最终 full-sweep-fail-fast 扫描。实测数值及异常说明见上表。
**下一步：**等待 run-sweep 运行 40082452212 完成全部 6 个基准和 7 个评测。