040a505ff5 · tduncan/Nvidia-Optimization-Stack-VLLM

@@ -15,7 +15,6 @@

15 # command(str): the single command to run for tests. incompatible with commands.

16 # commands(list): the list of commands to run for the test. incompatible with command.

17 # mirror_hardwares(list): the list of hardware to run the test on as well. currently only supports [amdexperimental]

18 -# gpu(str): override the GPU selection for the test. default is L4 GPUs. supports a100, b200, h200

18 # num_gpus(int): override the number of GPUs for the test. defaults to 1 GPU. currently supports 2,4.

19 # num_nodes(int): whether to simulate multi-node setup by launching multiple containers on one host,

20 # in this case, commands must be specified. the first command runs on the first host, the second

@@ -32,6 +31,80 @@

31 # - If the test takes more than 10min, then it is okay to create a new step.

32 # Note that all steps execute in parallel.

33

34 +

35 +#####################################################################################################################################

36 +# #

37 +# README #

38 +# #

39 +#####################################################################################################################################

40 +# #

41 +# IMPORTANT: #

42 +# * Currently AMD CI has MI300 agents, MI325 agents, and MI355 agents. Of those, AMD is using mostly MI325 and MI355. AMD team #

43 +# is actively working on enabling more MI300 machines. All upcoming feature improvements are tracked in: #

44 +# https://github.com/vllm-project/vllm/issues/34994 #

45 +# #

46 +#-----------------------------------------------------------------------------------------------------------------------------------#

47 +# #

48 +# NOTES: #

49 +# * [Pytorch Nightly Dependency Override Check]: if this test fails, it means the nightly torch version is not compatible with #

50 +# some of the dependencies. Please check the error message and add the package to #

51 +# whitelist in `/vllm/tools/pre_commit/generate_nightly_torch_test.py`. #

52 +# * [Entrypoints Integration Test (LLM)]: #

53 +# - {`pytest -v -s entrypoints/llm/test_generate.py`}: It needs a clean process #

54 +# - {`pytest -v -s entrypoints/offline_mode`}: Needs to avoid interference with other tests #

55 +# * [V1 Test e2e + engine]: The test uses 4 GPUs, but we schedule it on 8-GPU machines for stability. See discussion here: #

56 +# https://github.com/vllm-project/vllm/pull/31040 #

57 +# * [V1 others]: #

58 +# - Split the tests to avoid interference #

59 +# - Integration test for streaming correctness (requires special branch for __harness__ lib). #

60 +# * [V1 others (CPU)]: Split the tests to avoid interference #

61 +# * [PyTorch Compilation Unit Tests]: Run unit tests defined directly under `compile/`, not including subdirectories, which #

62 +# are usually heavier tests covered elsewhere. Use `find` to launch multiple instances #

63 +# of pytest so that they do not suffer from: #

64 +# https://github.com/vllm-project/vllm/issues/28965 #

65 +# * [PyTorch Fullgraph Smoke Test]: Run smoke tests under fullgraph directory, except `test_full_graph.py` as it is a heavy #

66 +# test that is covered in other steps. Use `find` to launch multiple instances of pytest #

67 +# so that they do not suffer from: https://github.com/vllm-project/vllm/issues/28965 #

68 +# * [PyTorch Fullgraph]: #

69 +# - Limit to no custom ops to reduce running time. Wrap with quotes to escape yaml and avoid starting `-k` string #

70 +# with a `-` #

71 +# - Old E2E tests such as: #

72 +# ```bash #

73 +# pytest -v -s compile/distributed/test_fusions_e2e.py -k 'TRITON and not +quant_fp8 and not Llama-4' #

74 +# ``` #

75 +# were removed in https://github.com/vllm-project/vllm/pull/33293 in favor of new tests in `fusions_e2e`. We #

76 +# avoid replicating the new jobs in this file as it's deprecated. #

77 +# * [Basic Models Tests (Extra Initialization) %N]: Only when vLLM model source is modified - test initialization of a #

78 +# large subset of supported models (the complement of the small subset in #

79 +# the above test.) Also run if model initialization test file is modified. #

80 +# * [Language Models Tests (Extra Standard) %N]: Shard slow subset of standard language models tests. Only run when model #

81 +# source is modified, or when specified test files are modified. #

82 +# * [Language Models Tests (Hybrid) %N]: Install fast path packages for testing against transformers (mamba, conv1d) and to #

83 +# run plamo2 model in vLLM. #

84 +# * [Language Models Test (Extended Generation)]: Install fast path packages for testing against transformers (mamba, conv1d) #

85 +# and to run plamo2 model in vLLM. #

86 +# * [Multi-Modal Models (Standard)]: #

87 +# - Do NOT remove `VLLM_WORKER_MULTIPROC_METHOD=spawn` setting as ROCm requires this for certain models to function. #

88 +# * [Transformers Nightly Models Test]: Whisper needs `VLLM_WORKER_MULTIPROC_METHOD=spawn` to avoid deadlock. #

89 +# * [Plugin Tests (2 GPUs)]: #

90 +# - {`pytest -v -s entrypoints/openai/test_oot_registration.py`}: It needs a clean process #

91 +# - {`pytest -v -s models/test_oot_registration.py`}: It needs a clean process #

92 +# - {`pytest -v -s plugins/lora_resolvers`}: Unit tests for in-tree lora resolver plugins #

93 +# * [LoRA TP (Distributed)]: #

94 +# - There is some Tensor Parallelism related processing logic in LoRA that requires multi-GPU testing for validation. #

95 +# - {`pytest -v -s -x lora/test_gptoss_tp.py`}: Disabled for now because MXFP4 backend on non-cuda platform doesn't support #

96 +# LoRA yet. #

97 +# * [Distributed Tests (GPU_TAG)]: Don't test llama model here, it seems hf implementation is buggy. See: #

98 +# https://github.com/vllm-project/vllm/pull/5689 #

99 +# * [Distributed Tests (GPU_TAG)]: Some old E2E tests were removed in https://github.com/vllm-project/vllm/pull/33293 in #

100 +# favor of new tests in fusions_e2e. We avoid replicating the new jobs in #

101 +# this file as it's deprecated. #

102 +# #

103 +#####################################################################################################################################

104 +

105 +

106 +

107 +

108 steps:

109

110

@@ -41,18 +114,25 @@ steps:

114 # #

115 #####################################################################################################################################

116

44 -- label: Pytorch Nightly Dependency Override Check # 2min

45 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

117 +- label: Pytorch Nightly Dependency Override Check # TBD

118 + timeout_in_minutes: 180

119 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

120 agent_pool: mi250_1

121 + optional: true

122 + soft_fail: true

123 + working_dir: "/vllm-workspace/tests"

124 source_file_dependencies:

125 - requirements/nightly_torch_test.txt

126 + - vllm/platforms/rocm.py

127 commands:

128 - bash standalone_tests/pytorch_nightly_dependency.sh

129

52 -- label: Async Engine, Inputs, Utils, Worker Test # 10min

53 - timeout_in_minutes: 15

54 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

130 +

131 +- label: Async Engine, Inputs, Utils, Worker # TBD

132 + timeout_in_minutes: 180

133 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

134 agent_pool: mi250_1

135 + working_dir: "/vllm-workspace/tests"

136 source_file_dependencies:

137 - vllm/

138 - tests/detokenizer

@@ -63,15 +143,20 @@ steps:

143 - pytest -v -s -m 'not cpu_test' multimodal

144 - pytest -v -s utils_

145

66 -- label: Async Engine, Inputs, Utils, Worker, Config Test (CPU) # 20min

67 - timeout_in_minutes: 30

68 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

146 +

147 +- label: Async Engine, Inputs, Utils, Worker, Config (CPU) # TBD

148 + timeout_in_minutes: 180

149 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

150 agent_pool: mi250_1

151 + optional: true

152 + no_gpu: true

153 + working_dir: "/vllm-workspace/tests"

154 source_file_dependencies:

155 - vllm/

156 - tests/test_inputs.py

157 - tests/test_outputs.py

158 - tests/test_pooling_params.py

159 + - tests/test_ray_env.py

160 - tests/multimodal

161 - tests/renderers

162 - tests/standalone_tests/lazy_imports.py

@@ -79,12 +164,12 @@ steps:

164 - tests/tool_parsers

165 - tests/transformers_utils

166 - tests/config

82 - no_gpu: true

167 commands:

168 - python3 standalone_tests/lazy_imports.py

169 - pytest -v -s test_inputs.py

170 - pytest -v -s test_outputs.py

171 - pytest -v -s test_pooling_params.py

172 + - pytest -v -s test_ray_env.py

173 - pytest -v -s -m 'cpu_test' multimodal

174 - pytest -v -s renderers

175 - pytest -v -s tokenizers_

@@ -92,22 +177,28 @@ steps:

177 - pytest -v -s transformers_utils

178 - pytest -v -s config

179

95 -- label: Python-only Installation Test # 10min

96 - timeout_in_minutes: 20

97 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

180 +

181 +- label: Python-only Installation # TBD

182 + timeout_in_minutes: 180

183 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

184 agent_pool: mi250_1

185 + optional: true

186 + working_dir: "/vllm-workspace/tests"

187 source_file_dependencies:

188 - tests/standalone_tests/python_only_compile.sh

189 - setup.py

190 + - vllm/platforms/rocm.py

191 commands:

192 - bash standalone_tests/python_only_compile.sh

193

105 -- label: Basic Correctness Test # 20min

106 - timeout_in_minutes: 30

107 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

194 +

195 +- label: Basic Correctness # TBD

196 + timeout_in_minutes: 180

197 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

198 agent_pool: mi250_1

199 fast_check: true

200 torch_nightly: true

201 + working_dir: "/vllm-workspace/tests"

202 source_file_dependencies:

203 - vllm/

204 - tests/basic_correctness/test_basic_correctness

@@ -119,22 +210,25 @@ steps:

210 - pytest -v -s basic_correctness/test_basic_correctness.py

211 - pytest -v -s basic_correctness/test_cpu_offload.py

212

122 -- label: Entrypoints Unit Tests # 5min

123 - timeout_in_minutes: 10

124 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

213 +

214 +- label: Entrypoints Unit Tests # TBD

215 + timeout_in_minutes: 180

216 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

217 agent_pool: mi250_1

126 - working_dir: "/vllm-workspace/tests"

218 fast_check: true

219 + working_dir: "/vllm-workspace/tests"

220 source_file_dependencies:

221 - vllm/entrypoints

222 - tests/entrypoints/

223 + - vllm/platforms/rocm.py

224 commands:

225 - pytest -v -s entrypoints/openai/tool_parsers

133 - - pytest -v -s entrypoints/ --ignore=entrypoints/llm --ignore=entrypoints/openai --ignore=entrypoints/rpc --ignore=entrypoints/instrumentator --ignore=entrypoints/offline_mode --ignore=entrypoints/test_chat_utils.py --ignore=entrypoints/pooling

226 + - pytest -v -s entrypoints/ --ignore=entrypoints/llm --ignore=entrypoints/rpc --ignore=entrypoints/sleep --ignore=entrypoints/instrumentator --ignore=entrypoints/openai --ignore=entrypoints/offline_mode --ignore=entrypoints/test_chat_utils.py --ignore=entrypoints/pooling

227

135 -- label: Entrypoints Integration Test (LLM) # 30min

136 - timeout_in_minutes: 40

137 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

228 +

229 +- label: Entrypoints Integration (LLM) # TBD

230 + timeout_in_minutes: 180

231 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

232 agent_pool: mi250_1

233 fast_check: true

234 torch_nightly: true

@@ -149,30 +243,14 @@ steps:

243 - pytest -v -s entrypoints/llm/test_generate.py

244 - pytest -v -s entrypoints/offline_mode

245

152 -- label: Entrypoints Integration Test (API Server 1) # 100min

153 - timeout_in_minutes: 130

154 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

155 - agent_pool: mi250_1

156 - working_dir: "/vllm-workspace/tests"

157 - fast_check: true

158 - torch_nightly: true

159 - source_file_dependencies:

160 - - vllm/

161 - - tests/entrypoints/openai

162 - - tests/entrypoints/test_chat_utils

163 - commands:

164 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

165 - - pytest -v -s entrypoints/openai --ignore=entrypoints/openai/chat_completion/test_chat_with_tool_reasoning.py --ignore=entrypoints/openai/chat_completion/test_oot_registration.py --ignore=entrypoints/openai/completion/test_tensorizer_entrypoint.py --ignore=entrypoints/openai/correctness/ --ignore=entrypoints/openai/tool_parsers/ --ignore=entrypoints/openai/responses

166 - - pytest -v -s entrypoints/test_chat_utils.py

246

168 -- label: Entrypoints Integration Test (API Server 2)

169 - timeout_in_minutes: 50

170 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

247 +- label: Entrypoints Integration (API Server 2) # TBD

248 + timeout_in_minutes: 180

249 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

250 agent_pool: mi250_1

172 - optional: true

173 - working_dir: "/vllm-workspace/tests"

251 fast_check: true

252 torch_nightly: true

253 + working_dir: "/vllm-workspace/tests"

254 source_file_dependencies:

255 - vllm/

256 - tests/entrypoints/rpc

@@ -184,29 +262,14 @@ steps:

262 - PYTHONPATH=/vllm-workspace pytest -v -s entrypoints/rpc

263 - pytest -v -s tool_use

264

187 -- label: Entrypoints Integration Test (Pooling)

188 - timeout_in_minutes: 50

189 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

190 - agent_pool: mi250_1

191 - optional: true

192 - fast_check: true

193 - torch_nightly: true

194 - working_dir: "/vllm-workspace/tests"

195 - source_file_dependencies:

196 - - vllm/

197 - - tests/entrypoints/pooling

198 - commands:

199 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

200 - - pytest -v -s entrypoints/pooling

265

202 -- label: Entrypoints Integration Test (Responses API)

203 - timeout_in_minutes: 50

204 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

266 +- label: Entrypoints Integration (Responses API) # TBD

267 + timeout_in_minutes: 180

268 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

269 agent_pool: mi250_1

206 - optional: true

207 - working_dir: "/vllm-workspace/tests"

270 fast_check: true

271 torch_nightly: true

272 + working_dir: "/vllm-workspace/tests"

273 source_file_dependencies:

274 - vllm/

275 - tests/entrypoints/openai/responses

@@ -214,103 +277,59 @@ steps:

277 - export VLLM_WORKER_MULTIPROC_METHOD=spawn

278 - pytest -v -s entrypoints/openai/responses

279

217 -- label: Distributed Tests (4 GPUs) # 35min

218 - timeout_in_minutes: 50

219 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

220 - agent_pool: mi250_4

221 - optional: true

222 - num_gpus: 4

223 - working_dir: "/vllm-workspace/tests"

224 - source_file_dependencies:

225 - - vllm/distributed/

226 - - tests/distributed/test_utils

227 - - tests/distributed/test_pynccl

228 - - tests/distributed/test_events

229 - - tests/compile/fullgraph/test_basic_correctness.py

230 - - examples/offline_inference/rlhf.py

231 - - examples/offline_inference/rlhf_colocate.py

232 - - examples/offline_inference/new_weight_syncing/

233 - - tests/examples/offline_inference/data_parallel.py

234 - - tests/v1/distributed

235 - - tests/v1/engine/test_engine_core_client.py

236 - - tests/distributed/test_symm_mem_allreduce.py

237 - commands:

238 - - export TORCH_NCCL_BLOCKING_WAIT=1

239 - - torchrun --nproc-per-node=4 distributed/test_torchrun_example.py

240 - - PP_SIZE=2 torchrun --nproc-per-node=4 distributed/test_torchrun_example.py

241 - - TP_SIZE=4 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

242 - - PP_SIZE=2 TP_SIZE=2 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

243 - - DP_SIZE=4 ENABLE_EP=1 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

244 - - TP_SIZE=2 DP_SIZE=2 ENABLE_EP=1 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

245 - - python3 ../examples/offline_inference/data_parallel.py --enforce-eager

246 - - TP_SIZE=2 DP_SIZE=2 pytest -v -s v1/distributed/test_async_llm_dp.py

247 - - TP_SIZE=2 DP_SIZE=2 pytest -v -s v1/distributed/test_eagle_dp.py

248 - - TP_SIZE=2 DP_SIZE=2 pytest -v -s v1/distributed/test_external_lb_dp.py

249 - - TP_SIZE=1 DP_SIZE=4 pytest -v -s v1/distributed/test_internal_lb_dp.py

250 - - TP_SIZE=1 DP_SIZE=4 pytest -v -s v1/distributed/test_hybrid_lb_dp.py

251 - - pytest -v -s v1/engine/test_engine_core_client.py::test_kv_cache_events_dp

252 - - pytest -v -s distributed/test_utils.py

253 - - pytest -v -s compile/fullgraph/test_basic_correctness.py

254 - - pytest -v -s distributed/test_pynccl.py

255 - - pytest -v -s distributed/test_events.py

256 - - pytest -v -s distributed/test_symm_mem_allreduce.py

257 - - pushd ../examples/offline_inference

258 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf.py

259 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 RAY_DEDUP_LOGS=0 python3 rlhf_colocate.py

260 - - popd

261 - - pushd ../examples/offline_inference/new_weight_syncing

262 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf_nccl.py

263 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf_ipc.py

264 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf_async_new_apis.py

265 - - popd

266 -

267 -- label: Distributed Tests (8 GPUs) # 4min

268 - timeout_in_minutes: 10

269 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

270 - agent_pool: mi250_8

271 - optional: true

272 - num_gpus: 8

273 - working_dir: "/vllm-workspace/tests"

274 - source_file_dependencies:

275 - - examples/offline_inference/torchrun_dp_example.py

276 - - vllm/config/parallel.py

277 - - vllm/distributed/

278 - - vllm/v1/engine/llm_engine.py

279 - - vllm/v1/executor/uniproc_executor.py

280 - - vllm/v1/worker/gpu_worker.py

281 - commands:

282 - - export TORCH_NCCL_BLOCKING_WAIT=1

283 - - torchrun --nproc-per-node=8 ../examples/offline_inference/torchrun_dp_example.py --tp-size=2 --pp-size=1 --dp-size=4 --enable-ep

280

285 -- label: EPLB Algorithm Test # 5min

286 - timeout_in_minutes: 15

287 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative, amdgfx90a]

281 +- label: EPLB Algorithm # TBD

282 + timeout_in_minutes: 180

283 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

284 agent_pool: mi250_1

285 working_dir: "/vllm-workspace/tests"

286 source_file_dependencies:

287 - vllm/distributed/eplb

288 - tests/distributed/test_eplb_algo.py

289 + - vllm/platforms/rocm.py

290 commands:

291 - pytest -v -s distributed/test_eplb_algo.py

292

296 -- label: EPLB Execution Test # 10min

297 - timeout_in_minutes: 20

298 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

293 +

294 +- label: EPLB Execution # TBD

295 + timeout_in_minutes: 180

296 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

297 agent_pool: mi250_4

298 num_gpus: 4

299 working_dir: "/vllm-workspace/tests"

300 source_file_dependencies:

301 - vllm/distributed/eplb

302 - tests/distributed/test_eplb_execute.py

303 + - tests/distributed/test_eplb_spec_decode.py

304 + - vllm/platforms/rocm.py

305 commands:

306 - pytest -v -s distributed/test_eplb_execute.py

307 - pytest -v -s distributed/test_eplb_spec_decode.py

308

309 -- label: Metrics, Tracing Test # 12min

310 - timeout_in_minutes: 20

311 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

312 - agent_pool: mi250_2

309 +

310 +- label: Elastic EP Scaling Test # TBD

311 + timeout_in_minutes: 180

312 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

313 + agent_pool: mi250_4

314 + num_gpus: 4

315 + working_dir: "/vllm-workspace/tests"

316 + source_file_dependencies:

317 + - vllm/distributed/

318 + - vllm/engine/

319 + - vllm/executor/

320 + - vllm/compilation/

321 + - tests/distributed/

322 + - vllm/platforms/rocm.py

323 + commands:

324 + - pytest -v -s distributed/test_elastic_ep.py

325 +

326 +

327 +- label: Metrics, Tracing (2 GPUs) # TBD

328 + timeout_in_minutes: 180

329 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

330 + agent_pool: mi250_1

331 num_gpus: 2

332 + working_dir: "/vllm-workspace/tests"

333 source_file_dependencies:

334 - vllm/

335 - tests/v1/tracing

@@ -322,9 +341,10 @@ steps:

341 'opentelemetry-semantic-conventions-ai>=0.4.1'"

342 - pytest -v -s v1/tracing

343

325 -- label: Regression Test # 7min

326 - timeout_in_minutes: 20

327 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

344 +

345 +- label: Regression # TBD

346 + timeout_in_minutes: 180

347 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

348 agent_pool: mi250_1

349 working_dir: "/vllm-workspace/tests"

350 source_file_dependencies:

@@ -334,10 +354,13 @@ steps:

354 - pip install modelscope

355 - pytest -v -s test_regression.py

356

337 -- label: Engine Test # 9min

338 - timeout_in_minutes: 15

339 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

357 +

358 +- label: Engine # TBD

359 + timeout_in_minutes: 180

360 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

361 agent_pool: mi250_1

362 + optional: true

363 + working_dir: "/vllm-workspace/tests"

364 source_file_dependencies:

365 - vllm/

366 - tests/engine

@@ -348,730 +371,824 @@ steps:

371 commands:

372 - pytest -v -s engine test_sequence.py test_config.py test_logger.py test_vllm_port.py

373

351 -- label: V1 Test e2e + engine # 65min

352 - timeout_in_minutes: 90

353 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

374 +

375 +- label: Engine (1 GPU) # TBD

376 + timeout_in_minutes: 180

377 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

378 agent_pool: mi250_1

379 optional: true

380 + working_dir: "/vllm-workspace/tests"

381 source_file_dependencies:

357 - - vllm/

358 - - tests/v1

382 + - vllm/v1/

383 + - tests/v1/engine/

384 + - vllm/platforms/rocm.py

385 commands:

360 - - pytest -v -s v1/e2e

361 - - pytest -v -s v1/engine

386 + - pytest -v -s v1/engine/test_preprocess_error_handling.py

387 + - pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py

388

363 -- label: V1 Test e2e (2 GPUs) # 65min

364 - timeout_in_minutes: 90

365 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

366 - agent_pool: mi250_2

389 +

390 +- label: e2e Scheduling (1 GPU) # TBD

391 + timeout_in_minutes: 180

392 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

393 + agent_pool: mi250_1

394 + optional: true

395 + working_dir: "/vllm-workspace/tests"

396 + source_file_dependencies:

397 + - vllm/v1/

398 + - tests/v1/e2e/general/

399 + - vllm/platforms/rocm.py

400 + commands:

401 + - pytest -v -s v1/e2e/general/test_async_scheduling.py

402 +

403 +

404 +- label: e2e Core (1 GPU) # TBD

405 + timeout_in_minutes: 180

406 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

407 + agent_pool: mi250_1

408 optional: true

409 + working_dir: "/vllm-workspace/tests"

410 source_file_dependencies:

369 - - vllm/

370 - - tests/v1

411 + - vllm/v1/

412 + - tests/v1/e2e/general/

413 + - vllm/platforms/rocm.py

414 commands:

372 - - pytest -v -s v1/e2e/spec_decode/test_spec_decode.py -k "tensor_parallelism"

415 + - pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py

416

374 -- label: V1 Test e2e (4 GPUs) # 65min

375 - timeout_in_minutes: 90

376 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

377 - agent_pool: mi250_4

417 +

418 +- label: Spec Decode Speculators + MTP # TBD

419 + timeout_in_minutes: 180

420 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

421 + agent_pool: mi250_1

422 optional: true

423 + working_dir: "/vllm-workspace/tests"

424 source_file_dependencies:

380 - - vllm/

381 - - tests/v1

425 + - vllm/v1/spec_decode/

426 + - vllm/v1/worker/gpu/spec_decode/

427 + - vllm/model_executor/model_loader/

428 + - vllm/v1/sample/

429 + - vllm/model_executor/layers/

430 + - vllm/transformers_utils/configs/speculators/

431 + - tests/v1/e2e/spec_decode/

432 + - vllm/platforms/rocm.py

433 commands:

383 - - pytest -v -s v1/e2e/spec_decode/test_spec_decode.py -k "eagle_correctness_heavy"

434 + - pytest -v -s v1/e2e/spec_decode -k "speculators or mtp_correctness"

435 +

436 +

437 +- label: Spec Decode Ngram + Suffix # TBD

438 + timeout_in_minutes: 180

439 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

440 + agent_pool: mi250_1

441 + optional: true

442 + working_dir: "/vllm-workspace/tests"

443 + source_file_dependencies:

444 + - vllm/v1/spec_decode/

445 + - vllm/v1/worker/gpu/spec_decode/

446 + - vllm/model_executor/model_loader/

447 + - vllm/v1/sample/

448 + - vllm/model_executor/layers/

449 + - tests/v1/e2e/spec_decode/

450 + - vllm/platforms/rocm.py

451 + commands:

452 + - pytest -v -s v1/e2e/spec_decode -k "ngram or suffix"

453 +

454 +

455 +- label: Spec Decode Draft Model # TBD

456 + timeout_in_minutes: 180

457 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

458 + agent_pool: mi250_1

459 + optional: true

460 + working_dir: "/vllm-workspace/tests"

461 + source_file_dependencies:

462 + - vllm/v1/spec_decode/

463 + - vllm/v1/worker/gpu/spec_decode/

464 + - vllm/model_executor/model_loader/

465 + - vllm/v1/sample/

466 + - vllm/model_executor/layers/

467 + - tests/v1/e2e/spec_decode/

468 + - vllm/platforms/rocm.py

469 + commands:

470 + - pytest -v -s v1/e2e/spec_decode -k "draft_model or no_sync or batch_inference"

471 +

472 +

473 +- label: V1 e2e (2 GPUs) # TBD

474 + timeout_in_minutes: 180

475 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

476 + agent_pool: mi250_1

477 + optional: true

478 + working_dir: "/vllm-workspace/tests"

479 + source_file_dependencies:

480 + - vllm/

481 + - tests/v1/e2e

482 + commands:

483 + - pytest -v -s v1/e2e/spec_decode/test_spec_decode.py -k "tensor_parallelism"

484

385 -- label: V1 Test entrypoints # 35min

386 - timeout_in_minutes: 50

387 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

485 +

486 +- label: Entrypoints V1 # TBD

487 + timeout_in_minutes: 180

488 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

489 agent_pool: mi250_1

490 + optional: true

491 + working_dir: "/vllm-workspace/tests"

492 source_file_dependencies:

390 - - vllm/

391 - - tests/v1

493 + - vllm/

494 + - tests/v1

495 commands:

393 - - pytest -v -s v1/entrypoints

496 + - pytest -v -s v1/entrypoints

497

395 -- label: V1 Test others # 42min

498 +

499 +- label: V1 Sample + Logits # TBD

500 timeout_in_minutes: 60

397 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

501 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

502 agent_pool: mi250_1

503 optional: true

504 + working_dir: "/vllm-workspace/tests"

505 source_file_dependencies:

401 - - vllm/

402 - - tests/v1

403 - commands:

404 - - uv pip install --system -r /vllm-workspace/requirements/kv_connectors.txt

405 - - pytest -v -s -m 'not cpu_test' v1/core

406 - - pytest -v -s v1/executor

407 - - pytest -v -s v1/kv_offload

408 - - pytest -v -s v1/sample

409 - - pytest -v -s v1/logits_processors

410 - - pytest -v -s v1/worker

411 - - pytest -v -s v1/spec_decode

412 - - pytest -v -s -m 'not cpu_test' v1/kv_connector/unit

413 - - pytest -v -s -m 'not cpu_test' v1/metrics

414 - - pytest -v -s v1/test_oracle.py

415 - - pytest -v -s v1/test_request.py

416 - - pytest -v -s v1/test_outputs.py

417 - - pip install -U git+https://github.com/robertgshaw2-redhat/lm-evaluation-harness.git@streaming-api

418 - - pytest -v -s entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine

419 -

420 -- label: V1 Test attention (H100) # 10min

421 - timeout_in_minutes: 30

422 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

506 + - vllm/

507 + - tests/v1/sample

508 + - tests/v1/logits_processors

509 + - tests/v1/test_oracle.py

510 + - tests/v1/test_request.py

511 + - tests/v1/test_outputs.py

512 + commands:

513 + - pytest -v -s v1/sample

514 + - pytest -v -s v1/logits_processors

515 + - pytest -v -s v1/test_oracle.py

516 + - pytest -v -s v1/test_request.py

517 + - pytest -v -s v1/test_outputs.py

518 +

519 +

520 +- label: V1 Core + KV + Metrics # TBD

521 + timeout_in_minutes: 60

522 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

523 agent_pool: mi250_1

524 optional: true

525 + working_dir: "/vllm-workspace/tests"

526 source_file_dependencies:

426 - - vllm/config/attention.py

427 - - vllm/model_executor/layers/attention

428 - - vllm/v1/attention

429 - - tests/v1/attention

527 + - vllm/

528 + - tests/v1/core

529 + - tests/v1/executor

530 + - tests/v1/kv_offload

531 + - tests/v1/worker

532 + - tests/v1/kv_connector/unit

533 + - tests/v1/metrics

534 + - tests/entrypoints/openai/correctness/test_lmeval.py

535 + commands:

536 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

537 + - pytest -v -s -m 'not cpu_test' v1/core

538 + - pytest -v -s v1/executor

539 + - pytest -v -s v1/kv_offload

540 + - pytest -v -s v1/worker

541 + - pytest -v -s -m 'not cpu_test' v1/kv_connector/unit

542 + - pytest -v -s -m 'not cpu_test' v1/metrics

543 + - pip install -U git+https://github.com/robertgshaw2-redhat/lm-evaluation-harness.git@streaming-api

544 + - pytest -v -s entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine

545 +

546 +

547 +- label: V1 Speculative Decoding (slow) # TBD

548 + timeout_in_minutes: 180

549 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

550 + agent_pool: mi250_1

551 + working_dir: "/vllm-workspace/tests"

552 + source_file_dependencies:

553 + - vllm/v1/spec_decode/

554 + - vllm/model_executor/models/

555 + - vllm/v1/attention/

556 + - vllm/model_executor/layers/

557 + - tests/v1/spec_decode/

558 + - vllm/platforms/rocm.py

559 commands:

431 - - pytest -v -s v1/attention

560 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_eagle.py

561 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_extract_hidden_states.py

562 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_max_len.py

563 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_mtp.py

564 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_ngram.py

565 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_speculators_eagle3.py

566 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_tree_attention.py

567 +

568

433 -- label: Batch Invariance Tests (H100) # 10min

434 - timeout_in_minutes: 25

435 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

569 +- label: V1 attention (H100-MI250) # TBD

570 + timeout_in_minutes: 180

571 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

572 agent_pool: mi250_1

573 + working_dir: "/vllm-workspace/tests"

574 source_file_dependencies:

438 - - vllm/v1/attention

439 - - vllm/model_executor/layers

440 - - tests/v1/determinism/

575 + - vllm/config/attention.py

576 + - vllm/model_executor/layers/attention

577 + - vllm/v1/attention

578 + - tests/v1/attention

579 + - vllm/_aiter_ops.py

580 + - vllm/envs.py

581 + - vllm/platforms/rocm.py

582 commands:

442 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

443 - - pip install pytest-timeout pytest-forked

444 - - pytest -v -s v1/determinism/test_batch_invariance.py

445 - - pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py

583 + - pytest -v -s v1/attention

584

447 -- label: V1 Test others (CPU) # 5 mins

448 - timeout_in_minutes: 15

449 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

585 +

586 +- label: V1 others (CPU) # TBD

587 + timeout_in_minutes: 180

588 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

589 agent_pool: mi250_1

590 no_gpu: true

591 + optional: true

592 + working_dir: "/vllm-workspace/tests"

593 source_file_dependencies:

453 - - vllm/

454 - - tests/v1

594 + - vllm/

595 + - tests/v1

596 commands:

456 - - pytest -v -s -m 'cpu_test' v1/core

457 - - pytest -v -s v1/structured_output

458 - - pytest -v -s v1/test_serial_utils.py

459 - - pytest -v -s -m 'cpu_test' v1/kv_connector/unit

460 - - pytest -v -s -m 'cpu_test' v1/metrics

597 + - pytest -v -s -m 'cpu_test' v1/core

598 + - pytest -v -s v1/structured_output

599 + - pytest -v -s v1/test_serial_utils.py

600 + - pytest -v -s -m 'cpu_test' v1/kv_connector/unit

601 + - pytest -v -s -m 'cpu_test' v1/metrics

602

603

463 -- label: Examples Test # 30min

464 - timeout_in_minutes: 45

465 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

604 +- label: Examples # TBD

605 + timeout_in_minutes: 180

606 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

607 agent_pool: mi250_1

608 + optional: true

609 working_dir: "/vllm-workspace/examples"

610 source_file_dependencies:

611 - vllm/entrypoints

612 - vllm/multimodal

613 - examples/

614 + - vllm/platforms/rocm.py

615 commands:

616 - pip install tensorizer

474 - - python3 offline_inference/basic/chat.py

475 - - python3 offline_inference/basic/generate.py --model facebook/opt-125m

476 - - python3 offline_inference/basic/generate.py --model meta-llama/Llama-2-13b-chat-hf --cpu-offload-gb 10

477 - - python3 offline_inference/basic/classify.py

478 - - python3 offline_inference/basic/embed.py

479 - - python3 offline_inference/basic/score.py

617 + # Basic

618 + - python3 basic/offline_inference/chat.py --attention-backend TRITON_ATTN

619 + - python3 basic/offline_inference/generate.py --model facebook/opt-125m

620 + - python3 basic/offline_inference/generate.py --model meta-llama/Llama-2-13b-chat-hf --cpu-offload-gb 10

621 + - python3 basic/offline_inference/classify.py

622 + - python3 basic/offline_inference/embed.py

623 + - python3 basic/offline_inference/score.py

624 + # Multi-modal models

625 - python3 offline_inference/audio_language.py --seed 0

626 - python3 offline_inference/vision_language.py --seed 0

627 - python3 offline_inference/vision_language_multi_image.py --seed 0

628 - python3 offline_inference/encoder_decoder_multimodal.py --model-type whisper --seed 0

629 + # Pooling models

630 - python3 pooling/embed/vision_embedding_offline.py --seed 0

631 + # Features demo

632 - python3 offline_inference/prefix_caching.py

633 - python3 offline_inference/llm_engine_example.py

634 - python3 others/tensorize_vllm_model.py --model facebook/opt-125m serialize --serialized-directory /tmp/ --suffix v1 && python3 others/tensorize_vllm_model.py --model facebook/opt-125m deserialize --path-to-tensors /tmp/vllm/facebook/opt-125m/v1/model.tensors

635 - python3 offline_inference/spec_decode.py --test --method eagle --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 2048

636 - python3 offline_inference/spec_decode.py --test --method eagle3 --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 1536

637

491 -- label: Platform Tests (CUDA) # 4min

492 - timeout_in_minutes: 15

493 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

638 +

639 +- label: Platform Tests (CUDA) # TBD

640 + timeout_in_minutes: 180

641 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

642 agent_pool: mi250_1

643 + working_dir: "/vllm-workspace/tests"

644 source_file_dependencies:

645 - vllm/

646 - tests/cuda

647 commands:

499 - - pytest -v -s cuda/test_cuda_context.py

500 - - pytest -v -s cuda/test_platform_no_cuda_init.py

648 + - pytest -v -s cuda/test_cuda_context.py

649 + - pytest -v -s cuda/test_platform_no_cuda_init.py

650

502 -- label: Samplers Test # 56min

503 - timeout_in_minutes: 75

504 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

651 +

652 +- label: Samplers Test # TBD

653 + timeout_in_minutes: 180

654 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

655 agent_pool: mi250_1

656 optional: true

657 + working_dir: "/vllm-workspace/tests"

658 source_file_dependencies:

659 - vllm/model_executor/layers

660 - vllm/sampling_metadata.py

661 + - vllm/v1/sample/

662 + - vllm/beam_search.py

663 - tests/samplers

664 - tests/conftest.py

665 + - vllm/_aiter_ops.py

666 + - vllm/platforms/rocm.py

667 commands:

513 - - pytest -v -s samplers

668 + - pytest -v -s samplers

669

515 -- label: LoRA Test %N # 20min

516 - timeout_in_minutes: 30

517 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

670 +

671 +- label: LoRA %N # TBD

672 + timeout_in_minutes: 180

673 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

674 agent_pool: mi250_1

519 - optional: true

675 parallelism: 4

676 + optional: true

677 + working_dir: "/vllm-workspace/tests"

678 source_file_dependencies:

679 - vllm/lora

680 - tests/lora

681 + - vllm/platforms/rocm.py

682 commands:

525 - - pytest -v -s lora \

526 - --shard-id=$$BUILDKITE_PARALLEL_JOB \

527 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

528 - --ignore=lora/test_chatglm3_tp.py \

529 - --ignore=lora/test_llama_tp.py \

530 - --ignore=lora/test_llm_with_multi_loras.py \

531 - --ignore=lora/test_olmoe_tp.py \

532 - --ignore=lora/test_deepseekv2_tp.py \

533 - --ignore=lora/test_gptoss_tp.py \

534 - --ignore=lora/test_qwen3moe_tp.py

535 -

536 -- label: PyTorch Compilation Unit Tests # 15min

537 - timeout_in_minutes: 30

538 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

683 + - pytest -v -s lora --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --ignore=lora/test_chatglm3_tp.py --ignore=lora/test_llama_tp.py --ignore=lora/test_llm_with_multi_loras.py --ignore=lora/test_olmoe_tp.py --ignore=lora/test_deepseekv2_tp.py --ignore=lora/test_gptoss_tp.py --ignore=lora/test_qwen3moe_tp.py

684 +

685 +

686 +- label: PyTorch Compilation Unit Tests # TBD

687 + timeout_in_minutes: 180

688 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

689 agent_pool: mi250_1

690 torch_nightly: true

691 + optional: true

692 + working_dir: "/vllm-workspace/tests"

693 source_file_dependencies:

542 - - vllm/

543 - - tests/compile

694 + - vllm/compilation/

695 + - vllm/model_executor/layers/

696 + - vllm/v1/worker/

697 + - vllm/v1/attention/

698 + - vllm/v1/cudagraph_dispatcher.py

699 + - vllm/config/compilation.py

700 + - csrc/

701 + - tests/compile

702 + - vllm/platforms/rocm.py

703 commands:

545 - - "find compile/ -maxdepth 1 -name 'test_*.py' -exec pytest -s -v {} \\\\;"

704 + - "find compile/ -maxdepth 1 -name 'test_*.py' -print0 | xargs -0 -n1 -I{} pytest -s -v '{}'"

705

547 -- label: PyTorch Compilation Passes Unit Tests

548 - timeout_in_minutes: 20

549 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

550 - agent_pool: mi250_1

551 - source_file_dependencies:

552 - - vllm/

553 - - tests/compile/passes

554 - commands:

555 - - "find compile/passes -maxdepth 1 -name 'test_*.py' -exec pytest -s -v {} \\\\;"

706

557 -- label: PyTorch Fullgraph Smoke Test # 15min

558 - timeout_in_minutes: 30

559 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

707 +- label: PyTorch Fullgraph Smoke Test # TBD

708 + timeout_in_minutes: 180

709 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

710 agent_pool: mi250_1

711 + optional: true

712 torch_nightly: true

713 + working_dir: "/vllm-workspace/tests"

714 source_file_dependencies:

563 - - vllm/

715 + - vllm/compilation/

716 + - vllm/model_executor/

717 + - vllm/v1/attention/

718 + - vllm/config/compilation.py

719 + - csrc/

720 - tests/compile

721 + - vllm/platforms/rocm.py

722 commands:

723 - "find compile/fullgraph/ -name 'test_*.py' -not -name 'test_full_graph.py' -exec pytest -s -v {} \\\\;"

724

568 -- label: PyTorch Fullgraph Test # 27min

569 - timeout_in_minutes: 40

570 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

725 +

726 +- label: PyTorch Fullgraph # TBD

727 + timeout_in_minutes: 180

728 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

729 agent_pool: mi250_1

730 + optional: true

731 torch_nightly: true

732 + working_dir: "/vllm-workspace/tests"

733 source_file_dependencies:

574 - - vllm/

575 - - tests/compile

576 - commands:

577 - - pytest -v -s compile/fullgraph/test_full_graph.py -k 'not test_fp8_kv_scale_compile'

734 + - vllm/compilation/

735 + - vllm/model_executor/

736 + - vllm/v1/attention/

737 + - vllm/config/compilation.py

738 + - csrc/

739 + - tests/compile

740 + - vllm/platforms/rocm.py

741 + commands:

742 + - pytest -v -s compile/fullgraph/test_full_graph.py -k 'not test_fp8_kv_scale_compile'

743

579 -- label: Cudagraph test # 15min

580 - timeout_in_minutes: 20

581 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

744 +

745 +- label: Cudagraph # TBD

746 + timeout_in_minutes: 180

747 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

748 agent_pool: mi250_1

749 + optional: true

750 + working_dir: "/vllm-workspace/tests"

751 source_file_dependencies:

752 - tests/v1/cudagraph

753 - vllm/v1/cudagraph_dispatcher.py

754 - vllm/config/compilation.py

755 - vllm/compilation

756 + - vllm/platforms/rocm.py

757 commands:

589 - - pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py

590 - - pytest -v -s v1/cudagraph/test_cudagraph_mode.py

758 + - pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py

759 + - pytest -v -s v1/cudagraph/test_cudagraph_mode.py

760 +

761

592 -- label: Kernels Core Operation Test # 48min

593 - timeout_in_minutes: 75

594 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

762 +- label: Kernels Core Operation Test # TBD

763 + timeout_in_minutes: 180

764 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

765 agent_pool: mi250_1

766 + working_dir: "/vllm-workspace/tests"

767 source_file_dependencies:

768 - csrc/

769 - tests/kernels/core

770 - tests/kernels/test_top_k_per_row.py

771 + - tests/kernels/test_concat_mla_q.py

772 + - vllm/model_executor/layers/rotary_embedding/

773 + - vllm/_aiter_ops.py

774 + - vllm/platforms/rocm.py

775 commands:

601 - - pytest -v -s kernels/core kernels/test_top_k_per_row.py

776 + - pytest -v -s kernels/core kernels/test_top_k_per_row.py

777

603 -- label: Kernels Attention Test %N # 23min

604 - timeout_in_minutes: 35

605 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

606 - agent_pool: mi250_1

607 - parallelism: 2

608 - source_file_dependencies:

609 - - csrc/attention/

610 - - vllm/v1/attention

611 - - vllm/model_executor/layers/attention

612 - - tests/kernels/attention

613 - commands:

614 - - pytest -v -s kernels/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

778

616 -- label: Kernels Quantization Test %N # 64min

617 - timeout_in_minutes: 90

618 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

779 +- label: Kernels Mamba Test # TBD

780 + timeout_in_minutes: 180

781 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

782 agent_pool: mi250_1

783 optional: true

621 - parallelism: 2

622 - source_file_dependencies:

623 - - csrc/quantization/

624 - - vllm/model_executor/layers/quantization

625 - - tests/kernels/quantization

626 - commands:

627 - - pytest -v -s kernels/quantization --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

628 -

629 -- label: Kernels MoE Test %N # 40min

630 - timeout_in_minutes: 60

631 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

632 - agent_pool: mi250_1

633 - parallelism: 2

634 - source_file_dependencies:

635 - - csrc/quantization/cutlass_w8a8/moe/

636 - - csrc/moe/

637 - - tests/kernels/moe

638 - - vllm/model_executor/layers/fused_moe/

639 - - vllm/distributed/device_communicators/

640 - - vllm/envs.py

641 - - vllm/config

642 - commands:

643 - - pytest -v -s kernels/moe --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

644 -

645 -- label: Kernels Mamba Test # 31min

646 - timeout_in_minutes: 45

647 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

648 - agent_pool: mi250_1

784 + working_dir: "/vllm-workspace/tests"

785 source_file_dependencies:

786 - csrc/mamba/

787 - tests/kernels/mamba

788 - vllm/model_executor/layers/mamba/ops

789 + - vllm/platforms/rocm.py

790 commands:

654 - - pytest -v -s kernels/mamba

791 + - pytest -v -s kernels/mamba

792 +

793

656 -- label: Kernels Helion Test # 20min

657 - timeout_in_minutes: 30

658 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

794 +- label: Kernels Helion Test # TBD

795 + timeout_in_minutes: 180

796 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

797 agent_pool: mi250_1

798 + optional: true

799 + working_dir: "/vllm-workspace/tests"

800 source_file_dependencies:

801 - vllm/utils/import_utils.py

802 - tests/kernels/helion/

803 + - vllm/platforms/rocm.py

804 commands:

664 - - pip install helion

665 - - pytest -v -s kernels/helion/

805 + - pip install helion

806 + - pytest -v -s kernels/helion/

807

667 -- label: Model Executor Test # 23min

668 - timeout_in_minutes: 35

669 - torch_nightly: true

670 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

808 +

809 +- label: Model Executor # TBD

810 + timeout_in_minutes: 180

811 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

812 agent_pool: mi250_1

813 + optional: true

814 + torch_nightly: true

815 + working_dir: "/vllm-workspace/tests"

816 source_file_dependencies:

817 - vllm/engine/arg_utils.py

818 - vllm/config/model.py

819 - vllm/model_executor

820 - tests/model_executor

821 - tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py

822 + - vllm/_aiter_ops.py

823 + - vllm/platforms/rocm.py

824 commands:

679 - - apt-get update && apt-get install -y curl libsodium23

680 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

681 - - pytest -v -s model_executor

682 - - pytest -v -s entrypoints/openai/completion/test_tensorizer_entrypoint.py

825 + - apt-get update && apt-get install -y curl libsodium23

826 + - export VLLM_WORKER_MULTIPROC_METHOD=spawn

827 + - pytest -v -s model_executor

828 + - pytest -v -s entrypoints/openai/completion/test_tensorizer_entrypoint.py

829

684 -- label: Benchmarks # 11min

685 - timeout_in_minutes: 20

686 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

830 +

831 +- label: Benchmarks # TBD

832 + timeout_in_minutes: 180

833 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

834 agent_pool: mi250_1

835 working_dir: "/vllm-workspace/.buildkite"

836 source_file_dependencies:

837 - benchmarks/

838 + - vllm/platforms/rocm.py

839 commands:

840 - bash scripts/run-benchmarks.sh

841

694 -- label: Benchmarks CLI Test # 7min

695 - timeout_in_minutes: 20

696 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

842 +

843 +- label: Benchmarks CLI Test # TBD

844 + timeout_in_minutes: 180

845 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

846 agent_pool: mi250_1

847 + working_dir: "/vllm-workspace/tests"

848 source_file_dependencies:

849 - vllm/

850 - tests/benchmarks/

851 commands:

852 - pytest -v -s benchmarks/

853

704 -- label: Quantization Test # 70min

705 - timeout_in_minutes: 90

706 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

707 - agent_pool: mi250_1

708 - source_file_dependencies:

709 - - csrc/

710 - - vllm/model_executor/layers/quantization

711 - - tests/quantization

712 - commands:

713 - - uv pip install --system torchao==0.14.1

714 - - uv pip install --system conch-triton-kernels

715 - - VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py

716 -

717 -- label: LM Eval Small Models # 53min

718 - timeout_in_minutes: 75

719 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

720 - agent_pool: mi250_1

721 - optional: true

722 - source_file_dependencies:

723 - - csrc/

724 - - vllm/model_executor/layers/quantization

725 - autorun_on_main: true

726 - commands:

727 - - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt

854

729 -- label: OpenAI API correctness # 10min

730 - timeout_in_minutes: 15

731 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

855 +- label: OpenAI API correctness # TBD

856 + timeout_in_minutes: 180

857 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

858 agent_pool: mi250_1

859 + working_dir: "/vllm-workspace/tests"

860 source_file_dependencies:

861 - csrc/

862 - vllm/entrypoints/openai/

863 - vllm/model_executor/models/whisper.py

737 - - tools/

864 + - vllm/model_executor/layers/

865 + - vllm/v1/attention/backends/

866 + - vllm/v1/attention/selector.py

867 + - vllm/_aiter_ops.py

868 + - vllm/platforms/rocm.py

869 + - vllm/model_executor/model_loader/

870 commands:

871 - bash ../tools/install_torchcodec_rocm.sh || exit 1

872 - pytest -s entrypoints/openai/correctness/

873

742 -- label: Basic Models Tests (Initialization) # 15min

743 - timeout_in_minutes: 45

744 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

874 +

875 +- label: Basic Models Tests (Initialization) # TBD

876 + timeout_in_minutes: 180

877 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

878 agent_pool: mi250_1

879 torch_nightly: true

880 + working_dir: "/vllm-workspace/tests"

881 source_file_dependencies:

882 - vllm/

883 - tests/models/test_initialization.py

884 + - tests/models/registry.py

885 commands:

751 - - pytest -v -s models/test_initialization.py::test_can_initialize_small_subset

886 + - pytest -v -s models/test_initialization.py::test_can_initialize_small_subset

887

753 -- label: Basic Models Tests (Extra Initialization) %N # 15min

754 - timeout_in_minutes: 45

755 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

888 +

889 +- label: Basic Models Tests (Extra Initialization) %N # TBD

890 + timeout_in_minutes: 180

891 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

892 agent_pool: mi250_1

893 torch_nightly: true

894 parallelism: 2

895 + working_dir: "/vllm-workspace/tests"

896 source_file_dependencies:

897 - vllm/model_executor/models/

761 - - vllm/transformers_utils/

898 + - vllm/model_executor/layers/

899 - tests/models/test_initialization.py

900 + - tests/models/registry.py

901 + - vllm/_aiter_ops.py

902 + - vllm/platforms/rocm.py

903 commands:

764 - - pytest -v -s models/test_initialization.py \

765 - -k 'not test_can_initialize_small_subset' \

766 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

767 - --shard-id=$$BUILDKITE_PARALLEL_JOB

904 + - pytest -v -s models/test_initialization.py -k 'not test_can_initialize_small_subset' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB

905

769 -- label: Basic Models Tests (Other) # 15min

770 - timeout_in_minutes: 45

771 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

906 +

907 +- label: Basic Models Tests (Other) # TBD

908 + timeout_in_minutes: 180

909 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

910 agent_pool: mi250_1

773 - optional: true

911 torch_nightly: true

912 + working_dir: "/vllm-workspace/tests"

913 source_file_dependencies:

914 - vllm/

915 - tests/models/test_terratorch.py

916 - tests/models/test_transformers.py

917 - tests/models/test_registry.py

918 commands:

781 - - pytest -v -s models/test_terratorch.py models/test_transformers.py models/test_registry.py

919 + - pytest -v -s models/test_terratorch.py models/test_transformers.py models/test_registry.py

920

783 -- label: Basic Models Test (Other CPU) # 5min

784 - timeout_in_minutes: 10

785 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

921 +

922 +- label: Basic Models Test (Other CPU) # TBD

923 + timeout_in_minutes: 180

924 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

925 agent_pool: mi250_1

787 - torch_nightly: true

926 no_gpu: true

927 + optional: true

928 + torch_nightly: true

929 + working_dir: "/vllm-workspace/tests"

930 source_file_dependencies:

931 - vllm/

932 - tests/models/test_utils.py

933 - tests/models/test_vision.py

934 commands:

794 - - pytest -v -s models/test_utils.py models/test_vision.py

935 + - pytest -v -s models/test_utils.py models/test_vision.py

936

796 -- label: Language Models Tests (Standard) # 18min

797 - timeout_in_minutes: 25

798 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

799 - agent_pool: mi250_1

800 - torch_nightly: true

801 - source_file_dependencies:

802 - - vllm/

803 - - tests/models/language

804 - commands:

805 - - pip freeze | grep -E 'torch'

806 - - pytest -v -s models/language -m 'core_model and (not slow_test)'

937

808 -- label: Language Models Tests (Extra Standard) %N # 27min

809 - timeout_in_minutes: 45

810 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

938 +- label: Language Models Tests (Extra Standard) %N # TBD

939 + timeout_in_minutes: 180

940 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

941 agent_pool: mi250_1

812 - optional: true

942 torch_nightly: true

943 parallelism: 2

944 + working_dir: "/vllm-workspace/tests"

945 source_file_dependencies:

946 - vllm/model_executor/models/

947 + - vllm/model_executor/model_loader/

948 + - vllm/model_executor/layers/

949 + - vllm/v1/attention/backends/

950 + - vllm/v1/attention/selector.py

951 - tests/models/language/pooling/test_embedding.py

952 - tests/models/language/generation/test_common.py

953 - tests/models/language/pooling/test_classification.py

954 + - vllm/_aiter_ops.py

955 + - vllm/platforms/rocm.py

956 commands:

821 - - pip freeze | grep -E 'torch'

822 - - export TORCH_NCCL_BLOCKING_WAIT=1

823 - - pytest -v -s models/language -m 'core_model and slow_test' \

824 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

825 - --shard-id=$$BUILDKITE_PARALLEL_JOB

957 + - pip freeze | grep -E 'torch'

958 + - export TORCH_NCCL_BLOCKING_WAIT=1

959 + - pytest -v -s models/language -m 'core_model and slow_test' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB

960

827 -- label: Language Models Tests (Hybrid) %N # 50min

828 - timeout_in_minutes: 75

829 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

961 +

962 +- label: Language Models Test (PPL) # TBD

963 + timeout_in_minutes: 180

964 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

965 agent_pool: mi250_1

831 - optional: true

832 - torch_nightly: true

833 - parallelism: 2

966 + working_dir: "/vllm-workspace/tests"

967 source_file_dependencies:

968 - vllm/

836 - - tests/models/language/generation

969 + - tests/models/language/generation_ppl_test

970 commands:

838 - - uv pip install --system --no-build-isolation 'git+https://github.com/AndreasKaratzas/mamba@fix-rocm-7.0-warp-size-constexpr'

839 - - uv pip install --system --no-build-isolation 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.2'

840 - - pytest -v -s models/language/generation \

841 - -m hybrid_model \

842 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

843 - --shard-id=$$BUILDKITE_PARALLEL_JOB

971 + - pytest -v -s models/language/generation_ppl_test

972 +

973

845 -- label: Language Models Test (Extended Generation) # 80min

846 - timeout_in_minutes: 110

847 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

974 +- label: Language Models Test (Extended Pooling) # TBD

975 + timeout_in_minutes: 180

976 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

977 agent_pool: mi250_1

849 - optional: true

978 + working_dir: "/vllm-workspace/tests"

979 source_file_dependencies:

980 - vllm/

852 - - tests/models/language/generation

981 + - tests/models/language/pooling

982 commands:

854 - - uv pip install --system --no-build-isolation 'git+https://github.com/AndreasKaratzas/mamba@fix-rocm-7.0-warp-size-constexpr'

855 - - uv pip install --system --no-build-isolation 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.2'

856 - - pytest -v -s models/language/generation -m '(not core_model) and (not hybrid_model)'

983 + - pytest -v -s models/language/pooling -m 'not core_model'

984

858 -- label: Language Models Test (PPL) # 80min

859 - timeout_in_minutes: 110

860 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

985 +

986 +- label: Language Models Test (MTEB) # TBD

987 + timeout_in_minutes: 180

988 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

989 agent_pool: mi250_1

862 - optional: true

990 + working_dir: "/vllm-workspace/tests"

991 source_file_dependencies:

992 - vllm/

865 - - tests/models/language/generation_ppl_test

993 + - tests/models/language/pooling_mteb_test

994 commands:

867 - - pytest -v -s models/language/generation_ppl_test

995 + - pytest -v -s models/language/pooling_mteb_test

996 +

997

869 -- label: Language Models Test (Extended Pooling) # 36min

870 - timeout_in_minutes: 50

871 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

998 +- label: Multi-Modal Processor (CPU) # TBD

999 + timeout_in_minutes: 180

1000 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1001 agent_pool: mi250_1

1002 + no_gpu: true

1003 optional: true

1004 + working_dir: "/vllm-workspace/tests"

1005 source_file_dependencies:

1006 - vllm/

876 - - tests/models/language/pooling

1007 + - tests/models/multimodal

1008 + - tests/models/registry.py

1009 commands:

878 - - pytest -v -s models/language/pooling -m 'not core_model'

1010 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1011 + - pytest -v -s models/multimodal/processing --ignore models/multimodal/processing/test_tensor_schema.py

1012 +

1013

880 -- label: Language Models Test (MTEB) # 80min

881 - timeout_in_minutes: 110

882 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1014 +- label: Multi-Modal Accuracy Eval (Small Models) # TBD

1015 + timeout_in_minutes: 180

1016 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1017 agent_pool: mi250_1

1018 optional: true

1019 + working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

1020 source_file_dependencies:

886 - - vllm/

887 - - tests/models/language/pooling_mteb_test

1021 + - vllm/multimodal/

1022 + - vllm/inputs/

1023 + - vllm/v1/core/

1024 + - vllm/platforms/rocm.py

1025 + - vllm/model_executor/model_loader/

1026 commands:

889 - - pytest -v -s models/language/pooling_mteb_test

1027 + - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-mm-small.txt --tp-size=1

1028

891 -- label: Multi-Modal Processor Test (CPU) # 15min

892 - timeout_in_minutes: 60

893 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1029 +

1030 +- label: "Multi-Modal Models (Standard) 1: qwen2" # TBD

1031 + timeout_in_minutes: 180

1032 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1033 agent_pool: mi250_1

895 - no_gpu: true

1034 + torch_nightly: true

1035 + working_dir: "/vllm-workspace/tests"

1036 source_file_dependencies:

1037 - vllm/

1038 - tests/models/multimodal

899 - - tests/models/registry.py

1039 commands:

901 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

902 - - pytest -v -s models/multimodal/processing --ignore models/multimodal/processing/test_tensor_schema.py

1040 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1041 + - pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen2"

1042 + - pytest -v -s models/multimodal/generation/test_ultravox.py -m core_model

1043

904 -- label: Multi-Modal Processor Test # 44min

905 - timeout_in_minutes: 60

906 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1044 +

1045 +- label: "Multi-Modal Models (Standard) 2: qwen3 + gemma" # TBD

1046 + timeout_in_minutes: 180

1047 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1048 agent_pool: mi250_1

1049 + torch_nightly: true

1050 + working_dir: "/vllm-workspace/tests"

1051 source_file_dependencies:

1052 - vllm/

1053 - tests/models/multimodal

911 - - tests/models/registry.py

1054 commands:

913 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

914 - - pytest -v -s models/multimodal/processing

1055 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1056 + - pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen3 or gemma"

1057 + - pytest -v -s models/multimodal/generation/test_qwen2_5_vl.py -m core_model

1058 +

1059

916 -- label: Multi-Modal Models Test (Standard) # 60min

917 - timeout_in_minutes: 100

918 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1060 +- label: "Multi-Modal Models (Standard) 3: llava + qwen2_vl" # TBD

1061 + timeout_in_minutes: 180

1062 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1063 agent_pool: mi250_1

1064 torch_nightly: true

1065 + working_dir: "/vllm-workspace/tests"

1066 source_file_dependencies:

1067 - vllm/

1068 - tests/models/multimodal

1069 commands:

925 - - export MIOPEN_DEBUG_CONV_DIRECT=0

926 - - export MIOPEN_DEBUG_CONV_GEMM=0

927 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

928 - - pip freeze | grep -E 'torch'

929 - - pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/processing --ignore models/multimodal/pooling/test_prithvi_mae.py

930 - - pytest -v -s models/multimodal/pooling/test_prithvi_mae.py -m core_model

931 - - cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model # Otherwise, mp_method="spawn" doesn't work

1070 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1071 + - pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma"

1072 + - pytest -v -s models/multimodal/generation/test_qwen2_vl.py -m core_model

1073

933 -- label: Multi-Modal Accuracy Eval (Small Models) # 5min

934 - timeout_in_minutes: 10

935 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1074 +

1075 +- label: "Multi-Modal Models (Standard) 4: other + whisper" # TBD

1076 + timeout_in_minutes: 180

1077 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1078 agent_pool: mi250_1

937 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

1079 + torch_nightly: true

1080 + working_dir: "/vllm-workspace/tests"

1081 source_file_dependencies:

939 - - vllm/multimodal/

940 - - vllm/inputs/

941 - - vllm/v1/core/

1082 + - vllm/

1083 + - tests/models/multimodal/generation

1084 + - tests/models/multimodal/test_mapping.py

1085 commands:

943 - - export MIOPEN_DEBUG_CONV_DIRECT=0

944 - - export MIOPEN_DEBUG_CONV_GEMM=0

945 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-mm-small.txt

1086 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1087 + - pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/processing

1088 + - cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model

1089

947 -- label: Multi-Modal Models Test (Extended Generation 1) # 60min

948 - timeout_in_minutes: 120

949 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1090 +

1091 +- label: Multi-Modal Models (Extended Generation 1) # TBD

1092 + timeout_in_minutes: 180

1093 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1094 agent_pool: mi250_1

951 - optional: true

1095 + working_dir: "/vllm-workspace/tests"

1096 source_file_dependencies:

1097 - vllm/

1098 - tests/models/multimodal/generation

1099 - tests/models/multimodal/test_mapping.py

1100 commands:

957 - - export MIOPEN_DEBUG_CONV_DIRECT=0

958 - - export MIOPEN_DEBUG_CONV_GEMM=0

959 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

960 - - pytest -v -s models/multimodal/generation -m 'not core_model' --ignore models/multimodal/generation/test_common.py

961 - - pytest -v -s models/multimodal/test_mapping.py

1101 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1102 + - pytest -v -s models/multimodal/generation -m 'not core_model' --ignore models/multimodal/generation/test_common.py

1103 + - pytest -v -s models/multimodal/test_mapping.py

1104 +

1105

963 -- label: Multi-Modal Models Test (Extended Generation 2) # 60min

964 - timeout_in_minutes: 120

965 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1106 +- label: Multi-Modal Models (Extended Generation 2) # TBD

1107 + timeout_in_minutes: 180

1108 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1109 agent_pool: mi250_1

967 - optional: true

1110 + working_dir: "/vllm-workspace/tests"

1111 source_file_dependencies:

1112 - vllm/

1113 - tests/models/multimodal/generation

1114 commands:

972 - - export MIOPEN_DEBUG_CONV_DIRECT=0

973 - - export MIOPEN_DEBUG_CONV_GEMM=0

974 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

975 - - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=0) and not core_model'

1115 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1116 + - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=0) and not core_model'

1117

977 -- label: Multi-Modal Models Test (Extended Generation 3) # 75min

978 - timeout_in_minutes: 150

979 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1118 +

1119 +- label: Multi-Modal Models (Extended Generation 3) # TBD

1120 + timeout_in_minutes: 180

1121 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1122 agent_pool: mi250_1

981 - optional: true

1123 + working_dir: "/vllm-workspace/tests"

1124 source_file_dependencies:

1125 - vllm/

1126 - tests/models/multimodal/generation

1127 commands:

986 - - export MIOPEN_DEBUG_CONV_DIRECT=0

987 - - export MIOPEN_DEBUG_CONV_GEMM=0

988 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

989 - - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model'

1128 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

1129 + - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model'

1130 +

1131

991 -- label: Multi-Modal Models Test (Extended Pooling) # 60min

992 - timeout_in_minutes: 120

993 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1132 +- label: Multi-Modal Models (Extended Pooling) # TBD

1133 + timeout_in_minutes: 180

1134 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1135 agent_pool: mi250_1

995 - optional: true

1136 + working_dir: "/vllm-workspace/tests"

1137 source_file_dependencies:

1138 - vllm/

1139 - tests/models/multimodal/pooling

1140 commands:

1000 - - export MIOPEN_DEBUG_CONV_DIRECT=0

1001 - - export MIOPEN_DEBUG_CONV_GEMM=0

1002 - - pytest -v -s models/multimodal/pooling -m 'not core_model'

1141 + - pytest -v -s models/multimodal/pooling -m 'not core_model'

1142

1004 -- label: Quantized Models Test # 45 min

1005 - timeout_in_minutes: 60

1006 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1007 - agent_pool: mi250_1

1008 - source_file_dependencies:

1009 - - vllm/model_executor/layers/quantization

1010 - - tests/models/quantization

1011 - commands:

1012 - - pytest -v -s models/quantization

1143

1014 -- label: Transformers Nightly Models Test # 60 min

1015 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1144 +- label: Distributed Comm Ops # TBD

1145 + timeout_in_minutes: 180

1146 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1147 agent_pool: mi250_1

1017 - working_dir: "/vllm-workspace/"

1018 - optional: true

1019 - commands:

1020 - - pip install --upgrade git+https://github.com/huggingface/transformers

1021 - - pytest -v -s tests/models/test_initialization.py -k 'not (Gemma3 or ModernBert or Qwen2_5_VL or Qwen2_5vl or Qwen2VL or TransformersMultiModalEmbeddingModel or TransformersMultiModalForSequenceClassification or Ultravox or Phi4Multimodal or LlavaNextVideo or MiniCPMO or Lfm2Moe or PaliGemma or RobertaForSequenceClassification or Ovis2_5 or Fuyu or DeepseekOCR or KimiVL)'

1022 - - pytest -v -s tests/models/test_transformers.py

1023 - - pytest -v -s tests/models/multimodal/test_mapping.py -k 'not (Gemma3 or Qwen2VL or Qwen2_5_VL)'

1024 - - python3 examples/offline_inference/basic/chat.py

1025 - - VLLM_WORKER_MULTIPROC_METHOD=spawn python3 examples/offline_inference/audio_language.py --model-type whisper

1026 -

1027 -- label: Distributed Comm Ops Test # 7min

1028 - timeout_in_minutes: 20

1029 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1030 - agent_pool: mi250_2

1148 num_gpus: 2

1149 working_dir: "/vllm-workspace/tests"

1150 source_file_dependencies:

1151 - vllm/distributed

1152 - tests/distributed

1153 + - vllm/platforms/rocm.py

1154 commands:

1155 - pytest -v -s distributed/test_comm_ops.py

1156 - pytest -v -s distributed/test_shm_broadcast.py

1157 - pytest -v -s distributed/test_shm_buffer.py

1158 - pytest -v -s distributed/test_shm_storage.py

1159

1042 -- label: 2 Node Tests (4 GPUs in total) # 16min

1043 - timeout_in_minutes: 30

1044 - mirror_hardwares: [amdexperimental, amdproduction, amdmultinode, amdgfx90a]

1045 - agent_pool: mi250_4

1046 - optional: true

1160 +

1161 +- label: Distributed DP Tests (2 GPUs) # TBD

1162 + timeout_in_minutes: 180

1163 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1164 + agent_pool: mi250_2

1165 num_gpus: 2

1048 - num_nodes: 2

1166 + optional: true

1167 working_dir: "/vllm-workspace/tests"

1168 source_file_dependencies:

1169 - vllm/distributed/

1170 - vllm/engine/

1171 - vllm/executor/

1054 - - vllm/model_executor/models/

1055 - - tests/distributed/

1056 - - tests/examples/offline_inference/data_parallel.py

1172 + - vllm/worker/worker_base.py

1173 + - vllm/v1/engine/

1174 + - vllm/v1/worker/

1175 + - tests/v1/distributed

1176 + - tests/v1/entrypoints/openai/test_multi_api_servers.py

1177 + - vllm/platforms/rocm.py

1178 commands:

1058 - - # the following commands are for the first node, with ip 192.168.10.10 (ray environment already set up) | grep 'Same node test passed' | grep 'Node count test passed'

1059 - - VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py

1060 - - NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py

1061 - - python3 ../examples/offline_inference/data_parallel.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code

1062 - - VLLM_MULTI_NODE=1 pytest -v -s distributed/test_multi_node_assignment.py

1063 - - VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py

1064 - - # the following commands are for the second node, with ip 192.168.10.11 (ray environment already set up)

1065 - - VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py

1066 - - NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py

1067 - - python3 ../examples/offline_inference/data_parallel.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code

1068 -

1069 -- label: Distributed Tests (2 GPUs) # 68min

1070 - timeout_in_minutes: 90

1071 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1179 + - export TORCH_NCCL_BLOCKING_WAIT=1

1180 + - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_async_llm_dp.py

1181 + - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_eagle_dp.py

1182 + - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_external_lb_dp.py

1183 + - DP_SIZE=2 pytest -v -s v1/entrypoints/openai/test_multi_api_servers.py

1184 +

1185 +

1186 +- label: Distributed Compile + RPC Tests (2 GPUs) # TBD

1187 + timeout_in_minutes: 180

1188 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1189 agent_pool: mi250_2

1073 - optional: true

1190 num_gpus: 2

1191 + optional: true

1192 working_dir: "/vllm-workspace/tests"

1193 source_file_dependencies:

1194 - vllm/compilation/

@@ -1083,40 +1200,58 @@ steps:

1200 - vllm/v1/worker/

1201 - tests/compile/fullgraph/test_basic_correctness.py

1202 - tests/compile/test_wrapper.py

1086 - - tests/distributed/

1203 - tests/entrypoints/llm/test_collective_rpc.py

1088 - - tests/v1/distributed

1089 - - tests/v1/entrypoints/openai/test_multi_api_servers.py

1090 - - tests/v1/shutdown

1091 - - tests/v1/worker/test_worker_memory_snapshot.py

1092 - - examples/offline_inference/new_weight_syncing/

1204 + - vllm/platforms/rocm.py

1205 commands:

1206 - export TORCH_NCCL_BLOCKING_WAIT=1

1095 - - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_async_llm_dp.py

1096 - - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_eagle_dp.py

1097 - - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_external_lb_dp.py

1098 - - DP_SIZE=2 pytest -v -s v1/entrypoints/openai/test_multi_api_servers.py

1207 - pytest -v -s entrypoints/llm/test_collective_rpc.py

1208 - pytest -v -s ./compile/fullgraph/test_basic_correctness.py

1209 - pytest -v -s ./compile/test_wrapper.py

1210 +

1211 +

1212 +- label: Distributed Torchrun + Shutdown Tests (2 GPUs) # TBD

1213 + timeout_in_minutes: 180

1214 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1215 + agent_pool: mi250_2

1216 + num_gpus: 2

1217 + optional: true

1218 + working_dir: "/vllm-workspace/tests"

1219 + source_file_dependencies:

1220 + - vllm/distributed/

1221 + - vllm/engine/

1222 + - vllm/executor/

1223 + - vllm/worker/worker_base.py

1224 + - vllm/v1/engine/

1225 + - vllm/v1/worker/

1226 + - tests/distributed/

1227 + - tests/v1/shutdown

1228 + - tests/v1/worker/test_worker_memory_snapshot.py

1229 + - vllm/platforms/rocm.py

1230 + commands:

1231 + - export TORCH_NCCL_BLOCKING_WAIT=1

1232 - VLLM_TEST_SAME_HOST=1 torchrun --nproc-per-node=4 distributed/test_same_node.py | grep 'Same node test passed'

1233 - VLLM_TEST_SAME_HOST=1 VLLM_TEST_WITH_DEFAULT_DEVICE_SET=1 torchrun --nproc-per-node=4 distributed/test_same_node.py | grep 'Same node test passed'

1234 - CUDA_VISIBLE_DEVICES=0,1 pytest -v -s v1/shutdown

1235 - pytest -v -s v1/worker/test_worker_memory_snapshot.py

1236

1107 -- label: Distributed Model Tests (2 GPUs) # 37min

1108 - timeout_in_minutes: 50

1109 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1237 +

1238 +- label: Distributed Model Tests (2 GPUs) # TBD

1239 + timeout_in_minutes: 180

1240 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1241 agent_pool: mi250_2

1111 - optional: true

1242 num_gpus: 2

1243 working_dir: "/vllm-workspace/tests"

1244 source_file_dependencies:

1245 - vllm/model_executor/model_loader/sharded_state_loader.py

1246 - vllm/model_executor/models/

1247 + - vllm/model_executor/layers/

1248 + - vllm/v1/attention/backends/

1249 + - vllm/v1/attention/selector.py

1250 - tests/basic_correctness/

1251 - tests/model_executor/model_loader/test_sharded_state_loader.py

1252 - tests/models/

1253 + - vllm/_aiter_ops.py

1254 + - vllm/platforms/rocm.py

1255 commands:

1256 - TARGET_TEST_SUITE=L4 pytest basic_correctness/ -v -s -m 'distributed(num_gpus=2)'

1257 - CUDA_VISIBLE_DEVICES=0,1 pytest -v -s model_executor/model_loader/test_sharded_state_loader.py

@@ -1125,46 +1260,52 @@ steps:

1260 - pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py

1261 - VLLM_WORKER_MULTIPROC_METHOD=spawn pytest models/multimodal/generation/test_whisper.py -v -s -m 'distributed(num_gpus=2)'

1262

1128 -- label: Plugin Tests (2 GPUs) # 40min

1129 - timeout_in_minutes: 60

1130 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1263 +

1264 +- label: Plugin Tests (2 GPUs) # TBD

1265 + timeout_in_minutes: 180

1266 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1267 agent_pool: mi250_2

1268 num_gpus: 2

1269 + optional: true

1270 working_dir: "/vllm-workspace/tests"

1271 source_file_dependencies:

1272 - vllm/plugins/

1273 - tests/plugins/

1274 + - vllm/platforms/rocm.py

1275 commands:

1138 - # begin platform plugin and general plugin tests, all the code in-between runs on dummy platform

1276 + # BEGIN: platform plugin and general plugin tests, all the code in-between runs on dummy platform

1277 - pip install -e ./plugins/vllm_add_dummy_platform

1278 - pytest -v -s plugins_tests/test_platform_plugins.py

1279 - pip uninstall vllm_add_dummy_platform -y

1142 - # end platform plugin tests

1143 - # begin io_processor plugins test, all the code in between uses the prithvi_io_processor plugin

1280 + # END: platform plugin tests

1281 + # BEGIN: `io_processor` plugins test, all the code in between uses the `prithvi_io_processor` plugin

1282 - pip install -e ./plugins/prithvi_io_processor_plugin

1283 - pytest -v -s plugins_tests/test_io_processor_plugins.py

1284 - pip uninstall prithvi_io_processor_plugin -y

1147 - # test bge_m3_sparse io_processor plugin

1285 + # END: `io_processor` plugins test

1286 + # BEGIN: `bge_m3_sparse io_processor` test

1287 - pip install -e ./plugins/bge_m3_sparse_plugin

1288 - pytest -v -s plugins_tests/test_bge_m3_sparse_io_processor_plugins.py

1289 - pip uninstall bge_m3_sparse_plugin -y

1151 - # end io_processor plugins test

1152 - # begin stat_logger plugins test

1290 + # END: `bge_m3_sparse io_processor` test

1291 + # BEGIN: `stat_logger` plugins test

1292 - pip install -e ./plugins/vllm_add_dummy_stat_logger

1293 - pytest -v -s plugins_tests/test_stats_logger_plugins.py

1294 - pip uninstall dummy_stat_logger -y

1156 - # end stat_logger plugins test

1157 - # other tests continue here:

1295 + # END: `stat_logger` plugins test

1296 + # BEGIN: other tests

1297 - pytest -v -s plugins_tests/test_scheduler_plugins.py

1298 - pip install -e ./plugins/vllm_add_dummy_model

1299 - pytest -v -s distributed/test_distributed_oot.py

1300 - pytest -v -s entrypoints/openai/chat_completion/test_oot_registration.py

1301 - pytest -v -s models/test_oot_registration.py

1302 - pytest -v -s plugins/lora_resolvers

1303 + # END: other tests

1304

1165 -- label: Pipeline + Context Parallelism Test # 45min

1166 - timeout_in_minutes: 60

1167 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1305 +

1306 +- label: Pipeline + Context Parallelism (4 GPUs) # TBD

1307 + timeout_in_minutes: 180

1308 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1309 agent_pool: mi250_4

1310 num_gpus: 4

1311 working_dir: "/vllm-workspace/tests"

@@ -1173,325 +1314,130 @@ steps:

1314 - vllm/engine/

1315 - vllm/executor/

1316 - vllm/model_executor/models/

1317 + - vllm/model_executor/layers/

1318 + - vllm/v1/attention/backends/

1319 + - vllm/v1/attention/selector.py

1320 - tests/distributed/

1321 + - vllm/_aiter_ops.py

1322 + - vllm/platforms/rocm.py

1323 commands:

1324 - pytest -v -s distributed/test_pp_cudagraph.py

1325 - pytest -v -s distributed/test_pipeline_parallel.py

1326

1181 -- label: LoRA TP Test (Distributed) # 17 min

1182 - timeout_in_minutes: 30

1183 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1327 +

1328 +- label: Ray Dependency Compatibility Check # TBD

1329 + timeout_in_minutes: 180

1330 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1331 + agent_pool: mi250_1

1332 + working_dir: "/"

1333 + source_file_dependencies:

1334 + - requirements/

1335 + - setup.py

1336 + - vllm/platforms/rocm.py

1337 + commands:

1338 + - bash /vllm-workspace/.buildkite/scripts/check-ray-compatibility.sh

1339 +

1340 +

1341 +- label: Distributed NixlConnector PD accuracy (4 GPUs) # TBD

1342 + timeout_in_minutes: 180

1343 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1344 agent_pool: mi250_4

1345 num_gpus: 4

1346 + working_dir: "/vllm-workspace/tests"

1347 source_file_dependencies:

1187 - - vllm/lora

1188 - - tests/lora

1348 + - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

1349 + - tests/v1/kv_connector/nixl_integration/

1350 + - vllm/platforms/rocm.py

1351 commands:

1190 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1191 - - pytest -v -s -x lora/test_chatglm3_tp.py

1192 - - pytest -v -s -x lora/test_llama_tp.py

1193 - - pytest -v -s -x lora/test_llm_with_multi_loras.py

1194 - - pytest -v -s -x lora/test_olmoe_tp.py

1352 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

1353 + - ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

1354

1196 -- label: Weight Loading Multiple GPU Test # 33min

1197 - timeout_in_minutes: 45

1198 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1199 - agent_pool: mi250_2

1200 - num_gpus: 2

1201 - optional: true

1202 - working_dir: "/vllm-workspace/tests"

1203 - source_file_dependencies:

1204 - - vllm/

1205 - - tests/weight_loading

1206 - commands:

1207 - - bash weight_loading/run_model_weight_loading_test.sh -c weight_loading/models-amd.txt

1208 -

1209 -- label: Weight Loading Multiple GPU Test - Large Models # optional

1210 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1211 - agent_pool: mi250_2

1212 - num_gpus: 2

1213 - optional: true

1214 - working_dir: "/vllm-workspace/tests"

1215 - source_file_dependencies:

1216 - - vllm/

1217 - - tests/weight_loading

1218 - commands:

1219 - - bash weight_loading/run_model_weight_loading_test.sh -c weight_loading/models-large-amd.txt

1355

1221 -- label: NixlConnector PD accuracy tests (Distributed) # 30min

1222 - timeout_in_minutes: 30

1223 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1356 +- label: DP EP Distributed NixlConnector PD accuracy tests (4 GPUs) # TBD

1357 + timeout_in_minutes: 180

1358 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1359 agent_pool: mi250_4

1360 num_gpus: 4

1361 working_dir: "/vllm-workspace/tests"

1362 source_file_dependencies:

1228 - - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

1229 - - tests/v1/kv_connector/nixl_integration/

1363 + - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

1364 + - tests/v1/kv_connector/nixl_integration/

1365 + - vllm/platforms/rocm.py

1366 commands:

1231 - - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

1232 - - ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

1367 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

1368 + - DP_EP=1 ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

1369

1234 -- label: DP EP NixlConnector PD accuracy tests (Distributed) # 15min

1235 - timeout_in_minutes: 15

1236 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1237 - agent_pool: mi250_4

1238 - num_gpus: 4

1370 +

1371 +- label: NixlConnector PD + Spec Decode acceptance (2 GPUs) # TBD

1372 + timeout_in_minutes: 180

1373 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1374 + agent_pool: mi250_2

1375 + num_gpus: 2

1376 working_dir: "/vllm-workspace/tests"

1377 source_file_dependencies:

1241 - - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

1242 - - tests/v1/kv_connector/nixl_integration/

1378 + - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

1379 + - vllm/v1/worker/kv_connector_model_runner_mixin.py

1380 + - tests/v1/kv_connector/nixl_integration/

1381 + - vllm/platforms/rocm.py

1382 commands:

1244 - - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

1245 - - DP_EP=1 ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

1383 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

1384 + - ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/spec_decode_acceptance_test.sh

1385

1247 -- label: Distributed Tests (A100) # 68min

1248 - timeout_in_minutes: 90

1249 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1250 - agent_pool: mi250_4

1251 - optional: true

1252 - num_gpus: 4

1253 - source_file_dependencies:

1254 - - vllm/

1255 - commands:

1256 - - export TORCH_NCCL_BLOCKING_WAIT=1

1257 - - pytest -v -s distributed/test_custom_all_reduce.py

1258 - - torchrun --nproc_per_node=2 distributed/test_ca_buffer_sharing.py

1259 - - TARGET_TEST_SUITE=A100 pytest basic_correctness/ -v -s -m 'distributed(num_gpus=2)'

1260 - - pytest -v -s -x lora/test_mixtral.py

1386

1262 -- label: LM Eval Large Models # 80min

1263 - timeout_in_minutes: 110

1264 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1387 +- label: CrossLayer KV layout Distributed NixlConnector PD accuracy tests (4 GPUs) # TBD

1388 + timeout_in_minutes: 180

1389 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1390 agent_pool: mi250_4

1266 - optional: true

1391 num_gpus: 4

1268 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

1392 + working_dir: "/vllm-workspace/tests"

1393 source_file_dependencies:

1270 - - csrc/

1271 - - vllm/model_executor/layers/quantization

1394 + - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

1395 + - tests/v1/kv_connector/nixl_integration/

1396 + - vllm/platforms/rocm.py

1397 commands:

1273 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1274 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4

1398 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

1399 + - CROSS_LAYERS_BLOCKS=True ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

1400

1276 -- label: LM Eval Large Models (H100) # 80min

1277 - timeout_in_minutes: 110

1278 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1279 - agent_pool: mi250_4

1280 - optional: true

1281 - num_gpus: 4

1282 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

1283 - source_file_dependencies:

1284 - - csrc/

1285 - - vllm/model_executor/layers/quantization

1286 - commands:

1287 - - export VLLM_USE_DEEP_GEMM=0

1288 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=4

1401

1290 -- label: Distributed Tests (H200) # 68min

1291 - timeout_in_minutes: 90

1292 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1402 +- label: Distributed Tests (2 GPUs)(H100-MI250) # TBD

1403 + timeout_in_minutes: 180

1404 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]

1405 agent_pool: mi250_2

1294 - optional: true

1406 num_gpus: 2

1407 working_dir: "/vllm-workspace/"

1297 - commands:

1298 - - VLLM_TEST_CLEAN_GPU_MEMORY=1 pytest -v -s tests/compile/passes/distributed/test_async_tp.py

1299 - - pytest -v -s tests/compile/passes/distributed/test_sequence_parallelism.py

1300 - - pytest -v -s tests/distributed/test_context_parallel.py

1301 - - HIP_VISIBLE_DEVICES=0,1 VLLM_LOGGING_LEVEL=DEBUG python3 examples/offline_inference/data_parallel.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=allgather_reducescatter --disable-nccl-for-dp-synchronization

1302 -

1303 -- label: LM Eval Small Models (1 Card) # 15min

1304 - timeout_in_minutes: 20

1305 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1306 - agent_pool: mi250_1

1307 - source_file_dependencies:

1308 - - csrc/

1309 - - vllm/model_executor/layers/quantization

1310 - commands:

1311 - - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt

1312 -

1313 -- label: LM Eval Large Models (4 Card) # 80min

1314 - timeout_in_minutes: 110

1315 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1316 - agent_pool: mi250_4

1317 - optional: true

1318 - num_gpus: 4

1319 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

1320 - source_file_dependencies:

1321 - - csrc/

1322 - - vllm/model_executor/layers/quantization

1323 - commands:

1324 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1325 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4

1326 -

1327 -- label: ROCm LM Eval Large Models (8 Card) # 80min

1328 - timeout_in_minutes: 110

1329 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1330 - agent_pool: mi250_8

1331 - num_gpus: 8

1332 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

1333 - commands:

1334 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1335 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8

1336 -

1337 -- label: ROCm GPT-OSS Eval # 80min

1338 - timeout_in_minutes: 60

1339 - working_dir: "/vllm-workspace/"

1340 - agent_pool: mi250_1

1341 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1342 - optional: true

1408 source_file_dependencies:

1344 - - tests/evals/gpt_oss

1345 - - vllm/model_executor/models/gpt_oss.py

1346 - - vllm/model_executor/layers/quantization/mxfp4.py

1347 - - vllm/v1/attention/backends/flashinfer.py

1348 - commands:

1349 - - uv pip install --system 'gpt-oss[eval]==0.0.5'

1350 - - VLLM_ROCM_USE_AITER_MHA=0 VLLM_ROCM_USE_AITER=1 VLLM_USE_AITER_UNIFIED_ATTENTION=1 pytest -s -v tests/evals/gpt_oss/test_gpqa_correctness.py --model openai/gpt-oss-20b --metric 0.58

1351 -

1352 -- label: DeepSeek V2-Lite Accuracy # 70min

1353 - timeout_in_minutes: 60

1354 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1355 - agent_pool: mi250_4

1356 - optional: true

1357 - num_gpus: 4

1358 - working_dir: "/vllm-workspace"

1359 - commands:

1360 - - bash .buildkite/scripts/scheduled_integration_test/deepseek_v2_lite_ep_eplb.sh 0.25 200 8010

1361 -

1362 -- label: Qwen3-Next-80B-A3B-Instruct MTP Async EPLB Accuracy # 70min

1363 - timeout_in_minutes: 60

1364 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90a]

1365 - agent_pool: mi250_4

1366 - optional: true

1367 - num_gpus: 4

1368 - working_dir: "/vllm-workspace"

1409 + - vllm/distributed/

1410 + - vllm/v1/distributed/

1411 + - vllm/model_executor/layers/fused_moe/

1412 + - vllm/v1/attention/backends/

1413 + - vllm/v1/attention/selector.py

1414 + - tests/distributed/test_context_parallel.py

1415 + - tests/v1/distributed/test_dbo.py

1416 + - examples/offline_inference/data_parallel.py

1417 + - vllm/_aiter_ops.py

1418 + - vllm/platforms/rocm.py

1419 commands:

1370 - - bash .buildkite/scripts/scheduled_integration_test/qwen3_next_mtp_async_eplb.sh 0.8 1319 8040

1371 -

1372 -

1373 -###################################################

1374 -# #

1375 -# MI325 test definitions #

1376 -# #

1377 -###################################################

1378 -

1379 -

1380 -##### fast check tests #####

1420 + - export TORCH_NCCL_BLOCKING_WAIT=1

1421 + - pytest -v -s tests/distributed/test_context_parallel.py

1422 + - VLLM_LOGGING_LEVEL=DEBUG python3 examples/offline_inference/data_parallel.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=allgather_reducescatter --disable-nccl-for-dp-synchronization

1423 + - pytest -v -s tests/v1/distributed/test_dbo.py

1424

1382 -- label: Pytorch Nightly Dependency Override Check # 2min

1383 - # if this test fails, it means the nightly torch version is not compatible with some

1384 - # of the dependencies. Please check the error message and add the package to whitelist

1385 - # in /vllm/tools/pre_commit/generate_nightly_torch_test.py

1386 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1387 - agent_pool: mi325_1

1388 - grade: Blocking

1389 - optional: true

1390 - soft_fail: true

1391 - source_file_dependencies:

1392 - - requirements/nightly_torch_test.txt

1393 - commands:

1394 - - bash standalone_tests/pytorch_nightly_dependency.sh

1425

1396 -- label: Async Engine, Inputs, Utils, Worker Test # 10min

1397 - timeout_in_minutes: 15

1398 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1399 - agent_pool: mi325_1

1400 - grade: Blocking

1401 - source_file_dependencies:

1402 - - vllm/

1403 - - tests/detokenizer

1404 - - tests/multimodal

1405 - - tests/utils_

1406 - commands:

1407 - - pytest -v -s detokenizer

1408 - - pytest -v -s -m 'not cpu_test' multimodal

1409 - - pytest -v -s utils_

1426 +#####################################################################################################################################

1427 +# #

1428 +# gfx942 #

1429 +# #

1430 +#####################################################################################################################################

1431

1411 -- label: Async Engine, Inputs, Utils, Worker, Config Test (CPU) # 20min

1412 - timeout_in_minutes: 30

1413 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1414 - agent_pool: mi325_1

1415 - optional: true

1416 - # grade: Blocking

1417 - source_file_dependencies:

1418 - - vllm/

1419 - - tests/test_inputs.py

1420 - - tests/test_outputs.py

1421 - - tests/test_pooling_params.py

1422 - - tests/multimodal

1423 - - tests/renderers

1424 - - tests/standalone_tests/lazy_imports.py

1425 - - tests/tokenizers_

1426 - - tests/tool_parsers

1427 - - tests/transformers_utils

1428 - - tests/config

1429 - no_gpu: true

1430 - commands:

1431 - - python3 standalone_tests/lazy_imports.py

1432 - - pytest -v -s test_inputs.py

1433 - - pytest -v -s test_outputs.py

1434 - - pytest -v -s test_pooling_params.py

1435 - - pytest -v -s -m 'cpu_test' multimodal

1436 - - pytest -v -s renderers

1437 - - pytest -v -s tokenizers_

1438 - - pytest -v -s tool_parsers

1439 - - pytest -v -s transformers_utils

1440 - - pytest -v -s config

1432

1442 -- label: Python-only Installation Test # 10min

1443 - timeout_in_minutes: 20

1444 - mirror_hardwares: [amdexperimental]

1433 +- label: Entrypoints Integration (LLM) # 13.1m

1434 + timeout_in_minutes: 22

1435 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1436 agent_pool: mi325_1

1437 optional: true

1447 - # grade: Blocking

1448 - source_file_dependencies:

1449 - - tests/standalone_tests/python_only_compile.sh

1450 - - setup.py

1451 - commands:

1452 - - bash standalone_tests/python_only_compile.sh

1453 -

1454 -- label: Basic Correctness Test # 20min

1455 - timeout_in_minutes: 30

1456 - mirror_hardwares: [amdexperimental, amdproduction]

1457 - agent_pool: mi325_1

1458 - # grade: Blocking

1438 fast_check: true

1439 torch_nightly: true

1461 - source_file_dependencies:

1462 - - vllm/

1463 - - tests/basic_correctness/test_basic_correctness

1464 - - tests/basic_correctness/test_cpu_offload

1465 - - tests/basic_correctness/test_cumem.py

1466 - commands:

1467 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1468 - - pytest -v -s basic_correctness/test_cumem.py

1469 - - pytest -v -s basic_correctness/test_basic_correctness.py

1470 - - pytest -v -s basic_correctness/test_cpu_offload.py

1471 -

1472 -- label: Entrypoints Unit Tests # 5min

1473 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1474 - agent_pool: mi325_1

1475 - grade: Blocking

1476 - timeout_in_minutes: 10

1477 - working_dir: "/vllm-workspace/tests"

1478 - fast_check: true

1479 - source_file_dependencies:

1480 - - vllm/entrypoints

1481 - - tests/entrypoints/

1482 - commands:

1483 - - pytest -v -s entrypoints/openai/tool_parsers

1484 - - pytest -v -s entrypoints/ --ignore=entrypoints/llm --ignore=entrypoints/openai --ignore=entrypoints/rpc --ignore=entrypoints/instrumentator --ignore=entrypoints/offline_mode --ignore=entrypoints/test_chat_utils.py --ignore=entrypoints/pooling

1485 -

1486 -- label: Entrypoints Integration Test (LLM) # 30min

1487 - timeout_in_minutes: 40

1488 - mirror_hardwares: [amdexperimental, amdproduction]

1489 - agent_pool: mi325_1

1490 - optional: true

1491 - # grade: Blocking

1440 working_dir: "/vllm-workspace/tests"

1493 - fast_check: true

1494 - torch_nightly: true

1441 source_file_dependencies:

1442 - vllm/

1443 - tests/entrypoints/llm

@@ -1499,36 +1445,35 @@ steps:

1445 commands:

1446 - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1447 - pytest -v -s entrypoints/llm --ignore=entrypoints/llm/test_generate.py --ignore=entrypoints/llm/test_collective_rpc.py

1502 - - pytest -v -s entrypoints/llm/test_generate.py # it needs a clean process

1503 - - pytest -v -s entrypoints/offline_mode # Needs to avoid interference with other tests

1448 + - pytest -v -s entrypoints/llm/test_generate.py

1449 + - pytest -v -s entrypoints/offline_mode

1450 +

1451

1505 -- label: Entrypoints Integration Test (API Server 1) # 100min

1506 - timeout_in_minutes: 130

1507 - mirror_hardwares: [amdexperimental, amdproduction]

1452 +- label: Entrypoints Integration (API Server 1) # 1h 7m

1453 + timeout_in_minutes: 180

1454 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1455 agent_pool: mi325_1

1509 - optional: true

1510 - # grade: Blocking

1511 - working_dir: "/vllm-workspace/tests"

1456 fast_check: true

1457 torch_nightly: true

1458 + working_dir: "/vllm-workspace/tests"

1459 source_file_dependencies:

1460 - vllm/

1461 - tests/entrypoints/openai

1462 - tests/entrypoints/test_chat_utils

1463 commands:

1464 - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1520 - - pytest -v -s entrypoints/openai --ignore=entrypoints/openai/chat_completion/test_chat_with_tool_reasoning.py --ignore=entrypoints/openai/chat_completion/test_oot_registration.py --ignore=entrypoints/openai/completion/test_tensorizer_entrypoint.py --ignore=entrypoints/openai/correctness/ --ignore=entrypoints/openai/tool_parsers/ --ignore=entrypoints/openai/responses

1465 + - pytest -v -s entrypoints/openai --ignore=entrypoints/openai/chat_completion/test_chat_with_tool_reasoning.py --ignore=entrypoints/openai/chat_completion/test_oot_registration.py --ignore=entrypoints/openai/completion/test_tensorizer_entrypoint.py --ignore=entrypoints/openai/correctness/ --ignore=entrypoints/openai/tool_parsers/ --ignore=entrypoints/openai/responses

1466 - pytest -v -s entrypoints/test_chat_utils.py

1467

1523 -- label: Entrypoints Integration Test (API Server 2)

1524 - timeout_in_minutes: 50

1525 - mirror_hardwares: [amdexperimental, amdproduction]

1468 +

1469 +- label: Entrypoints Integration (API Server 2) #26.9m

1470 + timeout_in_minutes: 45

1471 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1472 agent_pool: mi325_1

1473 optional: true

1528 - # grade: Blocking

1529 - working_dir: "/vllm-workspace/tests"

1474 fast_check: true

1475 torch_nightly: true

1476 + working_dir: "/vllm-workspace/tests"

1477 source_file_dependencies:

1478 - vllm/

1479 - tests/entrypoints/rpc

@@ -1540,15 +1485,14 @@ steps:

1485 - PYTHONPATH=/vllm-workspace pytest -v -s entrypoints/rpc

1486 - pytest -v -s tool_use

1487

1543 -- label: Entrypoints Integration Test (Pooling)

1544 - timeout_in_minutes: 50

1545 - mirror_hardwares: [amdexperimental, amdproduction]

1488 +

1489 +- label: Entrypoints Integration (Pooling) # 22.8m

1490 + timeout_in_minutes: 48

1491 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1492 agent_pool: mi325_1

1547 - optional: true

1548 - # grade: Blocking

1549 - working_dir: "/vllm-workspace/tests"

1493 fast_check: true

1494 torch_nightly: true

1495 + working_dir: "/vllm-workspace/tests"

1496 source_file_dependencies:

1497 - vllm/

1498 - tests/entrypoints/pooling

@@ -1556,61 +1500,50 @@ steps:

1500 - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1501 - pytest -v -s entrypoints/pooling

1502

1559 -- label: Entrypoints Integration Test (Responses API)

1560 - timeout_in_minutes: 50

1561 - mirror_hardwares: [amdexperimental, amdproduction]

1562 - agent_pool: mi325_1

1563 - optional: true

1564 - # grade: Blocking

1565 - working_dir: "/vllm-workspace/tests"

1566 - fast_check: true

1567 - torch_nightly: true

1568 - source_file_dependencies:

1569 - - vllm/

1570 - - tests/entrypoints/openai/responses

1571 - commands:

1572 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1573 - - pytest -v -s entrypoints/openai/responses

1503

1575 -- label: Distributed Tests (4 GPUs) # 35min

1576 - timeout_in_minutes: 50

1577 - mirror_hardwares: [amdexperimental, amdproduction]

1504 +- label: Distributed Torchrun + Examples (4 GPUs) # TBD

1505 + timeout_in_minutes: 80

1506 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1507 agent_pool: mi325_4

1579 - optional: true

1580 - # grade: Blocking

1581 - working_dir: "/vllm-workspace/tests"

1508 num_gpus: 4

1509 + working_dir: "/vllm-workspace/tests"

1510 source_file_dependencies:

1511 - vllm/distributed/

1585 - - tests/distributed/test_utils

1586 - - tests/distributed/test_pynccl

1587 - - tests/distributed/test_events

1588 - - tests/compile/fullgraph/test_basic_correctness.py

1512 + - tests/distributed/test_torchrun_example.py

1513 + - tests/distributed/test_torchrun_example_moe.py

1514 - examples/offline_inference/rlhf.py

1515 - examples/offline_inference/rlhf_colocate.py

1516 - examples/rl/

1517 - tests/examples/offline_inference/data_parallel.py

1593 - - tests/v1/distributed

1594 - - tests/v1/engine/test_engine_core_client.py

1595 - - tests/distributed/test_symm_mem_allreduce.py

1518 + - vllm/platforms/rocm.py

1519 commands:

1597 - # Work around HIP bug tracked here: https://github.com/ROCm/hip/issues/3876

1598 - # TODO: Remove when the bug is fixed in a future ROCm release

1520 - export TORCH_NCCL_BLOCKING_WAIT=1

1600 - # test with torchrun tp=2 and external_dp=2

1521 - torchrun --nproc-per-node=4 distributed/test_torchrun_example.py

1602 - # test with torchrun tp=2 and pp=2

1522 - PP_SIZE=2 torchrun --nproc-per-node=4 distributed/test_torchrun_example.py

1604 - # test with torchrun tp=4 and dp=1

1523 - TP_SIZE=4 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

1606 - # test with torchrun tp=2, pp=2 and dp=1

1524 - PP_SIZE=2 TP_SIZE=2 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

1608 - # test with torchrun tp=1 and dp=4 with ep

1525 - DP_SIZE=4 ENABLE_EP=1 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

1610 - # test with torchrun tp=2 and dp=2 with ep

1526 - TP_SIZE=2 DP_SIZE=2 ENABLE_EP=1 torchrun --nproc-per-node=4 distributed/test_torchrun_example_moe.py

1612 - # test with internal dp

1527 - python3 ../examples/offline_inference/data_parallel.py --enforce-eager

1528 + # rlhf examples

1529 + - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 ../examples/rl/rlhf_nccl.py

1530 + - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 ../examples/rl/rlhf_ipc.py

1531 +

1532 +

1533 +- label: Distributed DP Tests (4 GPUs) # TBD

1534 + timeout_in_minutes: 60

1535 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1536 + agent_pool: mi325_4

1537 + num_gpus: 4

1538 + working_dir: "/vllm-workspace/tests"

1539 + source_file_dependencies:

1540 + - vllm/distributed/

1541 + - tests/v1/distributed

1542 + - tests/v1/engine/test_engine_core_client.py

1543 + - tests/distributed/test_utils

1544 + - vllm/platforms/rocm.py

1545 + commands:

1546 + - export TORCH_NCCL_BLOCKING_WAIT=1

1547 - TP_SIZE=2 DP_SIZE=2 pytest -v -s v1/distributed/test_async_llm_dp.py

1548 - TP_SIZE=2 DP_SIZE=2 pytest -v -s v1/distributed/test_eagle_dp.py

1549 - TP_SIZE=2 DP_SIZE=2 pytest -v -s v1/distributed/test_external_lb_dp.py

@@ -1618,32 +1551,37 @@ steps:

1551 - TP_SIZE=1 DP_SIZE=4 pytest -v -s v1/distributed/test_hybrid_lb_dp.py

1552 - pytest -v -s v1/engine/test_engine_core_client.py::test_kv_cache_events_dp

1553 - pytest -v -s distributed/test_utils.py

1554 +

1555 +

1556 +- label: Distributed Compile + Comm (4 GPUs) # TBD

1557 + timeout_in_minutes: 40

1558 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1559 + agent_pool: mi325_4

1560 + num_gpus: 4

1561 + working_dir: "/vllm-workspace/tests"

1562 + source_file_dependencies:

1563 + - vllm/distributed/

1564 + - tests/distributed/test_pynccl

1565 + - tests/distributed/test_events

1566 + - tests/compile/fullgraph/test_basic_correctness.py

1567 + - tests/distributed/test_symm_mem_allreduce.py

1568 + - tests/distributed/test_multiproc_executor.py

1569 + - vllm/platforms/rocm.py

1570 + commands:

1571 + - export TORCH_NCCL_BLOCKING_WAIT=1

1572 - pytest -v -s compile/fullgraph/test_basic_correctness.py

1573 - pytest -v -s distributed/test_pynccl.py

1574 - pytest -v -s distributed/test_events.py

1575 - pytest -v -s distributed/test_symm_mem_allreduce.py

1625 - # TODO: create a dedicated test section for multi-GPU example tests

1626 - # when we have multiple distributed example tests

1627 - # OLD rlhf examples

1628 - - pushd ../examples/offline_inference

1629 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf.py

1630 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 RAY_DEDUP_LOGS=0 python3 rlhf_colocate.py

1631 - - popd

1632 - # NEW rlhf examples

1633 - - pushd ../examples/rl

1634 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf_nccl.py

1635 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf_ipc.py

1636 - - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 rlhf_async_new_apis.py

1637 - - popd

1638 -

1639 -- label: Distributed Tests (8 GPUs) # 4min

1576 + - pytest -v -s distributed/test_multiproc_executor.py::test_multiproc_executor_multi_node

1577 +

1578 +

1579 +- label: Distributed Tests (8 GPUs)(H100-MI325) # 6.4m

1580 timeout_in_minutes: 10

1641 - mirror_hardwares: [amdexperimental, amdproduction]

1581 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1582 agent_pool: mi325_8

1643 - optional: true

1644 - # grade: Blocking

1645 - gpu: h100

1583 num_gpus: 8

1584 + optional: true

1585 working_dir: "/vllm-workspace/tests"

1586 source_file_dependencies:

1587 - examples/offline_inference/torchrun_dp_example.py

@@ -1652,428 +1590,474 @@ steps:

1590 - vllm/v1/engine/llm_engine.py

1591 - vllm/v1/executor/uniproc_executor.py

1592 - vllm/v1/worker/gpu_worker.py

1593 + - vllm/platforms/rocm.py

1594 commands:

1656 - # test with torchrun tp=2 and dp=4 with ep

1657 - # Work around HIP bug tracked here: https://github.com/ROCm/hip/issues/3876

1658 - # TODO: Remove when the bug is fixed in a future ROCm release

1595 - export TORCH_NCCL_BLOCKING_WAIT=1

1596 - torchrun --nproc-per-node=8 ../examples/offline_inference/torchrun_dp_example.py --tp-size=2 --pp-size=1 --dp-size=4 --enable-ep

1597

1662 -- label: EPLB Algorithm Test # 5min

1663 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1664 - agent_pool: mi325_1

1665 - grade: Blocking

1666 - timeout_in_minutes: 15

1667 - working_dir: "/vllm-workspace/tests"

1668 - source_file_dependencies:

1669 - - vllm/distributed/eplb

1670 - - tests/distributed/test_eplb_algo.py

1671 - commands:

1672 - - pytest -v -s distributed/test_eplb_algo.py

1598

1674 -- label: EPLB Execution Test # 10min

1675 - mirror_hardwares: [amdexperimental, amdproduction]

1599 +- label: Elastic EP Scaling Test # TBD

1600 + timeout_in_minutes: 180

1601 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1602 agent_pool: mi325_4

1677 - # grade: Blocking

1678 - timeout_in_minutes: 20

1679 - working_dir: "/vllm-workspace/tests"

1603 num_gpus: 4

1604 + optional: true

1605 + working_dir: "/vllm-workspace/tests"

1606 source_file_dependencies:

1682 - - vllm/distributed/eplb

1683 - - tests/distributed/test_eplb_execute.py

1607 + - vllm/distributed/

1608 + - vllm/engine/

1609 + - vllm/executor/

1610 + - vllm/compilation/

1611 + - tests/distributed/

1612 + - vllm/platforms/rocm.py

1613 commands:

1685 - - pytest -v -s distributed/test_eplb_execute.py

1686 - - pytest -v -s distributed/test_eplb_spec_decode.py

1614 + - pytest -v -s distributed/test_elastic_ep.py

1615

1688 -- label: Metrics, Tracing Test # 12min

1689 - timeout_in_minutes: 20

1690 - mirror_hardwares: [amdexperimental, amdproduction]

1691 - agent_pool: mi325_2

1692 - # grade: Blocking

1693 - num_gpus: 2

1616 +

1617 +- label: Engine # 11.3m

1618 + timeout_in_minutes: 35

1619 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1620 + agent_pool: mi325_1

1621 + working_dir: "/vllm-workspace/tests"

1622 source_file_dependencies:

1623 - vllm/

1696 - - tests/v1/tracing

1624 + - tests/engine

1625 + - tests/test_sequence

1626 + - tests/test_config

1627 + - tests/test_logger

1628 + - tests/test_vllm_port

1629 commands:

1698 - - "pip install \

1699 - 'opentelemetry-sdk>=1.26.0' \

1700 - 'opentelemetry-api>=1.26.0' \

1701 - 'opentelemetry-exporter-otlp>=1.26.0' \

1702 - 'opentelemetry-semantic-conventions-ai>=0.4.1'"

1703 - - pytest -v -s v1/tracing

1630 + - pytest -v -s engine test_sequence.py test_config.py test_logger.py test_vllm_port.py

1631

1705 -##### fast check tests #####

1706 -##### 1 GPU test #####

1632

1708 -- label: Regression Test # 7min

1709 - timeout_in_minutes: 20

1710 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1633 +- label: Engine (1 GPU) # TBD

1634 + timeout_in_minutes: 180

1635 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1636 agent_pool: mi325_1

1712 - grade: Blocking

1637 + optional: true

1638 + working_dir: "/vllm-workspace/tests"

1639 source_file_dependencies:

1714 - - vllm/

1715 - - tests/test_regression

1640 + - vllm/v1/engine/

1641 + - tests/v1/engine/

1642 + - vllm/platforms/rocm.py

1643 commands:

1717 - - pip install modelscope

1718 - - pytest -v -s test_regression.py

1719 - working_dir: "/vllm-workspace/tests" # optional

1644 + - pytest -v -s v1/engine/test_preprocess_error_handling.py

1645 + - pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py

1646 +

1647 +

1648 +- label: e2e Scheduling (1 GPU) # TBD

1649 + timeout_in_minutes: 180

1650 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1651 + agent_pool: mi325_1

1652 + optional: true

1653 + working_dir: "/vllm-workspace/tests"

1654 + source_file_dependencies:

1655 + - vllm/v1/

1656 + - tests/v1/e2e/general/

1657 + - vllm/platforms/rocm.py

1658 + commands:

1659 + - pytest -v -s v1/e2e/general/test_async_scheduling.py

1660

1721 -- label: Engine Test # 9min

1722 - timeout_in_minutes: 15

1723 - mirror_hardwares: [amdexperimental, amdproduction]

1661 +

1662 +- label: e2e Core (1 GPU) # TBD

1663 + timeout_in_minutes: 180

1664 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1665 agent_pool: mi325_1

1725 - # grade: Blocking

1666 + optional: true

1667 + working_dir: "/vllm-workspace/tests"

1668 source_file_dependencies:

1727 - - vllm/

1728 - - tests/engine

1729 - - tests/test_sequence

1730 - - tests/test_config

1731 - - tests/test_logger

1732 - - tests/test_vllm_port

1669 + - vllm/v1/

1670 + - tests/v1/e2e/

1671 + - vllm/platforms/rocm.py

1672 commands:

1734 - - pytest -v -s engine test_sequence.py test_config.py test_logger.py test_vllm_port.py

1673 + - pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py

1674 +

1675 +

1676 +- label: Spec Decode Eagle # TBD

1677 + timeout_in_minutes: 180

1678 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1679 + agent_pool: mi325_1

1680 + working_dir: "/vllm-workspace/tests"

1681 + source_file_dependencies:

1682 + - vllm/v1/spec_decode/

1683 + - vllm/v1/worker/gpu/spec_decode/

1684 + - vllm/model_executor/model_loader/

1685 + - vllm/v1/sample/

1686 + - vllm/model_executor/layers/

1687 + - tests/v1/e2e/spec_decode/

1688 + - vllm/platforms/rocm.py

1689 + commands:

1690 + - pytest -v -s v1/e2e/spec_decode -k "eagle_correctness"

1691 +

1692 +

1693 +- label: Spec Decode Speculators + MTP # TBD

1694 + timeout_in_minutes: 180

1695 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1696 + agent_pool: mi325_1

1697 + optional: true

1698 + working_dir: "/vllm-workspace/tests"

1699 + source_file_dependencies:

1700 + - vllm/v1/spec_decode/

1701 + - vllm/v1/worker/gpu/spec_decode/

1702 + - vllm/model_executor/model_loader/

1703 + - vllm/v1/sample/

1704 + - vllm/model_executor/layers/

1705 + - vllm/transformers_utils/configs/speculators/

1706 + - tests/v1/e2e/spec_decode/

1707 + - vllm/platforms/rocm.py

1708 + commands:

1709 + - pytest -v -s v1/e2e/spec_decode -k "speculators or mtp_correctness"

1710 +

1711 +

1712 +- label: Spec Decode Ngram + Suffix # TBD

1713 + timeout_in_minutes: 180

1714 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1715 + agent_pool: mi325_1

1716 + optional: true

1717 + working_dir: "/vllm-workspace/tests"

1718 + source_file_dependencies:

1719 + - vllm/v1/spec_decode/

1720 + - vllm/v1/worker/gpu/spec_decode/

1721 + - vllm/model_executor/model_loader/

1722 + - vllm/v1/sample/

1723 + - vllm/model_executor/layers/

1724 + - tests/v1/e2e/spec_decode/

1725 + - vllm/platforms/rocm.py

1726 + commands:

1727 + - pytest -v -s v1/e2e/spec_decode -k "ngram or suffix"

1728

1736 -- label: V1 Test e2e + engine # 65min

1737 - timeout_in_minutes: 90

1738 - mirror_hardwares: [amdexperimental, amdproduction]

1729 +

1730 +- label: Spec Decode Draft Model # TBD

1731 + timeout_in_minutes: 180

1732 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1733 agent_pool: mi325_1

1734 optional: true

1741 - # grade: Blocking

1735 + working_dir: "/vllm-workspace/tests"

1736 source_file_dependencies:

1743 - - vllm/

1744 - - tests/v1

1737 + - vllm/v1/spec_decode/

1738 + - vllm/v1/worker/gpu/spec_decode/

1739 + - vllm/model_executor/model_loader/

1740 + - vllm/v1/sample/

1741 + - vllm/model_executor/layers/

1742 + - tests/v1/e2e/spec_decode/

1743 + - vllm/platforms/rocm.py

1744 commands:

1746 - # TODO: accuracy does not match, whether setting

1747 - # VLLM_USE_FLASHINFER_SAMPLER or not on H100.

1748 - - pytest -v -s v1/e2e

1749 - - pytest -v -s v1/engine

1745 + - pytest -v -s v1/e2e/spec_decode -k "draft_model or no_sync or batch_inference"

1746

1751 -- label: V1 Test e2e (2 GPUs) # 65min

1752 - timeout_in_minutes: 90

1753 - mirror_hardwares: [amdexperimental, amdproduction]

1747 +

1748 +- label: V1 e2e (2 GPUs) # 7.1m

1749 + timeout_in_minutes: 12

1750 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1751 agent_pool: mi325_2

1752 optional: true

1756 - # grade: Blocking

1753 + working_dir: "/vllm-workspace/tests"

1754 source_file_dependencies:

1758 - - vllm/

1759 - - tests/v1

1755 + - vllm/

1756 + - tests/v1/e2e

1757 commands:

1761 - # Only run tests that need exactly 2 GPUs

1758 - pytest -v -s v1/e2e/spec_decode/test_spec_decode.py -k "tensor_parallelism"

1759

1764 -- label: V1 Test e2e (4 GPUs) # 65min

1765 - timeout_in_minutes: 90

1766 - mirror_hardwares: [amdexperimental, amdproduction]

1767 - # The test uses 4 GPUs, but we schedule it on 8-GPU machines for stability.

1768 - # See discussion here: https://github.com/vllm-project/vllm/pull/31040

1760 +

1761 +- label: V1 e2e (4 GPUs) # 52.6m

1762 + timeout_in_minutes: 106

1763 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1764 agent_pool: mi325_4

1770 - optional: true

1771 - # grade: Blocking

1765 + working_dir: "/vllm-workspace/tests"

1766 source_file_dependencies:

1773 - - vllm/

1774 - - tests/v1

1767 + - vllm/

1768 + - tests/v1/e2e

1769 commands:

1776 - # Only run tests that need 4 GPUs

1770 - pytest -v -s v1/e2e/spec_decode/test_spec_decode.py -k "eagle_correctness_heavy"

1771

1779 -- label: V1 Test entrypoints # 35min

1780 - timeout_in_minutes: 50

1781 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1772 +

1773 +- label: Entrypoints V1 # 25.7m

1774 + timeout_in_minutes: 45

1775 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1776 + agent_pool: mi325_1

1777 + optional: true

1778 + working_dir: "/vllm-workspace/tests"

1779 + source_file_dependencies:

1780 + - vllm/

1781 + - tests/v1

1782 + commands:

1783 + - pytest -v -s v1/entrypoints

1784 +

1785 +

1786 +- label: V1 Spec Decode # TBD

1787 + timeout_in_minutes: 40

1788 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1789 agent_pool: mi325_1

1783 - grade: Blocking

1790 + working_dir: "/vllm-workspace/tests"

1791 source_file_dependencies:

1785 - - vllm/

1786 - - tests/v1

1792 + - vllm/

1793 + - tests/v1/spec_decode

1794 commands:

1788 - - pytest -v -s v1/entrypoints

1795 + - pytest -v -s -m 'not slow_test' v1/spec_decode

1796

1790 -- label: V1 Test others # 42min

1791 - timeout_in_minutes: 60

1792 - mirror_hardwares: [amdexperimental, amdproduction]

1797 +

1798 +- label: V1 Sample + Logits # TBD

1799 + timeout_in_minutes: 40

1800 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1801 + agent_pool: mi325_1

1802 + optional: true

1803 + working_dir: "/vllm-workspace/tests"

1804 + source_file_dependencies:

1805 + - vllm/

1806 + - tests/v1/sample

1807 + - tests/v1/logits_processors

1808 + - tests/v1/test_oracle.py

1809 + - tests/v1/test_request.py

1810 + - tests/v1/test_outputs.py

1811 + commands:

1812 + - pytest -v -s v1/sample

1813 + - pytest -v -s v1/logits_processors

1814 + - pytest -v -s v1/test_oracle.py

1815 + - pytest -v -s v1/test_request.py

1816 + - pytest -v -s v1/test_outputs.py

1817 +

1818 +

1819 +- label: V1 Core + KV + Metrics # TBD

1820 + timeout_in_minutes: 40

1821 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1822 + agent_pool: mi325_1

1823 + optional: true

1824 + working_dir: "/vllm-workspace/tests"

1825 + source_file_dependencies:

1826 + - vllm/

1827 + - tests/v1/core

1828 + - tests/v1/executor

1829 + - tests/v1/kv_offload

1830 + - tests/v1/worker

1831 + - tests/v1/kv_connector/unit

1832 + - tests/v1/metrics

1833 + - tests/entrypoints/openai/correctness/test_lmeval.py

1834 + commands:

1835 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

1836 + - pytest -v -s -m 'not cpu_test' v1/core

1837 + - pytest -v -s v1/executor

1838 + - pytest -v -s v1/kv_offload

1839 + - pytest -v -s v1/worker

1840 + - pytest -v -s -m 'not cpu_test' v1/kv_connector/unit

1841 + - pytest -v -s -m 'not cpu_test' v1/metrics

1842 + - pip install -U git+https://github.com/robertgshaw2-redhat/lm-evaluation-harness.git@streaming-api

1843 + # - export HSA_NO_SCRATCH_RECLAIM=1

1844 + - pytest -v -s entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine

1845 +

1846 +

1847 +- label: V1 Speculative Decoding (slow) # TBD

1848 + timeout_in_minutes: 180

1849 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1850 agent_pool: mi325_1

1851 optional: true

1795 - # grade: Blocking

1796 - source_file_dependencies:

1797 - - vllm/

1798 - - tests/v1

1799 - commands:

1800 - # split the test to avoid interference

1801 - - uv pip install --system -r /vllm-workspace/requirements/kv_connectors.txt

1802 - - pytest -v -s -m 'not cpu_test' v1/core

1803 - - pytest -v -s v1/executor

1804 - - pytest -v -s v1/kv_offload

1805 - - pytest -v -s v1/sample

1806 - - pytest -v -s v1/logits_processors

1807 - - pytest -v -s v1/worker

1808 - - pytest -v -s v1/spec_decode

1809 - - pytest -v -s -m 'not cpu_test' v1/kv_connector/unit

1810 - - pytest -v -s -m 'not cpu_test' v1/metrics

1811 - - pytest -v -s v1/test_oracle.py

1812 - - pytest -v -s v1/test_request.py

1813 - - pytest -v -s v1/test_outputs.py

1814 - # Integration test for streaming correctness (requires special branch).

1815 - - pip install -U git+https://github.com/robertgshaw2-redhat/lm-evaluation-harness.git@streaming-api

1816 - - pytest -v -s entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine

1817 -

1818 -# TODO: Add the "V1 Test attention (MI300)" test group

1819 -

1820 -- label: V1 Test attention (H100) # 10min

1821 - mirror_hardwares: [amdexperimental, amdproduction]

1852 + working_dir: "/vllm-workspace/tests"

1853 + source_file_dependencies:

1854 + - vllm/v1/spec_decode/

1855 + - vllm/model_executor/models/

1856 + - vllm/v1/attention/

1857 + - vllm/model_executor/layers/

1858 + - tests/v1/spec_decode/

1859 + - vllm/platforms/rocm.py

1860 + commands:

1861 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_eagle.py

1862 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_extract_hidden_states.py

1863 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_max_len.py

1864 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_mtp.py

1865 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_ngram.py

1866 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_speculators_eagle3.py

1867 + - pytest -v -s -m 'slow_test' v1/spec_decode/test_tree_attention.py

1868 +

1869 +

1870 +- label: Acceptance Length Test (Large Models) # TBD

1871 + timeout_in_minutes: 180

1872 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1873 agent_pool: mi325_1

1874 optional: true

1824 - # grade: Blocking

1825 - timeout_in_minutes: 30

1826 - gpu: h100

1875 + working_dir: "/vllm-workspace/tests"

1876 source_file_dependencies:

1828 - - vllm/config/attention.py

1829 - - vllm/model_executor/layers/attention

1830 - - vllm/v1/attention

1831 - - tests/v1/attention

1877 + - vllm/v1/spec_decode/

1878 + - vllm/model_executor/models/mlp_speculator.py

1879 + - tests/v1/spec_decode/test_acceptance_length.py

1880 + - vllm/platforms/rocm.py

1881 commands:

1833 - - pytest -v -s v1/attention

1882 + - export VLLM_ALLOW_INSECURE_SERIALIZATION=1

1883 + - pytest -v -s v1/spec_decode/test_acceptance_length.py -m slow_test

1884

1835 -- label: Batch Invariance Tests (H100) # 10min

1836 - mirror_hardwares: [amdexperimental]

1885 +

1886 +- label: V1 attention (H100-MI325) # 14.5m

1887 + timeout_in_minutes: 40

1888 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1889 agent_pool: mi325_1

1838 - timeout_in_minutes: 25

1839 - gpu: h100

1890 + working_dir: "/vllm-workspace/tests"

1891 source_file_dependencies:

1841 - - vllm/v1/attention

1842 - - vllm/model_executor/layers

1843 - - tests/v1/determinism/

1892 + - vllm/config/attention.py

1893 + - vllm/model_executor/layers/attention

1894 + - vllm/v1/attention

1895 + - tests/v1/attention

1896 + - vllm/_aiter_ops.py

1897 + - vllm/envs.py

1898 + - vllm/platforms/rocm.py

1899 commands:

1845 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1846 - - pip install pytest-timeout pytest-forked

1847 - - pytest -v -s v1/determinism/test_batch_invariance.py

1848 - - pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py

1900 + - pytest -v -s v1/attention

1901

1850 -- label: V1 Test others (CPU) # 5 mins

1851 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

1902 +

1903 +- label: Batch Invariance (H100-MI325) # 5.2m

1904 + timeout_in_minutes: 12

1905 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1906 agent_pool: mi325_1

1853 - grade: Blocking

1907 optional: true

1908 + working_dir: "/vllm-workspace/tests"

1909 source_file_dependencies:

1856 - - vllm/

1857 - - tests/v1

1910 + - vllm/v1/attention

1911 + - vllm/model_executor/layers

1912 + - tests/v1/determinism/

1913 + - vllm/_aiter_ops.py

1914 + - vllm/platforms/rocm.py

1915 + commands:

1916 + - export VLLM_WORKER_MULTIPROC_METHOD=spawn

1917 + - pip install pytest-timeout pytest-forked

1918 + - pytest -v -s v1/determinism/test_batch_invariance.py

1919 + - pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py

1920 +

1921 +

1922 +- label: V1 others (CPU) # 10.4m

1923 + timeout_in_minutes: 28

1924 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1925 + agent_pool: mi325_1

1926 no_gpu: true

1927 + optional: true

1928 + working_dir: "/vllm-workspace/tests"

1929 + source_file_dependencies:

1930 + - vllm/

1931 + - tests/v1

1932 commands:

1860 - # split the test to avoid interference

1861 - - pytest -v -s -m 'cpu_test' v1/core

1862 - - pytest -v -s v1/structured_output

1863 - - pytest -v -s v1/test_serial_utils.py

1864 - - pytest -v -s -m 'cpu_test' v1/kv_connector/unit

1865 - - pytest -v -s -m 'cpu_test' v1/metrics

1933 + - pytest -v -s -m 'cpu_test' v1/core

1934 + - pytest -v -s v1/structured_output

1935 + - pytest -v -s v1/test_serial_utils.py

1936 + - pytest -v -s -m 'cpu_test' v1/kv_connector/unit

1937 + - pytest -v -s -m 'cpu_test' v1/metrics

1938

1939

1868 -- label: Examples Test # 30min

1869 - timeout_in_minutes: 45

1870 - mirror_hardwares: [amdexperimental, amdproduction]

1940 +- label: Examples # 24.5m

1941 + timeout_in_minutes: 55

1942 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1943 agent_pool: mi325_1

1944 optional: true

1873 - # grade: Blocking

1945 working_dir: "/vllm-workspace/examples"

1946 source_file_dependencies:

1947 - vllm/entrypoints

1948 - vllm/multimodal

1949 - examples/

1950 + - vllm/platforms/rocm.py

1951 commands:

1880 - - pip install tensorizer # for tensorizer test

1881 - # for basic

1952 + - pip install tensorizer

1953 + # Basic

1954 - python3 basic/offline_inference/chat.py --attention-backend TRITON_ATTN

1955 - python3 basic/offline_inference/generate.py --model facebook/opt-125m

1956 - python3 basic/offline_inference/generate.py --model meta-llama/Llama-2-13b-chat-hf --cpu-offload-gb 10

1957 - python3 basic/offline_inference/classify.py

1958 - python3 basic/offline_inference/embed.py

1959 - python3 basic/offline_inference/score.py

1888 - # for multi-modal models

1960 + # Multi-modal models

1961 - python3 offline_inference/audio_language.py --seed 0

1962 - python3 offline_inference/vision_language.py --seed 0

1963 - python3 offline_inference/vision_language_multi_image.py --seed 0

1964 - python3 offline_inference/encoder_decoder_multimodal.py --model-type whisper --seed 0

1893 - # for pooling models

1965 + # Pooling models

1966 - python3 pooling/embed/vision_embedding_offline.py --seed 0

1895 - # for features demo

1967 + # Features demo

1968 - python3 offline_inference/prefix_caching.py

1969 - python3 offline_inference/llm_engine_example.py

1970 - python3 others/tensorize_vllm_model.py --model facebook/opt-125m serialize --serialized-directory /tmp/ --suffix v1 && python3 others/tensorize_vllm_model.py --model facebook/opt-125m deserialize --path-to-tensors /tmp/vllm/facebook/opt-125m/v1/model.tensors

1971 - python3 offline_inference/spec_decode.py --test --method eagle --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 2048

1900 - # https://github.com/vllm-project/vllm/pull/26682 uses slightly more memory in PyTorch 2.9+ causing this test to OOM in 1xL4 GPU

1972 - python3 offline_inference/spec_decode.py --test --method eagle3 --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 1536

1902 - #- python3 offline_inference/spec_decode.py --test --method eagle3 --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 2048

1903 -

1904 -- label: Platform Tests (CUDA) # 4min

1905 - timeout_in_minutes: 15

1906 - mirror_hardwares: [amdexperimental, amdproduction]

1907 - agent_pool: mi325_1

1908 - # grade: Blocking

1909 - source_file_dependencies:

1910 - - vllm/

1911 - - tests/cuda

1912 - commands:

1913 - - pytest -v -s cuda/test_cuda_context.py

1914 - - pytest -v -s cuda/test_platform_no_cuda_init.py

1973

1916 -- label: Samplers Test # 56min

1917 - timeout_in_minutes: 75

1918 - mirror_hardwares: [amdexperimental, amdproduction]

1919 - agent_pool: mi325_1

1920 - optional: true

1921 - # grade: Blocking

1922 - source_file_dependencies:

1923 - - vllm/model_executor/layers

1924 - - vllm/sampling_metadata.py

1925 - - tests/samplers

1926 - - tests/conftest.py

1927 - commands:

1928 - - pytest -v -s samplers

1974

1930 -- label: LoRA Test %N # 20min each

1931 - timeout_in_minutes: 30

1932 - mirror_hardwares: [amdexperimental, amdproduction]

1975 +- label: Platform Tests (CUDA) # 5.0m

1976 + timeout_in_minutes: 9

1977 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1978 agent_pool: mi325_1

1979 optional: true

1935 - # grade: Blocking

1936 - source_file_dependencies:

1937 - - vllm/lora

1938 - - tests/lora

1939 - commands:

1940 - - pytest -v -s lora \

1941 - --shard-id=$$BUILDKITE_PARALLEL_JOB \

1942 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

1943 - --ignore=lora/test_chatglm3_tp.py \

1944 - --ignore=lora/test_llama_tp.py \

1945 - --ignore=lora/test_llm_with_multi_loras.py \

1946 - --ignore=lora/test_olmoe_tp.py \

1947 - --ignore=lora/test_deepseekv2_tp.py \

1948 - --ignore=lora/test_gptoss_tp.py \

1949 - --ignore=lora/test_qwen3moe_tp.py

1950 - parallelism: 4

1951 -

1952 -##### .buildkite/test_areas/pytorch.yaml #####

1953 -# corresponds to .buildkite/test_areas/pytorch.yaml

1954 -- label: PyTorch Compilation Unit Tests # 15min

1955 - timeout_in_minutes: 30

1956 - mirror_hardwares: [amdexperimental, amdproduction]

1957 - agent_pool: mi325_1

1958 - # grade: Blocking

1959 - torch_nightly: true

1960 - source_file_dependencies:

1961 - - vllm/

1962 - - tests/compile

1963 - commands:

1964 - # Run unit tests defined directly under compile/,

1965 - # not including subdirectories, which are usually heavier

1966 - # tests covered elsewhere.

1967 - # Use `find` to launch multiple instances of pytest so that

1968 - # they do not suffer from https://github.com/vllm-project/vllm/issues/28965

1969 - - "find compile/ -maxdepth 1 -name 'test_*.py' -exec pytest -s -v {} \\\\;"

1970 -

1971 -# corresponds to .buildkite/test_areas/pytorch.yaml

1972 -- label: PyTorch Compilation Passes Unit Tests

1973 - timeout_in_minutes: 20

1974 - mirror_hardwares: [amdexperimental, amdproduction]

1975 - agent_pool: mi325_1

1976 - source_file_dependencies:

1977 - - vllm/

1978 - - tests/compile/passes

1979 - commands:

1980 - # TODO: clean up this comment if not needed. It is used to

1981 - # keep track of the tests changes during vLLM IR Ops refactoring.

1982 - # Use `find` to launch multiple instances of pytest.

1983 - - "find compile/passes -maxdepth 1 -name 'test_*.py' -exec pytest -s -v {} \\\\;"

1984 -

1985 -- label: PyTorch Fullgraph Smoke Test # 15min

1986 - timeout_in_minutes: 30

1987 - mirror_hardwares: [amdexperimental, amdproduction]

1988 - agent_pool: mi325_1

1989 - # grade: Blocking

1990 - torch_nightly: true

1980 + working_dir: "/vllm-workspace/tests"

1981 source_file_dependencies:

1982 - vllm/

1993 - - tests/compile

1983 + - tests/cuda

1984 commands:

1995 - # Run smoke tests under fullgraph directory, except test_full_graph.py

1996 - # as it is a heavy test that is covered in other steps.

1997 - # Use `find` to launch multiple instances of pytest so that

1998 - # they do not suffer from https://github.com/vllm-project/vllm/issues/28965

1999 - - "find compile/fullgraph/ -name 'test_*.py' -not -name 'test_full_graph.py' -exec pytest -s -v {} \\\\;"

1985 + - pytest -v -s cuda/test_cuda_context.py

1986 + - pytest -v -s cuda/test_platform_no_cuda_init.py

1987

2001 -- label: PyTorch Fullgraph Test # 27min

2002 - timeout_in_minutes: 40

2003 - mirror_hardwares: [amdexperimental, amdproduction]

1988 +

1989 +- label: PyTorch Compilation Passes Unit Tests # TBD

1990 + timeout_in_minutes: 180

1991 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

1992 agent_pool: mi325_1

2005 - # grade: Blocking

2006 - torch_nightly: true

1993 + working_dir: "/vllm-workspace/tests"

1994 source_file_dependencies:

1995 - vllm/

2009 - - tests/compile

1996 + - tests/compile/passes

1997 commands:

2011 - - pytest -v -s compile/fullgraph/test_full_graph.py -k 'not test_fp8_kv_scale_compile'

2012 - # # Limit to no custom ops to reduce running time

2013 - # # Wrap with quotes to escape yaml and avoid starting -k string with a -

2014 - # - "pytest -v -s compile/distributed/test_fusions_e2e.py -k 'TRITON and not +quant_fp8 and not Llama-4'"

2015 - # Old E2E tests were removed in https://github.com/vllm-project/vllm/pull/33293

2016 - # in favor of new tests in fusions_e2e. We avoid replicating the new jobs in this file as it's deprecated.

1998 + - pytest -s -v compile/passes --ignore compile/passes/distributed

1999

2018 -- label: Cudagraph test

2019 - timeout_in_minutes: 20

2020 - mirror_hardwares: [amdexperimental, amdproduction]

2021 - agent_pool: mi325_1

2022 - source_file_dependencies:

2023 - - tests/v1/cudagraph

2024 - - vllm/v1/cudagraph_dispatcher.py

2025 - - vllm/config/compilation.py

2026 - - vllm/compilation

2027 - commands:

2028 - - pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py

2029 - - pytest -v -s v1/cudagraph/test_cudagraph_mode.py

2000

2031 -- label: Kernels Core Operation Test # 48min

2032 - timeout_in_minutes: 75

2033 - mirror_hardwares: [amdexperimental, amdproduction]

2001 +- label: Kernels Core Operation Test # 26.8m

2002 + timeout_in_minutes: 38

2003 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2004 agent_pool: mi325_1

2035 - # grade: Blocking

2005 + optional: true

2006 + working_dir: "/vllm-workspace/tests"

2007 source_file_dependencies:

2008 - csrc/

2009 - tests/kernels/core

2010 - tests/kernels/test_top_k_per_row.py

2011 + - tests/kernels/test_concat_mla_q.py

2012 + - vllm/model_executor/layers/rotary_embedding/

2013 + - vllm/_aiter_ops.py

2014 + - vllm/platforms/rocm.py

2015 commands:

2041 - - pytest -v -s kernels/core kernels/test_top_k_per_row.py

2016 + - pytest -v -s kernels/core kernels/test_top_k_per_row.py

2017

2043 -- label: Kernels Attention Test %N # 23min

2044 - timeout_in_minutes: 35

2045 - mirror_hardwares: [amdexperimental, amdproduction]

2018 +

2019 +- label: Kernels Attention Test %N # 17.7m

2020 + timeout_in_minutes: 28

2021 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2022 agent_pool: mi325_1

2047 - # grade: Blocking

2023 + parallelism: 2

2024 + working_dir: "/vllm-workspace/tests"

2025 source_file_dependencies:

2026 - csrc/attention/

2027 - vllm/v1/attention

2051 - # TODO: remove this dependency (https://github.com/vllm-project/vllm/issues/32267)

2028 - vllm/model_executor/layers/attention

2029 - tests/kernels/attention

2030 + - vllm/_aiter_ops.py

2031 + - vllm/envs.py

2032 + - vllm/platforms/rocm.py

2033 commands:

2055 - - pytest -v -s kernels/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

2056 - parallelism: 2

2034 + - pytest -v -s kernels/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

2035 +

2036

2058 -- label: Kernels Quantization Test %N # 64min

2059 - timeout_in_minutes: 90

2060 - mirror_hardwares: [amdexperimental, amdproduction]

2037 +- label: Kernels Quantization Test %N # 15.2m

2038 + timeout_in_minutes: 24

2039 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2040 agent_pool: mi325_1

2062 - optional: true

2063 - # grade: Blocking

2041 + parallelism: 2

2042 + working_dir: "/vllm-workspace/tests"

2043 source_file_dependencies:

2044 - csrc/quantization/

2045 - vllm/model_executor/layers/quantization

2046 - tests/kernels/quantization

2047 + - tests/kernels/quantization/test_rocm_skinny_gemms.py

2048 + - vllm/_aiter_ops.py

2049 + - vllm/platforms/rocm.py

2050 + - vllm/model_executor/kernels/

2051 commands:

2069 - - pytest -v -s kernels/quantization --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

2070 - parallelism: 2

2052 + - pytest -v -s kernels/quantization --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

2053

2072 -- label: Kernels MoE Test %N # 40min

2073 - timeout_in_minutes: 60

2074 - mirror_hardwares: [amdexperimental, amdproduction]

2054 +

2055 +- label: Kernels MoE Test %N # TBD

2056 + timeout_in_minutes: 19

2057 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2058 agent_pool: mi325_1

2076 - # grade: Blocking

2059 + parallelism: 4

2060 + working_dir: "/vllm-workspace/tests"

2061 source_file_dependencies:

2062 - csrc/quantization/cutlass_w8a8/moe/

2063 - csrc/moe/

@@ -2082,533 +2066,301 @@ steps:

2066 - vllm/distributed/device_communicators/

2067 - vllm/envs.py

2068 - vllm/config

2069 + - vllm/_aiter_ops.py

2070 + - vllm/platforms/rocm.py

2071 commands:

2086 - - pytest -v -s kernels/moe --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

2087 - parallelism: 2

2072 + - pytest -v -s kernels/moe --ignore=kernels/moe/test_modular_oai_triton_moe.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

2073 + - pytest -v -s kernels/moe/test_modular_oai_triton_moe.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT

2074

2089 -- label: Kernels FP8 MoE Test

2090 - timeout_in_minutes: 60

2091 - mirror_hardwares: [amdexperimental, amdproduction]

2075 +

2076 +- label: Kernels FP8 MoE Test # TBD

2077 + timeout_in_minutes: 180

2078 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2079 agent_pool: mi325_2

2080 optional: true

2081 + working_dir: "/vllm-workspace/tests"

2082 + source_file_dependencies:

2083 + - csrc/moe/

2084 + - csrc/quantization/w8a8/cutlass/moe/

2085 + - vllm/model_executor/layers/fused_moe/

2086 + - tests/kernels/moe/test_deepep_moe.py

2087 + - vllm/_aiter_ops.py

2088 + - vllm/platforms/rocm.py

2089 + - vllm/envs.py

2090 commands:

2091 - pytest -v -s kernels/moe/test_deepep_moe.py

2092

2097 -- label: Kernels Mamba Test # 31min

2098 - timeout_in_minutes: 45

2099 - mirror_hardwares: [amdexperimental, amdproduction]

2100 - agent_pool: mi325_1

2101 - # grade: Blocking

2102 - source_file_dependencies:

2103 - - csrc/mamba/

2104 - - tests/kernels/mamba

2105 - - vllm/model_executor/layers/mamba/ops

2106 - commands:

2107 - - pytest -v -s kernels/mamba

2093

2109 -- label: Kernels DeepGEMM Test (H100) # Nvidia-centric

2110 -# Not replicating for CUTLAS & CuTe

2111 - timeout_in_minutes: 45

2112 - gpu: h100

2113 - num_gpus: 1

2114 - source_file_dependencies:

2115 - - tools/install_deepgemm.sh

2116 - - vllm/utils/deep_gemm.py

2117 - - vllm/model_executor/layers/fused_moe

2118 - - vllm/model_executor/layers/quantization

2119 - - tests/kernels/quantization/test_block_fp8.py

2120 - - tests/kernels/moe/test_deepgemm.py

2121 - - tests/kernels/moe/test_batched_deepgemm.py

2122 - - tests/kernels/attention/test_deepgemm_attention.py

2123 - commands:

2124 - - pytest -v -s kernels/quantization/test_block_fp8.py -k deep_gemm

2125 - - pytest -v -s kernels/moe/test_deepgemm.py

2126 - - pytest -v -s kernels/moe/test_batched_deepgemm.py

2127 - - pytest -v -s kernels/attention/test_deepgemm_attention.py

2128 -

2129 -- label: Kernels Helion Test

2130 - timeout_in_minutes: 30

2131 - mirror_hardwares: [amdexperimental, amdproduction]

2094 +- label: ROCm AITER Ops Test # TBD

2095 + timeout_in_minutes: 180

2096 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2097 agent_pool: mi325_1

2098 + working_dir: "/vllm-workspace/tests"

2099 source_file_dependencies:

2134 - - vllm/utils/import_utils.py

2135 - - tests/kernels/helion/

2100 + - vllm/_aiter_ops.py

2101 + - vllm/envs.py

2102 + - vllm/platforms/rocm.py

2103 + - tests/rocm/aiter/

2104 + - vllm/v1/attention/backends/mla/rocm_aiter_mla.py

2105 + - vllm/v1/attention/selector.py

2106 commands:

2137 - - pip install helion

2138 - - pytest -v -s kernels/helion/

2107 + - pytest -v -s rocm/aiter/

2108

2140 -- label: Model Executor Test # 23min

2141 - timeout_in_minutes: 35

2142 - torch_nightly: true

2143 - mirror_hardwares: [amdexperimental, amdproduction]

2144 - agent_pool: mi325_1

2145 - # grade: Blocking

2146 - source_file_dependencies:

2147 - - vllm/engine/arg_utils.py

2148 - - vllm/config/model.py

2149 - - vllm/model_executor

2150 - - tests/model_executor

2151 - - tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py

2152 - commands:

2153 - - apt-get update && apt-get install -y curl libsodium23

2154 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2155 - - pytest -v -s model_executor

2156 - - pytest -v -s entrypoints/openai/completion/test_tensorizer_entrypoint.py

2109

2158 -- label: Benchmarks # 11min

2110 +- label: Benchmarks # 8.2m

2111 timeout_in_minutes: 20

2160 - mirror_hardwares: [amdexperimental, amdproduction]

2112 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2113 agent_pool: mi325_1

2162 - # grade: Blocking

2114 + optional: true

2115 working_dir: "/vllm-workspace/.buildkite"

2116 source_file_dependencies:

2117 - benchmarks/

2118 + - vllm/platforms/rocm.py

2119 commands:

2120 - bash scripts/run-benchmarks.sh

2121

2169 -- label: Benchmarks CLI Test # 7min

2170 - timeout_in_minutes: 20

2171 - mirror_hardwares: [amdexperimental, amdproduction]

2172 - agent_pool: mi325_1

2173 - # grade: Blocking

2174 - source_file_dependencies:

2175 - - vllm/

2176 - - tests/benchmarks/

2177 - commands:

2178 - - pytest -v -s benchmarks/

2122

2180 -- label: Quantization Test # 70min

2181 - timeout_in_minutes: 90

2182 - mirror_hardwares: [amdexperimental, amdproduction]

2123 +- label: Quantization # 36.1m

2124 + timeout_in_minutes: 60

2125 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2126 agent_pool: mi325_1

2184 - optional: true

2185 - # grade: Blocking

2127 + working_dir: "/vllm-workspace/tests"

2128 source_file_dependencies:

2129 - csrc/

2130 - vllm/model_executor/layers/quantization

2131 + - vllm/_aiter_ops.py

2132 + - vllm/platforms/rocm.py

2133 - tests/quantization

2134 commands:

2191 - # temporary install here since we need nightly, will move to requirements/test.in

2192 - # after torchao 0.12 release, and pin a working version of torchao nightly here

2193 -

2194 - # since torchao nightly is only compatible with torch nightly currently

2195 - # https://github.com/pytorch/ao/issues/2919, we'll have to skip new torchao tests for now

2196 - # we can only upgrade after this is resolved

2197 - # TODO(jerryzh168): resolve the above comment

2135 - uv pip install --system torchao==0.14.1

2136 - uv pip install --system conch-triton-kernels

2137 - VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py

2138

2202 -- label: LM Eval Small Models # 53min

2203 - timeout_in_minutes: 75

2204 - mirror_hardwares: [amdexperimental, amdproduction]

2205 - agent_pool: mi325_1

2206 - optional: true

2207 - # grade: Blocking

2208 - source_file_dependencies:

2209 - - csrc/

2210 - - vllm/model_executor/layers/quantization

2211 - autorun_on_main: true

2212 - commands:

2213 - - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt

2214 -

2215 -- label: OpenAI API correctness # 10min

2216 - timeout_in_minutes: 15

2217 - mirror_hardwares: [amdexperimental, amdproduction]

2218 - agent_pool: mi325_1

2219 - # grade: Blocking

2220 - source_file_dependencies:

2221 - - csrc/

2222 - - vllm/entrypoints/openai/

2223 - - vllm/model_executor/models/whisper.py

2224 - - tools/

2225 - commands: # LMEval+Transcription WER check

2226 - - bash ../tools/install_torchcodec_rocm.sh || exit 1

2227 - - pytest -s entrypoints/openai/correctness/

2228 -

2229 -

2230 -##### models test #####

2231 -

2232 -- label: Basic Models Tests (Initialization)

2233 - timeout_in_minutes: 45

2234 - mirror_hardwares: [amdexperimental, amdproduction]

2235 - agent_pool: mi325_1

2236 - # grade: Blocking

2237 - torch_nightly: true

2238 - source_file_dependencies:

2239 - - vllm/

2240 - - tests/models/test_initialization.py

2241 - commands:

2242 - # Run a subset of model initialization tests

2243 - - pytest -v -s models/test_initialization.py::test_can_initialize_small_subset

2244 -

2245 -- label: Basic Models Tests (Extra Initialization) %N

2246 - timeout_in_minutes: 45

2247 - mirror_hardwares: [amdexperimental, amdproduction]

2248 - agent_pool: mi325_1

2249 - # grade: Blocking

2250 - torch_nightly: true

2251 - source_file_dependencies:

2252 - - vllm/model_executor/models/

2253 - - vllm/transformers_utils/

2254 - - tests/models/test_initialization.py

2255 - commands:

2256 - # Only when vLLM model source is modified - test initialization of a large

2257 - # subset of supported models (the complement of the small subset in the above

2258 - # test.) Also run if model initialization test file is modified

2259 - - pytest -v -s models/test_initialization.py \

2260 - -k 'not test_can_initialize_small_subset' \

2261 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

2262 - --shard-id=$$BUILDKITE_PARALLEL_JOB

2263 - parallelism: 2

2264 -

2265 -- label: Basic Models Tests (Other)

2266 - timeout_in_minutes: 45

2267 - mirror_hardwares: [amdexperimental, amdproduction]

2268 - agent_pool: mi325_1

2269 - optional: true

2270 - # grade: Blocking

2271 - torch_nightly: true

2272 - source_file_dependencies:

2273 - - vllm/

2274 - - tests/models/test_terratorch.py

2275 - - tests/models/test_transformers.py

2276 - - tests/models/test_registry.py

2277 - commands:

2278 - - pytest -v -s models/test_terratorch.py models/test_transformers.py models/test_registry.py

2279 -

2280 -- label: Basic Models Test (Other CPU) # 5min

2281 - mirror_hardwares: [amdexperimental, amdproduction]

2282 - agent_pool: mi325_1

2283 - # grade: Blocking

2284 - timeout_in_minutes: 10

2285 - torch_nightly: true

2286 - source_file_dependencies:

2287 - - vllm/

2288 - - tests/models/test_utils.py

2289 - - tests/models/test_vision.py

2290 - no_gpu: true

2291 - commands:

2292 - - pytest -v -s models/test_utils.py models/test_vision.py

2139

2294 -- label: Language Models Tests (Standard)

2295 - timeout_in_minutes: 25

2296 - mirror_hardwares: [amdexperimental, amdproduction]

2140 +- label: Language Models Tests (Standard) # 22.8m

2141 + timeout_in_minutes: 38

2142 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2143 agent_pool: mi325_1

2144 optional: true

2299 - # grade: Blocking

2145 torch_nightly: true

2146 + working_dir: "/vllm-workspace/tests"

2147 source_file_dependencies:

2148 - vllm/

2149 - tests/models/language

2150 commands:

2305 - # Test standard language models, excluding a subset of slow tests

2306 - - pip freeze | grep -E 'torch'

2307 - - pytest -v -s models/language -m 'core_model and (not slow_test)'

2151 + - pip freeze | grep -E 'torch'

2152 + - pytest -v -s models/language -m 'core_model and (not slow_test)'

2153

2309 -- label: Language Models Tests (Extra Standard) %N

2310 - timeout_in_minutes: 45

2311 - mirror_hardwares: [amdexperimental, amdproduction]

2312 - agent_pool: mi325_1

2313 - optional: true

2314 - # grade: Blocking

2315 - torch_nightly: true

2316 - source_file_dependencies:

2317 - - vllm/model_executor/models/

2318 - - tests/models/language/pooling/test_embedding.py

2319 - - tests/models/language/generation/test_common.py

2320 - - tests/models/language/pooling/test_classification.py

2321 - commands:

2322 - # Shard slow subset of standard language models tests. Only run when model

2323 - # source is modified, or when specified test files are modified

2324 - - pip freeze | grep -E 'torch'

2325 - - export TORCH_NCCL_BLOCKING_WAIT=1

2326 - - pytest -v -s models/language -m 'core_model and slow_test' \

2327 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

2328 - --shard-id=$$BUILDKITE_PARALLEL_JOB

2329 - parallelism: 2

2154

2331 -- label: Language Models Tests (Hybrid) %N

2332 - timeout_in_minutes: 75

2333 - mirror_hardwares: [amdexperimental, amdproduction]

2155 +- label: Language Models Tests (Hybrid) %N # 34.9m

2156 + timeout_in_minutes: 55

2157 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2158 agent_pool: mi325_1

2335 - optional: true

2336 - # grade: Blocking

2159 torch_nightly: true

2160 + parallelism: 2

2161 + working_dir: "/vllm-workspace/tests"

2162 source_file_dependencies:

2163 - vllm/

2164 - tests/models/language/generation

2165 commands:

2342 - # Install fast path packages for testing against transformers

2343 - # Note: also needed to run plamo2 model in vLLM

2344 - - uv pip install --system --no-build-isolation 'git+https://github.com/AndreasKaratzas/mamba@fix-rocm-7.0-warp-size-constexpr'

2345 - - uv pip install --system --no-build-isolation 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.2'

2346 - # Shard hybrid language model tests

2347 - - pytest -v -s models/language/generation \

2348 - -m hybrid_model \

2349 - --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT \

2350 - --shard-id=$$BUILDKITE_PARALLEL_JOB

2351 - parallelism: 2

2166 + - uv pip install --system --no-build-isolation 'git+https://github.com/AndreasKaratzas/mamba@fix-rocm-7.0-warp-size-constexpr'

2167 + - uv pip install --system --no-build-isolation 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.2'

2168 + - pytest -v -s models/language/generation -m hybrid_model --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB

2169

2353 -- label: Language Models Test (Extended Generation) # 80min

2354 - timeout_in_minutes: 110

2355 - mirror_hardwares: [amdexperimental, amdproduction]

2170 +

2171 +- label: Language Models Test (Extended Generation) # 32.2m

2172 + timeout_in_minutes: 55

2173 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2174 agent_pool: mi325_1

2357 - # grade: Blocking

2358 - optional: true

2175 + working_dir: "/vllm-workspace/tests"

2176 source_file_dependencies:

2177 - vllm/

2178 - tests/models/language/generation

2179 commands:

2363 - # Install fast path packages for testing against transformers

2364 - # Note: also needed to run plamo2 model in vLLM

2365 - - uv pip install --system --no-build-isolation 'git+https://github.com/AndreasKaratzas/mamba@fix-rocm-7.0-warp-size-constexpr'

2366 - - uv pip install --system --no-build-isolation 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.2'

2367 - - pytest -v -s models/language/generation -m '(not core_model) and (not hybrid_model)'

2180 + - uv pip install --system --no-build-isolation 'git+https://github.com/AndreasKaratzas/mamba@fix-rocm-7.0-warp-size-constexpr'

2181 + - uv pip install --system --no-build-isolation 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.2'

2182 + - pytest -v -s models/language/generation -m '(not core_model) and (not hybrid_model)'

2183

2369 -- label: Language Models Test (PPL)

2370 - timeout_in_minutes: 110

2371 - mirror_hardwares: [amdexperimental, amdproduction]

2372 - agent_pool: mi325_1

2373 - # grade: Blocking

2374 - optional: true

2375 - source_file_dependencies:

2376 - - vllm/

2377 - - tests/models/language/generation_ppl_test

2378 - commands:

2379 - - pytest -v -s models/language/generation_ppl_test

2184

2381 -- label: Language Models Test (Extended Pooling) # 36min

2382 - timeout_in_minutes: 50

2383 - mirror_hardwares: [amdexperimental, amdproduction]

2185 +- label: Multi-Modal Processor # 1h 42m

2186 + timeout_in_minutes: 138

2187 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2188 agent_pool: mi325_1

2385 - # grade: Blocking

2189 optional: true

2190 + working_dir: "/vllm-workspace/tests"

2191 source_file_dependencies:

2192 - vllm/

2389 - - tests/models/language/pooling

2193 + - tests/models/multimodal

2194 + - tests/models/registry.py

2195 commands:

2391 - - pytest -v -s models/language/pooling -m 'not core_model'

2196 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2197 + - pytest -v -s models/multimodal/processing/test_tensor_schema.py

2198

2393 -- label: Language Models Test (MTEB)

2394 - timeout_in_minutes: 110

2395 - mirror_hardwares: [amdexperimental, amdproduction]

2396 - agent_pool: mi325_1

2397 - # grade: Blocking

2398 - optional: true

2399 - source_file_dependencies:

2400 - - vllm/

2401 - - tests/models/language/pooling_mteb_test

2402 - commands:

2403 - - pytest -v -s models/language/pooling_mteb_test

2199

2405 -- label: Multi-Modal Processor Test (CPU)

2406 - timeout_in_minutes: 60

2407 - mirror_hardwares: [amdexperimental, amdproduction]

2200 +- label: "Multi-Modal Models (Standard) 1: qwen2" # TBD

2201 + timeout_in_minutes: 180

2202 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2203 agent_pool: mi325_1

2409 - optional: true

2204 + torch_nightly: true

2205 + working_dir: "/vllm-workspace/tests"

2206 source_file_dependencies:

2207 - vllm/

2208 - tests/models/multimodal

2413 - - tests/models/registry.py

2414 - no_gpu: true

2209 commands:

2416 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2417 - - pytest -v -s models/multimodal/processing --ignore models/multimodal/processing/test_tensor_schema.py

2210 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2211 + - pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen2"

2212 + - pytest -v -s models/multimodal/generation/test_ultravox.py -m core_model

2213

2419 -- label: Multi-Modal Processor Test # 44min

2420 - timeout_in_minutes: 60

2421 - mirror_hardwares: [amdexperimental, amdproduction]

2214 +

2215 +- label: "Multi-Modal Models (Standard) 2: qwen3 + gemma" # TBD

2216 + timeout_in_minutes: 180

2217 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2218 agent_pool: mi325_1

2423 - optional: true

2424 - # grade: Blocking

2219 + torch_nightly: true

2220 + working_dir: "/vllm-workspace/tests"

2221 source_file_dependencies:

2222 - vllm/

2223 - tests/models/multimodal

2428 - - tests/models/registry.py

2224 commands:

2430 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2431 - - pytest -v -s models/multimodal/processing

2225 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2226 + - pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen3 or gemma"

2227 + - pytest -v -s models/multimodal/generation/test_qwen2_5_vl.py -m core_model

2228 +

2229

2433 -- label: Multi-Modal Models Test (Standard) # 60min

2434 - timeout_in_minutes: 100

2435 - mirror_hardwares: [amdexperimental, amdproduction]

2230 +- label: "Multi-Modal Models (Standard) 3: llava + qwen2_vl" # TBD

2231 + timeout_in_minutes: 180

2232 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2233 agent_pool: mi325_1

2437 - optional: true

2438 - # grade: Blocking

2234 torch_nightly: true

2235 + working_dir: "/vllm-workspace/tests"

2236 source_file_dependencies:

2237 - vllm/

2442 - - tests/models/multimodal

2238 + - tests/models/multimodal/generation

2239 + - tests/models/multimodal/test_mapping.py

2240 commands:

2444 - - export MIOPEN_DEBUG_CONV_DIRECT=0

2445 - - export MIOPEN_DEBUG_CONV_GEMM=0

2446 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2447 - - pip freeze | grep -E 'torch'

2448 - - pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/processing --ignore models/multimodal/pooling/test_prithvi_mae.py

2449 - - pytest -v -s models/multimodal/pooling/test_prithvi_mae.py -m core_model

2450 - - cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model # Otherwise, mp_method="spawn" doesn't work

2241 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2242 + - pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma"

2243 + - pytest -v -s models/multimodal/generation/test_qwen2_vl.py -m core_model

2244

2452 -- label: Multi-Modal Accuracy Eval (Small Models) # 5min

2453 - timeout_in_minutes: 10

2454 - mirror_hardwares: [amdexperimental, amdproduction]

2245 +

2246 +- label: "Multi-Modal Models (Standard) 4: other + whisper" # TBD

2247 + timeout_in_minutes: 180

2248 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2249 agent_pool: mi325_1

2456 - optional: true

2457 - # grade: Blocking

2458 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

2250 + torch_nightly: true

2251 + working_dir: "/vllm-workspace/tests"

2252 source_file_dependencies:

2460 - - vllm/multimodal/

2461 - - vllm/inputs/

2462 - - vllm/v1/core/

2253 + - vllm/

2254 + - tests/models/multimodal/generation

2255 commands:

2464 - - export MIOPEN_DEBUG_CONV_DIRECT=0

2465 - - export MIOPEN_DEBUG_CONV_GEMM=0

2466 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-mm-small.txt

2256 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2257 + - pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/processing

2258 + - cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model

2259

2468 -- label: Multi-Modal Models Test (Extended 1) # 60min

2469 - timeout_in_minutes: 120

2470 - mirror_hardwares: [amdexperimental, amdproduction]

2260 +

2261 +- label: Multi-Modal Models (Extended Generation 1) # 1h 2m

2262 + timeout_in_minutes: 106

2263 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2264 agent_pool: mi325_1

2472 - # grade: Blocking

2473 - optional: true

2265 + working_dir: "/vllm-workspace/tests"

2266 source_file_dependencies:

2267 - vllm/

2268 - tests/models/multimodal/generation

2269 - tests/models/multimodal/test_mapping.py

2270 commands:

2479 - - export MIOPEN_DEBUG_CONV_DIRECT=0

2480 - - export MIOPEN_DEBUG_CONV_GEMM=0

2481 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2482 - - pytest -v -s models/multimodal/generation -m 'not core_model' --ignore models/multimodal/generation/test_common.py

2483 - - pytest -v -s models/multimodal/test_mapping.py

2271 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2272 + - pytest -v -s models/multimodal/generation -m 'not core_model' --ignore models/multimodal/generation/test_common.py

2273 + - pytest -v -s models/multimodal/test_mapping.py

2274 +

2275

2485 -- label: Multi-Modal Models Test (Extended 2) # 60min

2486 - timeout_in_minutes: 120

2487 - mirror_hardwares: [amdexperimental, amdproduction]

2276 +- label: Multi-Modal Models (Extended Generation 2) # TBD

2277 + timeout_in_minutes: 180

2278 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2279 agent_pool: mi325_1

2489 - # grade: Blocking

2490 - optional: true

2280 + working_dir: "/vllm-workspace/tests"

2281 source_file_dependencies:

2282 - vllm/

2283 - tests/models/multimodal/generation

2284 commands:

2495 - - export MIOPEN_DEBUG_CONV_DIRECT=0

2496 - - export MIOPEN_DEBUG_CONV_GEMM=0

2497 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2498 - - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=0) and not core_model'

2285 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2286 + - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=0) and not core_model'

2287

2500 -- label: Multi-Modal Models Test (Extended 3) # 75min

2501 - timeout_in_minutes: 150

2502 - mirror_hardwares: [amdexperimental, amdproduction]

2288 +

2289 +- label: Multi-Modal Models (Extended Generation 3) # TBD

2290 + timeout_in_minutes: 180

2291 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2292 agent_pool: mi325_1

2504 - # grade: Blocking

2505 - optional: true

2293 + working_dir: "/vllm-workspace/tests"

2294 source_file_dependencies:

2295 - vllm/

2296 - tests/models/multimodal/generation

2297 commands:

2510 - - export MIOPEN_DEBUG_CONV_DIRECT=0

2511 - - export MIOPEN_DEBUG_CONV_GEMM=0

2512 - - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2513 - - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model'

2298 + - pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

2299 + - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model'

2300 +

2301

2515 -- label: Multi-Modal Models Test (Extended Pooling) # 60min

2516 - timeout_in_minutes: 120

2517 - mirror_hardwares: [amdexperimental, amdproduction]

2302 +- label: Multi-Modal Models (Extended Pooling) # TBD

2303 + timeout_in_minutes: 180

2304 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2305 agent_pool: mi325_1

2519 - # grade: Blocking

2520 - optional: true

2306 + working_dir: "/vllm-workspace/tests"

2307 source_file_dependencies:

2308 - vllm/

2309 - tests/models/multimodal/pooling

2310 commands:

2525 - - export MIOPEN_DEBUG_CONV_DIRECT=0

2526 - - export MIOPEN_DEBUG_CONV_GEMM=0

2527 - - pytest -v -s models/multimodal/pooling -m 'not core_model'

2311 + - pytest -v -s models/multimodal/pooling -m 'not core_model'

2312

2529 -- label: Quantized Models Test # 45 min

2530 - timeout_in_minutes: 60

2531 - mirror_hardwares: [amdexperimental, amdproduction]

2313 +

2314 +- label: Quantized Models Test # 21.4m

2315 + timeout_in_minutes: 38

2316 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2317 agent_pool: mi325_1

2533 - # grade: Blocking

2318 + working_dir: "/vllm-workspace/tests"

2319 source_file_dependencies:

2320 - vllm/model_executor/layers/quantization

2321 + - vllm/_aiter_ops.py

2322 + - vllm/platforms/rocm.py

2323 - tests/models/quantization

2324 + - vllm/model_executor/model_loader/

2325 commands:

2538 - - pytest -v -s models/quantization

2326 + - pytest -v -s models/quantization

2327 +

2328

2540 -- label: Transformers Nightly Models Test

2541 - mirror_hardwares: [amdexperimental, amdproduction]

2329 +- label: Transformers Nightly Models # 50.9m

2330 + timeout_in_minutes: 102

2331 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2332 agent_pool: mi325_1

2543 - # grade: Blocking

2544 - working_dir: "/vllm-workspace/"

2333 optional: true

2546 - commands:

2547 - - pip install --upgrade git+https://github.com/huggingface/transformers

2548 - - pytest -v -s tests/models/test_initialization.py -k 'not (Gemma3 or ModernBert or Qwen2_5_VL or Qwen2_5vl or Qwen2VL or TransformersMultiModalEmbeddingModel or TransformersMultiModalForSequenceClassification or Ultravox or Phi4Multimodal or LlavaNextVideo or MiniCPMO or Lfm2Moe or PaliGemma or RobertaForSequenceClassification or Ovis2_5 or Fuyu or DeepseekOCR or KimiVL)'

2549 - - pytest -v -s tests/models/test_transformers.py

2550 - # - pytest -v -s tests/models/multimodal/processing/

2551 - - pytest -v -s tests/models/multimodal/test_mapping.py -k 'not (Gemma3 or Qwen2VL or Qwen2_5_VL)'

2552 - - python3 examples/basic/offline_inference/chat.py

2553 - # - python3 examples/offline_inference/vision_language.py --model-type qwen2_5_vl

2554 - # Whisper needs spawn method to avoid deadlock

2555 - - VLLM_WORKER_MULTIPROC_METHOD=spawn python3 examples/offline_inference/audio_language.py --model-type whisper

2556 -

2557 -- label: Blackwell Fusion and Compile Tests # 30 min

2558 - timeout_in_minutes: 40

2559 - working_dir: "/vllm-workspace/"

2560 - gpu: b200

2561 - source_file_dependencies:

2562 - - csrc/quantization/fp4/

2563 - - vllm/model_executor/layers/quantization/utils/flashinfer_utils.py

2564 - - vllm/v1/attention/backends/flashinfer.py

2565 - - vllm/v1/worker/

2566 - - vllm/v1/cudagraph_dispatcher.py

2567 - - vllm/compilation/

2568 - # can affect pattern matching

2569 - - vllm/model_executor/layers/layernorm.py

2570 - - vllm/model_executor/layers/activation.py

2571 - - vllm/model_executor/layers/quantization/input_quant_fp8.py

2572 - - tests/compile/passes/test_fusion_attn.py

2573 - - tests/compile/passes/test_silu_mul_quant_fusion.py

2574 - - tests/compile/passes/distributed/test_fusion_all_reduce.py

2575 - - tests/compile/fullgraph/test_full_graph.py

2576 - commands:

2577 - - nvidia-smi

2578 - - pytest -v -s tests/compile/passes/test_fusion_attn.py

2579 - - pytest -v -s tests/compile/passes/test_silu_mul_quant_fusion.py

2580 - # this runner has 2 GPUs available even though num_gpus=2 is not set

2581 - - pytest -v -s tests/compile/passes/distributed/test_fusion_all_reduce.py

2582 -

2583 - # # Limit to Inductor partition, no custom ops, and allreduce & attn fusion to reduce running time

2584 - # # Wrap with quotes to escape yaml

2585 - # - "pytest -v -s tests/compile/distributed/test_fusions_e2e.py::test_tp2_attn_quant_allreduce_rmsnorm -k 'True and not +quant_fp8 and not +rms_norm'"

2586 - # Old E2E tests were removed in https://github.com/vllm-project/vllm/pull/33293

2587 - # in favor of new tests in fusions_e2e. We avoid replicating the new jobs in this file as it's deprecated.

2588 -

2589 - # test_fp8_kv_scale_compile requires FlashAttention (not supported on default L4/L40)

2590 - - pytest -v -s tests/compile/fullgraph/test_full_graph.py::test_fp8_kv_scale_compile

2591 -

2592 -- label: Blackwell GPT-OSS Eval

2593 - timeout_in_minutes: 60

2334 working_dir: "/vllm-workspace/"

2595 - gpu: b200

2596 - optional: true # run on nightlies

2335 source_file_dependencies:

2598 - - tests/evals/gpt_oss

2599 - - vllm/model_executor/models/gpt_oss.py

2600 - - vllm/model_executor/layers/quantization/mxfp4.py

2601 - - vllm/v1/attention/backends/flashinfer.py

2336 + - vllm/model_executor/models/

2337 + - vllm/model_executor/model_loader/

2338 + - vllm/multimodal/

2339 + - vllm/model_executor/layers/

2340 + - vllm/v1/attention/backends/

2341 + - vllm/v1/attention/selector.py

2342 + - vllm/_aiter_ops.py

2343 + - vllm/platforms/rocm.py

2344 + - tests/models/

2345 + - examples/

2346 commands:

2603 - - uv pip install --system 'gpt-oss[eval]==0.0.5'

2604 - - pytest -s -v tests/evals/gpt_oss/test_gpqa_correctness.py --model openai/gpt-oss-20b --metric 0.58

2347 + - pip install --upgrade git+https://github.com/huggingface/transformers

2348 + - pytest -v -s tests/models/test_initialization.py

2349 + - pytest -v -s tests/models/test_transformers.py

2350 + - pytest -v -s tests/models/multimodal/processing/

2351 + - pytest -v -s tests/models/multimodal/test_mapping.py

2352 + - python3 examples/basic/offline_inference/chat.py

2353 + - python3 examples/offline_inference/vision_language.py --model-type qwen2_5_vl

2354 + - VLLM_WORKER_MULTIPROC_METHOD=spawn python3 examples/offline_inference/audio_language.py --model-type whisper

2355

2606 -- label: Blackwell Quantized MoE Test

2607 - timeout_in_minutes: 60

2356 +

2357 +- label: Quantized MoE Test (B200-MI325) # TBD

2358 + timeout_in_minutes: 180

2359 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2360 + agent_pool: mi325_1

2361 working_dir: "/vllm-workspace/"

2609 - gpu: b200

2362 source_file_dependencies:

2611 - - tests/quantization/test_blackwell_moe.py

2363 + - tests/quantization/test_gfx3xx_moe.py

2364 - vllm/model_executor/models/deepseek_v2.py

2365 - vllm/model_executor/models/gpt_oss.py

2366 - vllm/model_executor/models/llama4.py

@@ -2616,65 +2368,49 @@ steps:

2368 - vllm/model_executor/layers/quantization/compressed_tensors

2369 - vllm/model_executor/layers/quantization/modelopt.py

2370 - vllm/model_executor/layers/quantization/mxfp4.py

2619 - - vllm/v1/attention/backends/flashinfer.py

2371 + - vllm/v1/attention/backends/triton_attn.py

2372 + - vllm/v1/attention/backends/rocm_attn.py

2373 + - vllm/v1/attention/backends/rocm_aiter_fa.py

2374 + - vllm/v1/attention/backends/mla/

2375 + - vllm/v1/attention/selector.py

2376 + - vllm/model_executor/layers/layernorm.py

2377 + - vllm/_aiter_ops.py

2378 + - vllm/platforms/rocm.py

2379 + - vllm/model_executor/model_loader/

2380 commands:

2621 - - pytest -s -v tests/quantization/test_blackwell_moe.py

2381 + - pytest -s -v tests/quantization/test_gfx3xx_moe.py

2382

2623 -##### 1 GPU test #####

2624 -##### multi gpus test #####

2383

2626 -- label: Distributed Comm Ops Test # 7min

2627 - timeout_in_minutes: 20

2628 - mirror_hardwares: [amdexperimental, amdproduction]

2384 +- label: Distributed DP Tests (2 GPUs) # 56.1m

2385 + timeout_in_minutes: 102

2386 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2387 agent_pool: mi325_2

2630 - # grade: Blocking

2631 - working_dir: "/vllm-workspace/tests"

2388 num_gpus: 2

2633 - source_file_dependencies:

2634 - - vllm/distributed

2635 - - tests/distributed

2636 - commands:

2637 - - pytest -v -s distributed/test_comm_ops.py

2638 - - pytest -v -s distributed/test_shm_broadcast.py

2639 - - pytest -v -s distributed/test_shm_buffer.py

2640 - - pytest -v -s distributed/test_shm_storage.py

2641 -

2642 -- label: 2 Node Tests (4 GPUs in total) # 16min

2643 - timeout_in_minutes: 30

2644 - mirror_hardwares: [amdexperimental, amdproduction, amdmultinode]

2645 - agent_pool: mi325_4

2646 - optional: true

2647 - # grade: Blocking

2389 working_dir: "/vllm-workspace/tests"

2649 - num_gpus: 2

2650 - num_nodes: 2

2390 source_file_dependencies:

2391 - vllm/distributed/

2392 - vllm/engine/

2393 - vllm/executor/

2655 - - vllm/model_executor/models/

2656 - - tests/distributed/

2657 - - tests/examples/offline_inference/data_parallel.py

2394 + - vllm/worker/worker_base.py

2395 + - vllm/v1/engine/

2396 + - vllm/v1/worker/

2397 + - tests/v1/distributed

2398 + - tests/v1/entrypoints/openai/test_multi_api_servers.py

2399 + - vllm/platforms/rocm.py

2400 commands:

2659 - - # the following commands are for the first node, with ip 192.168.10.10 (ray environment already set up) | grep 'Same node test passed' | grep 'Node count test passed'

2660 - - VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py

2661 - - NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py

2662 - - python3 ../examples/offline_inference/data_parallel.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code

2663 - - VLLM_MULTI_NODE=1 pytest -v -s distributed/test_multi_node_assignment.py

2664 - - VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py

2665 - - # the following commands are for the second node, with ip 192.168.10.11 (ray environment already set up)

2666 - - VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py

2667 - - NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py

2668 - - python3 ../examples/offline_inference/data_parallel.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code

2669 -

2670 -- label: Distributed Tests (2 GPUs) # 68min

2671 - timeout_in_minutes: 90

2672 - mirror_hardwares: [amdexperimental, amdproduction]

2401 + - export TORCH_NCCL_BLOCKING_WAIT=1

2402 + - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_async_llm_dp.py

2403 + - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_eagle_dp.py

2404 + - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_external_lb_dp.py

2405 + - DP_SIZE=2 pytest -v -s v1/entrypoints/openai/test_multi_api_servers.py

2406 +

2407 +

2408 +- label: Distributed Compile + RPC Tests (2 GPUs) # 56.1m

2409 + timeout_in_minutes: 102

2410 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2411 agent_pool: mi325_2

2674 - optional: true

2675 - # grade: Blocking

2676 - working_dir: "/vllm-workspace/tests"

2412 num_gpus: 2

2413 + working_dir: "/vllm-workspace/tests"

2414 source_file_dependencies:

2415 - vllm/compilation/

2416 - vllm/distributed/

@@ -2685,381 +2421,446 @@ steps:

2421 - vllm/v1/worker/

2422 - tests/compile/fullgraph/test_basic_correctness.py

2423 - tests/compile/test_wrapper.py

2688 - - tests/distributed/

2424 - tests/entrypoints/llm/test_collective_rpc.py

2690 - - tests/v1/distributed

2691 - - tests/v1/entrypoints/openai/test_multi_api_servers.py

2692 - - tests/v1/shutdown

2693 - - tests/v1/worker/test_worker_memory_snapshot.py

2694 - - examples/rl/

2425 + - vllm/platforms/rocm.py

2426 commands:

2696 - # Work around HIP bug tracked here: https://github.com/ROCm/hip/issues/3876

2697 - # TODO: Remove when the bug is fixed in a future ROCm release

2427 - export TORCH_NCCL_BLOCKING_WAIT=1

2699 - - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_async_llm_dp.py

2700 - - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_eagle_dp.py

2701 - - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_external_lb_dp.py

2702 - - DP_SIZE=2 pytest -v -s v1/entrypoints/openai/test_multi_api_servers.py

2428 - pytest -v -s entrypoints/llm/test_collective_rpc.py

2429 - pytest -v -s ./compile/fullgraph/test_basic_correctness.py

2430 - pytest -v -s ./compile/test_wrapper.py

2431 +

2432 +

2433 +- label: Distributed Torchrun + Shutdown Tests (2 GPUs) # 56.1m

2434 + timeout_in_minutes: 102

2435 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2436 + agent_pool: mi325_2

2437 + num_gpus: 2

2438 + working_dir: "/vllm-workspace/tests"

2439 + source_file_dependencies:

2440 + - vllm/distributed/

2441 + - vllm/engine/

2442 + - vllm/executor/

2443 + - vllm/worker/worker_base.py

2444 + - vllm/v1/engine/

2445 + - vllm/v1/worker/

2446 + - tests/distributed/

2447 + - tests/v1/shutdown

2448 + - tests/v1/worker/test_worker_memory_snapshot.py

2449 + - vllm/platforms/rocm.py

2450 + commands:

2451 + - export TORCH_NCCL_BLOCKING_WAIT=1

2452 - VLLM_TEST_SAME_HOST=1 torchrun --nproc-per-node=4 distributed/test_same_node.py | grep 'Same node test passed'

2453 - VLLM_TEST_SAME_HOST=1 VLLM_TEST_WITH_DEFAULT_DEVICE_SET=1 torchrun --nproc-per-node=4 distributed/test_same_node.py | grep 'Same node test passed'

2454 - CUDA_VISIBLE_DEVICES=0,1 pytest -v -s v1/shutdown

2455 - pytest -v -s v1/worker/test_worker_memory_snapshot.py

2456

2711 -- label: Distributed Model Tests (2 GPUs) # 37min

2712 - timeout_in_minutes: 50

2713 - mirror_hardwares: [amdexperimental, amdproduction]

2457 +

2458 +- label: Distributed Model Tests (2 GPUs) # 19.3m

2459 + timeout_in_minutes: 38

2460 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2461 agent_pool: mi325_2

2462 + num_gpus: 2

2463 optional: true

2716 - # grade: Blocking

2464 working_dir: "/vllm-workspace/tests"

2718 - num_gpus: 2

2465 source_file_dependencies:

2466 - vllm/model_executor/model_loader/sharded_state_loader.py

2467 - vllm/model_executor/models/

2468 + - vllm/model_executor/layers/

2469 + - vllm/v1/attention/backends/

2470 + - vllm/v1/attention/selector.py

2471 + - vllm/_aiter_ops.py

2472 + - vllm/platforms/rocm.py

2473 - tests/basic_correctness/

2474 - tests/model_executor/model_loader/test_sharded_state_loader.py

2475 - tests/models/

2476 commands:

2477 - TARGET_TEST_SUITE=L4 pytest basic_correctness/ -v -s -m 'distributed(num_gpus=2)'

2478 - CUDA_VISIBLE_DEVICES=0,1 pytest -v -s model_executor/model_loader/test_sharded_state_loader.py

2728 - # Avoid importing model tests that cause CUDA reinitialization error

2479 - pytest models/test_transformers.py -v -s -m 'distributed(num_gpus=2)'

2480 - pytest models/language -v -s -m 'distributed(num_gpus=2)'

2481 - pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py

2482 - VLLM_WORKER_MULTIPROC_METHOD=spawn pytest models/multimodal/generation/test_whisper.py -v -s -m 'distributed(num_gpus=2)'

2483

2734 -- label: Plugin Tests (2 GPUs) # 40min

2735 - timeout_in_minutes: 60

2736 - mirror_hardwares: [amdexperimental, amdproduction]

2737 - agent_pool: mi325_2

2738 - optional: true

2739 - # grade: Blocking

2484 +

2485 +- label: LoRA TP (Distributed) # 9.8m

2486 + timeout_in_minutes: 18

2487 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2488 + agent_pool: mi325_4

2489 + num_gpus: 4

2490 working_dir: "/vllm-workspace/tests"

2491 + source_file_dependencies:

2492 + - vllm/lora

2493 + - tests/lora

2494 + - vllm/platforms/rocm.py

2495 + commands:

2496 + - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2497 + - export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

2498 + - pytest -v -s -x lora/test_chatglm3_tp.py

2499 + - pytest -v -s -x lora/test_llama_tp.py

2500 + - pytest -v -s -x lora/test_llm_with_multi_loras.py

2501 + - pytest -v -s -x lora/test_olmoe_tp.py

2502 + - pytest -v -s -x lora/test_gptoss_tp.py

2503 +

2504 +

2505 +- label: Weight Loading Multiple GPU # 7.5m

2506 + timeout_in_minutes: 14

2507 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2508 + agent_pool: mi325_2

2509 num_gpus: 2

2510 + working_dir: "/vllm-workspace/tests"

2511 source_file_dependencies:

2743 - - vllm/plugins/

2744 - - tests/plugins/

2512 + - vllm/

2513 + - tests/weight_loading

2514 commands:

2746 - # begin platform plugin and general plugin tests, all the code in-between runs on dummy platform

2747 - - pip install -e ./plugins/vllm_add_dummy_platform

2748 - - pytest -v -s plugins_tests/test_platform_plugins.py

2749 - - pip uninstall vllm_add_dummy_platform -y

2750 - # end platform plugin tests

2751 - # begin io_processor plugins test, all the code in between uses the prithvi_io_processor plugin

2752 - - pip install -e ./plugins/prithvi_io_processor_plugin

2753 - - pytest -v -s plugins_tests/test_io_processor_plugins.py

2754 - - pip uninstall prithvi_io_processor_plugin -y

2755 - # test bge_m3_sparse io_processor plugin

2756 - - pip install -e ./plugins/bge_m3_sparse_plugin

2757 - - pytest -v -s plugins_tests/test_bge_m3_sparse_io_processor_plugins.py

2758 - - pip uninstall bge_m3_sparse_plugin -y

2759 - # end io_processor plugins test

2760 - # begin stat_logger plugins test

2761 - - pip install -e ./plugins/vllm_add_dummy_stat_logger

2762 - - pytest -v -s plugins_tests/test_stats_logger_plugins.py

2763 - - pip uninstall dummy_stat_logger -y

2764 - # end stat_logger plugins test

2765 - # other tests continue here:

2766 - - pytest -v -s plugins_tests/test_scheduler_plugins.py

2767 - - pip install -e ./plugins/vllm_add_dummy_model

2768 - - pytest -v -s distributed/test_distributed_oot.py

2769 - - pytest -v -s entrypoints/openai/chat_completion/test_oot_registration.py # it needs a clean process

2770 - - pytest -v -s models/test_oot_registration.py # it needs a clean process

2771 - - pytest -v -s plugins/lora_resolvers # unit tests for in-tree lora resolver plugins

2772 -

2773 -- label: Pipeline + Context Parallelism Test # 45min

2774 - timeout_in_minutes: 60

2775 - mirror_hardwares: [amdexperimental, amdproduction]

2776 - agent_pool: mi325_4

2777 - optional: true

2778 - # grade: Blocking

2779 - working_dir: "/vllm-workspace/tests"

2780 - num_gpus: 4

2781 - source_file_dependencies:

2782 - - vllm/distributed/

2783 - - vllm/engine/

2784 - - vllm/executor/

2785 - - vllm/model_executor/models/

2786 - - tests/distributed/

2787 - commands:

2788 - - pytest -v -s distributed/test_pp_cudagraph.py

2789 - - pytest -v -s distributed/test_pipeline_parallel.py

2790 -

2791 -- label: LoRA TP Test (Distributed) # 17 min

2792 - timeout_in_minutes: 30

2793 - mirror_hardwares: [amdexperimental, amdproduction]

2794 - agent_pool: mi325_4

2795 - optional: true

2796 - # grade: Blocking

2797 - num_gpus: 4

2798 - source_file_dependencies:

2799 - - vllm/lora

2800 - - tests/lora

2801 - commands:

2802 - # FIXIT: find out which code initialize cuda before running the test

2803 - # before the fix, we need to use spawn to test it

2804 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2805 - # There is some Tensor Parallelism related processing logic in LoRA that

2806 - # requires multi-GPU testing for validation.

2807 - - pytest -v -s -x lora/test_chatglm3_tp.py

2808 - - pytest -v -s -x lora/test_llama_tp.py

2809 - - pytest -v -s -x lora/test_llm_with_multi_loras.py

2810 - - pytest -v -s -x lora/test_olmoe_tp.py

2515 + - bash weight_loading/run_model_weight_loading_test.sh -c weight_loading/models-amd.txt

2516

2812 - # Disabled for now because MXFP4 backend on non-cuda platform

2813 - # doesn't support LoRA yet

2814 - #- pytest -v -s -x lora/test_gptoss_tp.py

2517

2816 -

2817 -- label: Weight Loading Multiple GPU Test # 33min

2818 - timeout_in_minutes: 45

2819 - mirror_hardwares: [amdexperimental, amdproduction]

2518 +- label: Weight Loading Multiple GPU - Large Models # 12.6m

2519 + timeout_in_minutes: 26

2520 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2521 agent_pool: mi325_2

2821 - # grade: Blocking

2822 - working_dir: "/vllm-workspace/tests"

2522 num_gpus: 2

2523 optional: true

2825 - source_file_dependencies:

2826 - - vllm/

2827 - - tests/weight_loading

2828 - commands:

2829 - - bash weight_loading/run_model_weight_loading_test.sh -c weight_loading/models-amd.txt

2830 -

2831 -- label: Weight Loading Multiple GPU Test - Large Models # optional

2832 - mirror_hardwares: [amdexperimental, amdproduction]

2833 - agent_pool: mi325_2

2834 - # grade: Blocking

2524 working_dir: "/vllm-workspace/tests"

2836 - num_gpus: 2

2837 - optional: true

2525 source_file_dependencies:

2526 - vllm/

2527 - tests/weight_loading

2528 commands:

2842 - - bash weight_loading/run_model_weight_loading_test.sh -c weight_loading/models-large-amd.txt

2529 + - bash weight_loading/run_model_weight_loading_test.sh -c weight_loading/models-large-amd.txt

2530

2844 -- label: NixlConnector PD accuracy tests (Distributed) # 30min

2845 - mirror_hardwares: [amdexperimental, amdproduction]

2846 - agent_pool: mi325_4

2531 +

2532 +- label: Ray Dependency Compatibility Check # TBD

2533 + timeout_in_minutes: 180

2534 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2535 + agent_pool: mi325_1

2536 optional: true

2848 - # grade: Blocking

2849 - timeout_in_minutes: 30

2850 - working_dir: "/vllm-workspace/tests"

2851 - num_gpus: 4

2537 + working_dir: "/"

2538 source_file_dependencies:

2853 - - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

2854 - - tests/v1/kv_connector/nixl_integration/

2539 + - requirements/

2540 + - setup.py

2541 + - vllm/platforms/rocm.py

2542 commands:

2856 - - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

2857 - - ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

2543 + - bash /vllm-workspace/.buildkite/scripts/check-ray-compatibility.sh

2544 +

2545

2859 -- label: DP EP NixlConnector PD accuracy tests (Distributed) # 15min

2860 - mirror_hardwares: [amdexperimental, amdproduction]

2546 +- label: Distributed NixlConnector PD accuracy (4 GPUs) # 27.4m

2547 + timeout_in_minutes: 44

2548 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2549 agent_pool: mi325_4

2550 + num_gpus: 4

2551 optional: true

2863 - # grade: Blocking

2864 - timeout_in_minutes: 15

2552 working_dir: "/vllm-workspace/tests"

2866 - num_gpus: 4

2553 source_file_dependencies:

2868 - - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

2869 - - tests/v1/kv_connector/nixl_integration/

2554 + - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

2555 + - tests/v1/kv_connector/nixl_integration/

2556 + - vllm/platforms/rocm.py

2557 commands:

2871 - - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

2872 - - DP_EP=1 ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

2558 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

2559 + - ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

2560

2874 -- label: CrossLayer KV layout Distributed NixlConnector PD accuracy tests (4 GPUs)

2875 - mirror_hardwares: [amdexperimental, amdproduction]

2561 +

2562 +- label: CrossLayer KV layout Distributed NixlConnector PD accuracy tests (4 GPUs) # TBD

2563 + timeout_in_minutes: 180

2564 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2565 agent_pool: mi325_4

2877 - # grade: Blocking

2878 - timeout_in_minutes: 30

2566 + num_gpus: 4

2567 + optional: true

2568 working_dir: "/vllm-workspace/tests"

2880 - num_devices: 4

2569 source_file_dependencies:

2882 - - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

2883 - - tests/v1/kv_connector/nixl_integration/

2570 + - vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

2571 + - tests/v1/kv_connector/nixl_integration/

2572 + - vllm/platforms/rocm.py

2573 commands:

2885 - - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

2886 - - CROSS_LAYERS_BLOCKS=1 ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

2574 + - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt

2575 + - CROSS_LAYERS_BLOCKS=True ROCM_ATTN=1 bash v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh

2576

2888 -##### multi gpus test #####

2889 -##### A100 test #####

2577

2891 -- label: Distributed Tests (A100) # optional

2892 - mirror_hardwares: [amdexperimental, amdproduction]

2578 +- label: Distributed Tests (4 GPUs)(A100-MI325) # 20.9m

2579 + timeout_in_minutes: 37

2580 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2581 agent_pool: mi325_4

2894 - # grade: Blocking

2895 - gpu: a100

2896 - optional: true

2582 num_gpus: 4

2583 + working_dir: "/vllm-workspace/tests"

2584 source_file_dependencies:

2585 - vllm/

2586 commands:

2901 - # Work around HIP bug tracked here: https://github.com/ROCm/hip/issues/3876

2902 - # TODO: Remove when the bug is fixed in a future ROCm release

2587 - export TORCH_NCCL_BLOCKING_WAIT=1

2904 - # NOTE: don't test llama model here, it seems hf implementation is buggy

2905 - # see https://github.com/vllm-project/vllm/pull/5689 for details

2588 - pytest -v -s distributed/test_custom_all_reduce.py

2589 - torchrun --nproc_per_node=2 distributed/test_ca_buffer_sharing.py

2590 - TARGET_TEST_SUITE=A100 pytest basic_correctness/ -v -s -m 'distributed(num_gpus=2)'

2591 - pytest -v -s -x lora/test_mixtral.py

2592

2593

2912 -- label: LM Eval Large Models # optional

2913 - gpu: a100

2594 +- label: Distributed Tests (2 GPUs)(H100-MI325) # TBD

2595 + timeout_in_minutes: 180

2596 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2597 + agent_pool: mi325_2

2598 + num_gpus: 2

2599 optional: true

2915 - mirror_hardwares: [amdexperimental, amdproduction]

2916 - agent_pool: mi325_4

2917 - # grade: Blocking

2918 - num_gpus: 4

2919 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

2600 + working_dir: "/vllm-workspace/"

2601 source_file_dependencies:

2921 - - csrc/

2922 - - vllm/model_executor/layers/quantization

2602 + - vllm/distributed/

2603 + - vllm/v1/distributed/

2604 + - vllm/model_executor/layers/fused_moe/

2605 + - tests/distributed/test_context_parallel.py

2606 + - tests/v1/distributed/test_dbo.py

2607 + - examples/offline_inference/data_parallel.py

2608 + - vllm/_aiter_ops.py

2609 + - vllm/platforms/rocm.py

2610 commands:

2924 - - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2925 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4

2611 + - export TORCH_NCCL_BLOCKING_WAIT=1

2612 + - pytest -v -s tests/distributed/test_context_parallel.py

2613 + - VLLM_LOGGING_LEVEL=DEBUG python3 examples/offline_inference/data_parallel.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=deepep_high_throughput

2614 + - pytest -v -s tests/v1/distributed/test_dbo.py

2615

2927 -##### FP8 test #####

2928 -- label: LM Eval Large Models (H100) # optional, still use H100 for consistency

2929 - gpu: h100

2930 - optional: true

2931 - mirror_hardwares: [amdexperimental, amdproduction]

2932 - agent_pool: mi325_4

2933 - # grade: Blocking

2934 - num_gpus: 4

2935 - working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

2616 +

2617 +- label: Distributed Compile Unit Tests (2xH100-2xMI325) # 14.3m

2618 + timeout_in_minutes: 32

2619 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2620 + agent_pool: mi325_2

2621 + num_gpus: 2

2622 + working_dir: "/vllm-workspace/"

2623 + source_file_dependencies:

2624 + - vllm/compilation/

2625 + - vllm/model_executor/layers

2626 + - tests/compile/passes/distributed/

2627 + - vllm/_aiter_ops.py

2628 + - vllm/platforms/rocm.py

2629 + commands:

2630 + - export VLLM_TEST_CLEAN_GPU_MEMORY=1

2631 + - VLLM_TEST_CLEAN_GPU_MEMORY=1 pytest -v -s tests/compile/passes/distributed/test_async_tp.py

2632 + - pytest -v -s tests/compile/passes/distributed/test_sequence_parallelism.py

2633 + # TODO: this test is not supported on ROCm, there are aiter kernels for this.

2634 + # - pytest -v -s tests/compile/passes/distributed/test_fusion_all_reduce.py

2635 + # - pytest -v -s tests/compile/distributed/test_fusions_e2e.py::test_tp2_attn_quant_allreduce_rmsnorm

2636 + # - "VLLM_TEST_CLEAN_GPU_MEMORY=1 pytest -v -s tests/compile/distributed/test_fusions_e2e.py -k 'not Llama-4'"

2637 +

2638 +

2639 +- label: LM Eval Small Models # 13.3m

2640 + timeout_in_minutes: 23

2641 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2642 + agent_pool: mi325_1

2643 + working_dir: "/vllm-workspace/tests"

2644 source_file_dependencies:

2645 - csrc/

2646 - vllm/model_executor/layers/quantization

2647 + - vllm/model_executor/models/

2648 + - vllm/model_executor/model_loader/

2649 + - vllm/v1/attention/backends/

2650 + - vllm/v1/attention/selector.py

2651 + - vllm/_aiter_ops.py

2652 + - vllm/platforms/rocm.py

2653 commands:

2940 - - export VLLM_USE_DEEP_GEMM=0

2941 - - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=4

2654 + - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt

2655

2656

2944 -##### H200 test #####

2945 -- label: Distributed Tests (H200) # optional

2946 - mirror_hardwares: [amdexperimental, amdproduction]

2657 +- label: LM Eval Small Models (B200-MI325) # TBD

2658 + timeout_in_minutes: 180

2659 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2660 agent_pool: mi325_2

2948 - # grade: Blocking

2949 - gpu: h200

2950 - optional: true

2951 - working_dir: "/vllm-workspace/"

2952 - num_gpus: 2

2661 + working_dir: "/vllm-workspace/tests"

2662 + source_file_dependencies:

2663 + - csrc/

2664 + - vllm/model_executor/layers/quantization

2665 + - vllm/model_executor/models/

2666 + - vllm/model_executor/model_loader/

2667 + - vllm/v1/attention/backends/

2668 + - vllm/v1/attention/selector.py

2669 + - vllm/_aiter_ops.py

2670 + - vllm/platforms/rocm.py

2671 commands:

2954 - - VLLM_TEST_CLEAN_GPU_MEMORY=1 pytest -v -s tests/compile/passes/distributed/test_async_tp.py

2955 - - pytest -v -s tests/compile/passes/distributed/test_sequence_parallelism.py

2956 - # TODO: this test is not supported on ROCm, there are aiter kernels for this.

2957 - # - pytest -v -s tests/compile/passes/distributed/test_fusion_all_reduce.py

2958 - #- pytest -v -s tests/compile/distributed/test_fusions_e2e.py::test_tp2_attn_quant_allreduce_rmsnorm

2959 - # - "VLLM_TEST_CLEAN_GPU_MEMORY=1 pytest -v -s tests/compile/distributed/test_fusions_e2e.py -k 'not Llama-4'"

2960 - # Old E2E tests were removed in https://github.com/vllm-project/vllm/pull/33293

2961 - # in favor of new tests in fusions_e2e. We avoid replicating the new jobs in this file as it's deprecated.

2962 - - pytest -v -s tests/distributed/test_context_parallel.py

2963 - - HIP_VISIBLE_DEVICES=0,1 VLLM_LOGGING_LEVEL=DEBUG python3 examples/offline_inference/data_parallel.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=allgather_reducescatter --disable-nccl-for-dp-synchronization

2964 - # this test is not supported on ROCm

2965 - # - pytest -v -s tests/v1/distributed/test_dbo.py

2966 -

2967 -##### B200 test #####

2968 -- label: Distributed Tests (B200) # optional

2969 - gpu: b200

2672 + - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx-fp8.txt

2673 +

2674 +

2675 +- label: LM Eval Large Models (H200-MI325) # TBD

2676 + timeout_in_minutes: 180

2677 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2678 + agent_pool: mi325_8

2679 optional: true

2971 - working_dir: "/vllm-workspace/"

2972 - num_gpus: 2

2680 + num_gpus: 8

2681 + working_dir: "/vllm-workspace/tests"

2682 + source_file_dependencies:

2683 + - vllm/model_executor/models/

2684 + - vllm/model_executor/model_loader/

2685 + - vllm/model_executor/layers/quantization/

2686 + - vllm/v1/attention/backends/

2687 + - vllm/v1/attention/selector.py

2688 + - vllm/model_executor/layers/layernorm.py

2689 + - csrc/

2690 + - vllm/_aiter_ops.py

2691 + - vllm/platforms/rocm.py

2692 + - tests/evals/

2693 commands:

2974 - - pytest -v -s tests/distributed/test_context_parallel.py

2975 - - pytest -v -s tests/distributed/test_nccl_symm_mem_allreduce.py

2976 - - pytest -v -s tests/v1/distributed/test_dbo.py

2694 + - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2695 + - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt

2696

2978 -##### E2E Eval Tests #####

2979 -- label: LM Eval Small Models (1 Card) # 15min

2980 - timeout_in_minutes: 20

2981 - mirror_hardwares: [amdexperimental, amdproduction]

2982 - agent_pool: mi325_1

2983 - # grade: Blocking

2697 +

2698 +- label: LM Eval Large Models (4 GPUs)(FP8) # 24.8m

2699 + timeout_in_minutes: 42

2700 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2701 + agent_pool: mi325_4

2702 + num_gpus: 4

2703 + working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

2704 source_file_dependencies:

2705 - csrc/

2706 - vllm/model_executor/layers/quantization

2707 + - vllm/model_executor/models/

2708 + - vllm/model_executor/model_loader/

2709 + - vllm/v1/attention/backends/

2710 + - vllm/v1/attention/selector.py

2711 + - vllm/_aiter_ops.py

2712 + - vllm/platforms/rocm.py

2713 commands:

2988 - - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt

2714 + - export VLLM_USE_DEEP_GEMM=0

2715 + - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm-fp8.txt --tp-size=4

2716

2990 -- label: LM Eval Large Models (4 Card)

2991 - mirror_hardwares: [amdexperimental, amdproduction]

2717 +

2718 +- label: LM Eval Large Models (4 GPUs)(A100-MI325) # 17.3m

2719 + timeout_in_minutes: 27

2720 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2721 agent_pool: mi325_4

2993 - # grade: Blocking

2994 - gpu: a100

2995 - optional: true

2722 num_gpus: 4

2723 working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

2724 source_file_dependencies:

2725 - csrc/

2726 - vllm/model_executor/layers/quantization

2727 + - vllm/model_executor/models/

2728 + - vllm/model_executor/model_loader/

2729 + - vllm/v1/attention/backends/

2730 + - vllm/v1/attention/selector.py

2731 + - vllm/_aiter_ops.py

2732 + - vllm/platforms/rocm.py

2733 commands:

2734 - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2735 - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4

2736

3005 -- label: ROCm LM Eval Large Models (8 Card)

3006 - mirror_hardwares: [amdexperimental, amdproduction]

2737 +

2738 +- label: ROCm LM Eval Large Models (8 Card) # TBD

2739 + timeout_in_minutes: 180

2740 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2741 agent_pool: mi325_8

2742 optional: true

2743 num_gpus: 8

2744 working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"

2745 + source_file_dependencies:

2746 + - vllm/model_executor/models/

2747 + - vllm/model_executor/model_loader/

2748 + - vllm/model_executor/layers/quantization/

2749 + - vllm/v1/attention/backends/

2750 + - vllm/v1/attention/selector.py

2751 + - vllm/model_executor/layers/layernorm.py

2752 + - csrc/

2753 + - vllm/_aiter_ops.py

2754 + - vllm/platforms/rocm.py

2755 commands:

2756 - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2757 - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8

2758

3015 -- label: ROCm GPT-OSS Eval

3016 - timeout_in_minutes: 60

2759 +

2760 +- label: GPQA Eval (GPT-OSS) (H100-MI325) # TBD

2761 + timeout_in_minutes: 180

2762 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2763 + agent_pool: mi325_2

2764 + num_gpus: 2

2765 + optional: true

2766 working_dir: "/vllm-workspace/tests"

3018 - agent_pool: mi325_1

3019 - mirror_hardwares: [amdexperimental, amdproduction]

3020 - optional: true # run on nightlies

2767 source_file_dependencies:

3022 - - tests/evals/gpt_oss

3023 - - vllm/model_executor/models/gpt_oss.py

3024 - - vllm/model_executor/layers/quantization/mxfp4.py

3025 - - vllm/v1/attention/backends/flashinfer.py

2768 + - csrc/

2769 + - vllm/model_executor/layers/quantization

2770 + - vllm/model_executor/models/

2771 + - vllm/model_executor/model_loader/

2772 + - vllm/v1/attention/backends/

2773 + - vllm/v1/attention/selector.py

2774 + - vllm/model_executor/layers/fused_moe/

2775 + - vllm/_aiter_ops.py

2776 + - vllm/platforms/rocm.py

2777 + - tests/evals/gpt_oss/

2778 commands:

3027 - - uv pip install --system 'gpt-oss[eval]==0.0.5'

3028 - - pytest -s -v evals/gpt_oss/test_gpqa_correctness.py --config-list-file=configs/models-gfx942.txt

2779 + - uv pip install --system 'gpt-oss[eval]==0.0.5'

2780 + - pytest -s -v evals/gpt_oss/test_gpqa_correctness.py --config-list-file=configs/models-gfx942.txt

2781

3030 -##### EPLB Accuracy Tests #####

3031 -- label: DeepSeek V2-Lite Accuracy

3032 - mirror_hardwares: [amdexperimental, amdproduction]

2782 +

2783 +- label: DeepSeek V2-Lite Accuracy # 6.7m

2784 + timeout_in_minutes: 12

2785 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2786 agent_pool: mi325_4

3034 - # grade: Blocking

3035 - timeout_in_minutes: 60

3036 - gpu: h100

3037 - optional: true

2787 num_gpus: 4

2788 working_dir: "/vllm-workspace"

2789 + source_file_dependencies:

2790 + - vllm/model_executor/models/

2791 + - vllm/model_executor/model_loader/

2792 + - vllm/distributed/eplb

2793 + - vllm/model_executor/layers/fused_moe/

2794 + - vllm/model_executor/layers/quantization/

2795 + - vllm/v1/attention/backends/

2796 + - vllm/v1/attention/backends/mla/

2797 + - vllm/v1/attention/selector.py

2798 + - .buildkite/scripts/scheduled_integration_test/

2799 + - vllm/_aiter_ops.py

2800 + - vllm/platforms/rocm.py

2801 commands:

2802 - bash .buildkite/scripts/scheduled_integration_test/deepseek_v2_lite_ep_eplb.sh 0.25 200 8010

2803

3043 -- label: Qwen3-30B-A3B-FP8-block Accuracy (H100)

3044 - mirror_hardwares: [amdexperimental, amdproduction]

2804 +

2805 +- label: DeepSeek V2-Lite Prefetch Offload Accuracy (H100-MI325) # TBD

2806 + timeout_in_minutes: 180

2807 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2808 + agent_pool: mi325_1

2809 + num_gpus: 1

2810 + working_dir: "/vllm-workspace"

2811 + source_file_dependencies:

2812 + - vllm/model_executor/models/

2813 + - vllm/model_executor/model_loader/

2814 + - vllm/model_executor/layers/fused_moe/

2815 + - vllm/model_executor/layers/quantization/

2816 + - vllm/v1/attention/backends/

2817 + - vllm/v1/attention/backends/mla/

2818 + - vllm/v1/attention/selector.py

2819 + - .buildkite/scripts/scheduled_integration_test/

2820 + - vllm/_aiter_ops.py

2821 + - vllm/platforms/rocm.py

2822 + commands:

2823 + - bash .buildkite/scripts/scheduled_integration_test/deepseek_v2_lite_prefetch_offload.sh 0.25 200 8030

2824 +

2825 +

2826 +- label: Qwen3-30B-A3B-FP8-block Accuracy # 6.4m

2827 + timeout_in_minutes: 11

2828 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2829 agent_pool: mi325_4

3046 - # grade: Blocking

3047 - timeout_in_minutes: 60

3048 - gpu: h100

3049 - optional: true

3050 - num_gpus: 4

2830 working_dir: "/vllm-workspace"

2831 + source_file_dependencies:

2832 + - vllm/model_executor/models/

2833 + - vllm/model_executor/model_loader/

2834 + - vllm/model_executor/layers/quantization/

2835 + - vllm/distributed/eplb

2836 + - vllm/model_executor/layers/fused_moe/

2837 + - vllm/v1/attention/backends/

2838 + - vllm/v1/attention/selector.py

2839 + - .buildkite/scripts/scheduled_integration_test/

2840 + - vllm/_aiter_ops.py

2841 + - vllm/platforms/rocm.py

2842 commands:

2843 - bash .buildkite/scripts/scheduled_integration_test/qwen30b_a3b_fp8_block_ep_eplb.sh 0.8 200 8020

2844

3055 -- label: Qwen3-Next-80B-A3B-Instruct MTP Async EPLB Accuracy

3056 - timeout_in_minutes: 60

3057 - mirror_hardwares: [amdexperimental, amdproduction]

2845 +

2846 +- label: Qwen3-Next-80B-A3B-Instruct MTP Async EPLB Accuracy # 10.9m

2847 + timeout_in_minutes: 22

2848 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2849 agent_pool: mi325_4

3059 - # grade: Blocking

3060 - optional: true

2850 num_gpus: 4

2851 working_dir: "/vllm-workspace"

2852 + source_file_dependencies:

2853 + - vllm/model_executor/models/

2854 + - vllm/model_executor/model_loader/

2855 + - vllm/v1/spec_decode/

2856 + - vllm/distributed/eplb

2857 + - vllm/model_executor/layers/fused_moe/

2858 + - vllm/model_executor/layers/quantization/

2859 + - vllm/v1/attention/backends/

2860 + - vllm/v1/attention/selector.py

2861 + - .buildkite/scripts/scheduled_integration_test/

2862 + - vllm/_aiter_ops.py

2863 + - vllm/platforms/rocm.py

2864 commands:

2865 - bash .buildkite/scripts/scheduled_integration_test/qwen3_next_mtp_async_eplb.sh 0.8 1319 8040

2866

@@ -3072,11 +2873,12 @@ steps:

2873

2874 ## TODO: Enable the test in this group

2875 # # corresponds to .buildkite/test_areas/compile.yaml

3075 -# - label: Fusion and Compile Unit Tests (2xMI325 GPUs)

3076 -# timeout_in_minutes: 20

3077 -# working_dir: "/vllm-workspace/"

3078 -# mirror_hardwares: [amdexperimental, amdproduction, tj]

2876 +# - label: Fusion and Compile Unit Tests (2xB200-2xMI325) # TBD

2877 +# timeout_in_minutes: 180

2878 +# mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325, tj]

2879 # agent_pool: mi325_1 # changed to 1 GPU until the fusion all reduce is enabled then only revert back to 2 GPUs

2880 +# num_gpus: 1

2881 +# working_dir: "/vllm-workspace/"

2882 # source_file_dependencies:

2883 # - csrc/quantization/fp4/

2884 # - vllm/model_executor/layers/quantization/

@@ -3100,1521 +2902,565 @@ steps:

2902 # # TODO: find out more details

2903 # # - pytest -v -s tests/compile/fullgraph/test_full_graph.py::test_fp8_kv_scale_compile

2904

3103 -# corresponds to .buildkite/test_areas/compile.yaml

3104 -- label: Fusion E2E Quick (MI325)

3105 - timeout_in_minutes: 15

3106 - working_dir: "/vllm-workspace/"

3107 - mirror_hardwares: [amdexperimental, amdproduction]

2905 +

2906 +- label: Fusion E2E Quick (H100-MI325) # TBD

2907 + timeout_in_minutes: 180

2908 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2909 agent_pool: mi325_1

3109 - num_devices: 1

3110 - source_file_dependencies:

3111 - - csrc/quantization/

3112 - - vllm/model_executor/

3113 - - vllm/v1/attention/

3114 - - vllm/compilation/

3115 - - tests/compile/fusions_e2e/

3116 - commands:

3117 - - rocm-smi

3118 - # Run all models and attn backends but only Inductor partition and native custom ops

3119 - - "pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k 'inductor_partition and not +rms_norm and not +quant_fp8'"

3120 - # Different from CUDA, Qwen requires +rms_norm and +quant_fp8 as rms+quant fusion is only supported on AITER

3121 - - "pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k 'inductor_partition and +rms_norm and +quant_fp8 and qwen3'"

3122 -

3123 -# corresponds to .buildkite/test_areas/compile.yaml

3124 -- label: Fusion E2E Config Sweep (MI325)

3125 - timeout_in_minutes: 30

2910 + num_gpus: 1

2911 working_dir: "/vllm-workspace/"

3127 - mirror_hardwares: [amdexperimental, amdproduction]

2912 + source_file_dependencies:

2913 + - csrc/quantization/

2914 + - vllm/model_executor/

2915 + - vllm/v1/attention/

2916 + - vllm/compilation/

2917 + - tests/compile/fusions_e2e/

2918 + - vllm/_aiter_ops.py

2919 + - vllm/platforms/rocm.py

2920 + commands:

2921 + - rocm-smi

2922 + # Run all models and attn backends but only Inductor partition and native custom ops

2923 + - "pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k 'inductor_partition and not +rms_norm and not +quant_fp8'"

2924 + # Different from CUDA, Qwen requires +rms_norm and +quant_fp8 as rms+quant fusion is only supported on AITER

2925 + - "pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k 'inductor_partition and +rms_norm and +quant_fp8 and qwen3'"

2926 +

2927 +

2928 +- label: Fusion E2E Config Sweep (H100-MI325) # TBD

2929 + timeout_in_minutes: 180

2930 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi325]

2931 agent_pool: mi325_1

3129 - num_devices: 1

2932 + num_gpus: 1

2933 + working_dir: "/vllm-workspace/"

2934 source_file_dependencies:

3131 - - csrc/quantization/

3132 - - vllm/compilation/

3133 - # can affect pattern matching

3134 - - vllm/model_executor/layers/layernorm.py

3135 - - vllm/model_executor/layers/activation.py

3136 - - vllm/model_executor/layers/attention/attention.py

3137 - - vllm/model_executor/layers/quantization/input_quant_fp8.py

3138 - - tests/compile/fusions_e2e/

2935 + - csrc/quantization/

2936 + - vllm/compilation/

2937 + - vllm/model_executor/layers/layernorm.py

2938 + - vllm/model_executor/layers/activation.py

2939 + - vllm/model_executor/layers/attention/attention.py

2940 + - vllm/model_executor/layers/quantization/input_quant_fp8.py

2941 + - tests/compile/fusions_e2e/

2942 + - vllm/_aiter_ops.py

2943 + - vllm/platforms/rocm.py

2944 commands:

3140 - - rocm-smi

3141 - # Run just llama3 (fp8) for all config combinations

3142 - - pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k "llama-3"

2945 + - rocm-smi

2946 + - pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k "llama-3"

2947

2948 ## There are no ops on ROCm for these tests.

2949 ## The test still passes but the logs are not useful.

2950 ## fused ops just call torch.ops.symm_mem which

2951 ## exists in ROCm even though they don't work

3148 -# - label: AsyncTP Correctness Tests (2xMI325 GPUs)

3149 -# - label: Fusion E2E TP2 Quick (MI325)

3150 -# - label: Fusion E2E TP2 AsyncTP Config Sweep (MI325)

3151 -# - label: Fusion E2E TP2 (MI325)

3152 -# - label: Sequence Parallel Correctness Tests (2xMI325 GPUs)

2952 +# - label: AsyncTP Correctness Tests (2xH100-2xMI325)

2953 +# - label: Fusion E2E TP2 Quick (H100-MI325)

2954 +# - label: Fusion E2E TP2 AsyncTP Config Sweep (H100-MI325)

2955 +# - label: Fusion E2E TP2 (B200-MI325)

2956 +# - label: Sequence Parallel Correctness Tests (2xH100-2xMI325)

2957

2958

2959 #####################################################################################################################################

2960 # #

3157 -# MI355 test definitions ( currently the test set is completely mirrored // TBD which tests are to be routed there ultimately) #

2961 +# gfx950 #

2962 # #

2963 #####################################################################################################################################

2964

3161 -- label: Pytorch Nightly Dependency Override Check # 2min

3162 - # if this test fails, it means the nightly torch version is not compatible with some

3163 - # of the dependencies. Please check the error message and add the package to whitelist

3164 - # in /vllm/tools/pre_commit/generate_nightly_torch_test.py

3165 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

2965 +- label: Entrypoints Integration (API Server 1) # TBD

2966 + timeout_in_minutes: 180

2967 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]

2968 agent_pool: mi355_1

3167 - optional: true

3168 - soft_fail: true

2969 + fast_check: true

2970 + torch_nightly: true

2971 + working_dir: "/vllm-workspace/tests"

2972 source_file_dependencies:

3170 - - requirements/nightly_torch_test.txt

2973 + - vllm/

2974 + - tests/entrypoints/openai

2975 + - tests/entrypoints/test_chat_utils

2976 commands:

3172 - - bash standalone_tests/pytorch_nightly_dependency.sh

2977 + - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2978 + - pytest -v -s entrypoints/openai --ignore=entrypoints/openai/chat_completion/test_chat_with_tool_reasoning.py --ignore=entrypoints/openai/chat_completion/test_oot_registration.py --ignore=entrypoints/openai/completion/test_tensorizer_entrypoint.py --ignore=entrypoints/openai/correctness/ --ignore=entrypoints/openai/tool_parsers/ --ignore=entrypoints/openai/responses

2979 + - pytest -v -s entrypoints/test_chat_utils.py

2980

3174 -- label: Async Engine, Inputs, Utils, Worker Test # 10min

3175 - timeout_in_minutes: 15

3176 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

2981 +

2982 +- label: Entrypoints Integration (API Server 2) # TBD

2983 + timeout_in_minutes: 180

2984 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]

2985 agent_pool: mi355_1

2986 optional: true

2987 + fast_check: true

2988 + torch_nightly: true

2989 + working_dir: "/vllm-workspace/tests"

2990 source_file_dependencies:

2991 - vllm/

3181 - - tests/multimodal

3182 - - tests/utils_

2992 + - tests/entrypoints/rpc

2993 + - tests/entrypoints/instrumentator

2994 + - tests/tool_use

2995 commands:

3184 - - pytest -v -s -m 'not cpu_test' multimodal

3185 - - pytest -v -s utils_

2996 + - export VLLM_WORKER_MULTIPROC_METHOD=spawn

2997 + - pytest -v -s entrypoints/instrumentator

2998 + - PYTHONPATH=/vllm-workspace pytest -v -s entrypoints/rpc

2999 + - pytest -v -s tool_use

3000 +

3001 +

3002 +- label: Entrypoints Integration (Pooling) # TBD

3003 + timeout_in_minutes: 180

3004 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]

3005 + agent_pool: mi355_1

3006 + fast_check: true

3007 + torch_nightly: true

3008 + working_dir: "/vllm-workspace/tests"

3009 + source_file_dependencies:

3010 + - vllm/

3011 + - tests/entrypoints/pooling

3012 + commands:

3013 + - export VLLM_WORKER_MULTIPROC_METHOD=spawn

3014 + - pytest -v -s entrypoints/pooling

3015 +

3016

3187 -- label: Async Engine, Inputs, Utils, Worker, Config Test (CPU) # 20min

3188 - timeout_in_minutes: 30

3189 - mirror_hardwares: [amdexperimental, amdproduction, amdtentative]

3017 +- label: Regression # TBD

3018 + timeout_in_minutes: 180

3019 + mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]

3020 agent_pool: mi355_1

3021 optional: true

3022 + working_dir: "/vllm-workspace/tests"

3023 source_file_dependencies:

3024 - vllm/

3194 - - tests/test_inputs.py

3195 - - tests/test_outputs.py

3196 - - tests/test_pooling_params.py

3197 - - tests/multimodal

3198 - - tests/renderers

3199 - - tests/standalone_tests/lazy_imports.py

3200 - - tests/tokenizers_

3201 - - tests/tool_parsers

3202 - - tests/transformers_utils

3203 - - tests/config

3204 - no_gpu: true

3205 - commands:

3206 - - python3 standalone_tests/lazy_imports.py

3207 - - pytest -v -s test_inputs.py

3208 - - pytest -v -s test_outputs.py

3209 - - pytest -v -s test_pooling_params.py

3210 - - pytest -v -s -m 'cpu_test' multimodal

3211 - - pytest -v -s renderers

3212 - - pytest -v -s tokenizers_

3213 - - pytest -v -s tool_parsers

3214 - - pytest -v -s transformers_utils

3215 - - pytest -v -s config

3216 -

3217 -- label: Python-only Installation Test # 10min

3218 - timeout_in_minutes: 20

3219 - mirror_hardwares: [amdexperimental]

3220 - agent_pool: mi355_1

3221 - optional: true

3222 - source_file_dependencies:

3223 - - tests/standalone_tests/python_only_compile.sh

3224 - - setup.py

3225 - commands:

3226 - - bash standalone_tests/python_only_compile.sh

3227 -

3228 -- label: Basic Correctness Test # 20min

3229 - timeout_in_minutes: 30

3230 - mirror_hardwares: [amdexperimental, amdproduction]

3231 - agent_pool: mi355_1

3232 - optional: true

3233 - fast_check: true

3234 - torch_nightly: true

3235 - source_file_dependencies:

3236 - - vllm/

3237 - - tests/basic_correctness/test_basic_correctness

3238 - - tests/basic_correctness/test_cpu_offload

3239 - - tests/basic_correctness/test_cumem.py

[ROCm][CI] Cleaning and restructuring amd-ci legacy pipeline (#34839)