A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 2, 09:45 UTC
[CI] Make Model Executor test hangs fail fast with a traceback (#43971)
Signed-off-by: khluu <khluu000@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
Kevin H. Luu committed
May 29, 2026 at 11:58 UTC
6aabe221a56052965e6bb0a95e9ec682d046a6e7
1 file changed
+9
-2
.buildkite/test_areas/model_executor.yaml
+9
-2
@@ -14,5 +14,12 @@ steps:
14
commands:
15
- apt-get update && apt-get install -y curl libsodium23
16
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
17
- - pytest -v -s model_executor -m '(not slow_test)'
18
- - pytest -v -s entrypoints/openai/completion/test_tensorizer_entrypoint.py
17
+ # Dump tracebacks of all threads if a test hangs, so a wedged GPU/CUDA
18
+ # init surfaces a stack instead of silently stalling.
19
+ - export PYTHONFAULTHANDLER=1
20
+ # Per-test watchdog: a single hung test (e.g. stuck during engine/CUDA
21
+ # init) fails fast with a traceback instead of running until the global
22
+ # build timeout. The `thread` method also handles hangs inside C/CUDA
23
+ # calls that the signal method cannot interrupt.
24
+ - pytest -v -s model_executor -m '(not slow_test)' --timeout=900 --timeout-method=thread
25
+ - pytest -v -s entrypoints/openai/completion/test_tensorizer_entrypoint.py --timeout=900 --timeout-method=thread