@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 51 lines 3.02 KB
Rendered Raw
1 # CI timing and GPU samples
2
3 The pipeline generator activates these helpers for trusted, traced CI commands.
4 `ci_otel.sh` records command spans and `ci_pytest.sh` injects the pytest timing
5 plugin. `ci_gpu.py` runs once per command, outside pytest workers, when
6 `nvidia-smi` is available. It queries up to 16 container-visible NVIDIA devices
7 once per second without importing PyTorch or creating a CUDA context.
8 Set `CI_INFRA_GPU_SAMPLING=0` in the job environment to disable sampling.
9
10 Samples contain device UUID, index, name, utilization percent, and used/total
11 device memory in bytes. They are `ci.gpu.sample` OTLP events inside
12 `ci.gpu.samples` spans, parented to the command span and authenticated using the
13 existing job-scoped Buildkite OIDC identity. Batches upload every 30 seconds;
14 the final batch is spooled for the shell's normal job-end upload. No host agent
15 or long-lived dashboard token is needed. The dashboard companion change adds
16 GPU controls alongside jobs, commands, and pytest rows in the Builds timeline.
17
18 Device queries and periodic uploads each have a two-second timeout. Three
19 consecutive query failures stop sampling. Failed periodic uploads are dropped;
20 memory stays bounded, and the dashboard shows the missing intervals as gaps.
21 Uploads can delay the next poll. The shell stops and joins the sampler before
22 flushing traces, allows at most three seconds for shutdown, and preserves the
23 original command exit status. Samples already uploaded survive an interrupted
24 job; the last unflushed batch may be lost on forced termination.
25
26 These are device readings during each test's time interval, not process-level
27 kernel attribution. Other processes and overlapping tests can contribute to
28 the readings. Subsecond tests may have no sample. Missing readings never become
29 zero utilization. MIG parent-device memory and utilization are omitted because
30 they cannot be attributed to the job's partition. When the environment identifies
31 an assigned MIG UUID, the collector queries that instance's used/total memory
32 through vLLM's standalone bundled NVML binding in a bounded subprocess.
33 The binding is loaded directly to avoid vLLM/Torch import side effects. Numeric
34 `CUDA_VISIBLE_DEVICES` indices are resolved against explicit MIG UUIDs in
35 `NVIDIA_VISIBLE_DEVICES`. For CDI assignments without an explicit UUID, a bounded
36 helper initializes the CUDA driver and reads only CUDA-visible device UUIDs,
37 without creating CUDA contexts or importing Torch. NVML verifies each MIG handle.
38 Discovery or permission failures remain unknown; NVML sibling instances are never
39 enumerated. MIG utilization stays absent.
40 CPU jobs and AMD mirrors do not collect GPU samples.
41
42 Run the helper tests without loading vLLM or requiring a GPU:
43
44 ```bash
45 PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest \
46 .buildkite/scripts/ci-otel/tests/test_ci_otel.py -q
47 ```
48
49 The suite uses a fake `nvidia-smi` to exercise the shell lifecycle, device-query
50 failure handling, disable switch, event encoding, and bounded uploads. Real
51 hardware sampling overhead still needs a canary run before broad deployment.