@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 86 lines 3.06 KB
Rendered Raw
1 # Setup OpenTelemetry POC
2
3 > **Note:** The core OpenTelemetry packages (`opentelemetry-sdk`, `opentelemetry-api`, `opentelemetry-exporter-otlp`, `opentelemetry-semantic-conventions-ai`) are bundled with vLLM. Manual installation is not required.
4
5 1. Start Jaeger in a docker container:
6
7 ```bash
8 # From: https://www.jaegertracing.io/docs/1.57/getting-started/
9 docker run --rm --name jaeger \
10 -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
11 -p 6831:6831/udp \
12 -p 6832:6832/udp \
13 -p 5778:5778 \
14 -p 16686:16686 \
15 -p 4317:4317 \
16 -p 4318:4318 \
17 -p 14250:14250 \
18 -p 14268:14268 \
19 -p 14269:14269 \
20 -p 9411:9411 \
21 jaegertracing/all-in-one:1.57
22 ```
23
24 1. In a new shell, export Jaeger IP:
25
26 ```bash
27 export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)
28 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
29 ```
30
31 Then set vLLM's service name for OpenTelemetry, enable insecure connections to Jaeger and run vLLM:
32
33 ```bash
34 export OTEL_SERVICE_NAME="vllm-server"
35 export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
36 vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
37 ```
38
39 1. In a new shell, send requests with trace context from a dummy client
40
41 ```bash
42 export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)
43 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
44 export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
45 export OTEL_SERVICE_NAME="client-service"
46 python dummy_client.py
47 ```
48
49 1. Open Jaeger webui: <http://localhost:16686/>
50
51 In the search pane, select `vllm-server` service and hit `Find Traces`. You should get a list of traces, one for each request.
52 ![Traces](https://i.imgur.com/GYHhFjo.png)
53
54 1. Clicking on a trace will show its spans and their tags. In this demo, each trace has 2 spans. One from the dummy client containing the prompt text and one from vLLM containing metadata about the request.
55 ![Spans details](https://i.imgur.com/OPf6CBL.png)
56
57 ## Exporter Protocol
58
59 OpenTelemetry supports either `grpc` or `http/protobuf` as the transport protocol for trace data in the exporter.
60 By default, `grpc` is used. To set `http/protobuf` as the protocol, configure the `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` environment variable as follows:
61
62 ```bash
63 export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
64 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
65 vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
66 ```
67
68 ## Instrumentation of FastAPI
69
70 OpenTelemetry allows automatic instrumentation of FastAPI.
71
72 1. Install the instrumentation library
73
74 ```bash
75 pip install opentelemetry-instrumentation-fastapi
76 ```
77
78 1. Run vLLM with `opentelemetry-instrument`
79
80 ```bash
81 opentelemetry-instrument vllm serve facebook/opt-125m
82 ```
83
84 1. Send a request to vLLM and find its trace in Jaeger. It should contain spans from FastAPI.
85
86 ![FastAPI Spans](https://i.imgur.com/hywvoOJ.png)