A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 6, 04:30 UTC
[Frontend][Responses API] Fix arrival_time recording for TTFT on initial request (#37498)
Signed-off-by: Andrew Xia <axia@meta.com>
Andrew Xia committed
Mar 23, 2026 at 02:58 UTC
9ace378a63cac49129829ac30e9645bb8af4d2d5
2 files changed
+4
-1
docs/design/metrics.md
+2
-1
@@ -244,6 +244,7 @@ statistics relating to that iteration:
244
prefill in this iteration. However, we calculate this interval
245
relative to when the request was first received by the frontend
246
(`arrival_time`) in order to account for input processing time.
247
+ Currently `arrival_time` starts when tokenization begins.
248
249
For any requests that were completed in a given iteration, we also
250
record:
@@ -587,7 +588,7 @@ see:
588
- [Benchmarking LLM Workloads for Performance Evaluation and Autoscaling in Kubernetes](https://docs.google.com/document/d/1k4Q4X14hW4vftElIuYGDu5KDe2LtV1XammoG-Xi3bbQ)
589
- [Inference Perf](https://github.com/kubernetes-sigs/wg-serving/tree/main/proposals/013-inference-perf)
590
- <https://github.com/vllm-project/vllm/issues/5041> and <https://github.com/vllm-project/vllm/pull/12726>.
590
-
591
+
592
This is a non-trivial topic. Consider this comment from Rob:
593
594
> I think this metric should focus on trying to estimate what the max
vllm/entrypoints/openai/responses/serving.py
+2
@@ -710,9 +710,11 @@ class OpenAIServingResponses(OpenAIServing):
710
"Only 'auto' tool_choice is supported in response API with Harmony"
711
)
712
713
+ arrival_time = time.time()
714
messages = self._construct_input_messages_with_harmony(request, prev_response)
715
prompt_token_ids = render_for_completion(messages)
716
engine_prompt = token_inputs(prompt_token_ids)
717
+ engine_prompt["arrival_time"] = arrival_time
718
719
# Add cache_salt if provided in the request
720
if request.cache_salt is not None: