hydra: stop nginx recycling the queue-runner gRPC connection
Builders keep a single long-lived HTTP/2 channel to the queue runner and push a large number of small RPCs over it. nginx's default keepalive_requests (1000) makes nginx send GOAWAY and tear the connection down mid-stream, cancelling in-flight upload-completion RPCs. These surface on builders as "connection closed / operation was canceled" and abort the upload step. Raise keepalive_requests and keepalive_timeout on the queue-runner vhost so the connection is not recycled under normal load.
Jörg Thalheim committed
Jun 17, 2026 at 18:05 UTC
8233fa9b736136c1a51bfe62bf0278e60a2e1bc9
1 file changed
+6
build/hydra-queue-runner.nix
+6
@@ -61,6 +61,12 @@ in
61
grpc_send_timeout 31536000s; # 1 year in seconds
62
grpc_socket_keepalive on;
63
64
+ # Builders reuse one long-lived HTTP/2 channel for many RPCs. The
65
+ # default keepalive_requests (1000) makes nginx GOAWAY mid-stream,
66
+ # cancelling in-flight RPCs and aborting builds.
67
+ keepalive_requests 1000000;
68
+ keepalive_timeout 600s;
69
+
70
grpc_set_header Host $host;
71
grpc_set_header X-Real-IP $remote_addr;
72
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;