Bound the Onde Cloud request timeouts
A lone 300s read_timeout with no connect or write timeout meant a slow or hung Onde Cloud could sit on a puma thread for five minutes. A handful at once emptied the pool and took the site down, which is what happened on 2026-06-24. Added a 5s connect timeout and a 15s write timeout, and cut the read timeout to 90s. read_timeout fires per read rather than over the whole request, so a streaming reply can still take as long as it needs while tokens keep arriving; it only trips when the upstream goes quiet. A stall now returns a 504 quickly instead of parking a worker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
keypair34 committed
Jun 24, 2026 at 23:58 UTC
d8e590b83c464cb26648642d87fd41bfd5038652
1 file changed
+16
-1
app/services/onde_cloud_service.rb
+16
-1
@@ -34,7 +34,20 @@ class OndeCloudService
34
end
35
36
DEFAULT_BASE_URL = "https://cloud.ondeinference.com/v1"
37
- READ_TIMEOUT = 300 # seconds — long enough for a full agent turn
37
+
38
+ # Net::HTTP timeouts. These bound how long a single request can pin a puma
39
+ # thread — load-bearing for availability: a hung/slow Onde Cloud must never be
40
+ # able to exhaust the web pool and take the whole app down (it did, 2026-06-24,
41
+ # when an Onde app was pinned to a server-side GGUF that stalled on CPU
42
+ # inference).
43
+ #
44
+ # read_timeout is PER-READ, not total, so streaming agent turns of any length
45
+ # stay safe as long as tokens keep flowing; it only fires when the upstream
46
+ # goes silent. Keep it short enough that a stall fails fast as a 504 instead of
47
+ # blocking a worker.
48
+ OPEN_TIMEOUT = 5 # connect
49
+ WRITE_TIMEOUT = 15 # sending the request body
50
+ READ_TIMEOUT = 90 # silence between reads before we give up
51
52
# Non-streaming completion. Returns the parsed JSON response (a Hash) verbatim.
53
def self.create(payload)
@@ -77,6 +90,8 @@ class OndeCloudService
90
uri.hostname,
91
uri.port,
92
use_ssl: uri.scheme == "https",
93
+ open_timeout: OPEN_TIMEOUT,
94
+ write_timeout: WRITE_TIMEOUT,
95
read_timeout: READ_TIMEOUT,
96
&block
97
)