@devries-cameron20 / composio Mirror · read-only / commits / d544006a2
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 16:00 UTC

fix(sdk): pin the validated address when fetching URLs (SSRF DNS rebinding) (#4172)

Fixes #4151. ## The problem Both SDKs validated a URL by resolving its hostname, and then handed the *hostname* to the HTTP client, which resolved it again when it opened the socket. Two lookups, two answers: a short-TTL record under an attacker's control answers publicly for the check and with `169.254.169.254`, `127.0.0.1`, or RFC 1918 space for the connect. The guard passes and the connection lands inside the network — classic TOCTOU DNS rebinding, documented in both modules until now as a known residual. ```mermaid sequenceDiagram participant SDK participant DNS as Attacker DNS participant Meta as 169.254.169.254 Note over SDK,Meta: before SDK->>DNS: resolve evil.example.com (validate) DNS-->>SDK: 93.184.216.34 — passes the guard SDK->>DNS: resolve evil.example.com (connect) DNS-->>SDK: 169.254.169.254 SDK->>Meta: GET /latest/meta-data/… Meta-->>SDK: credentials ``` ## The fix Resolve once, validate every answer, then connect to the address that was validated. There is no second lookup left to rebind. - **Python** — `safe_get` / `safe_request` mount a transport adapter that swaps the connect target for the duration of the socket connect only. The `Host` header and TLS SNI keep the hostname, so certificate verification is unchanged; rewriting `conn._dns_host` for the whole connection would have sent `Host: <ip>` and offered the IP as SNI, failing against every real origin. Every fetch call site now goes through those two helpers, so no `requests.get` sits next to a bare check any more: - `_files.py::_fetch_file_from_url`, `_files.py::FileDownloadable.download` - `tool_router_session_files.py::_fetch_url_bytes` - `safe_request`, per redirect hop - **TypeScript** — `assertSafeFetchTarget` returns the validated address and `ssrfSafeFetch` hands `fetch` a dispatcher pinned to it, re-pinned per redirect hop. The dispatcher goes to the runtime's own `fetch`, so callers that stub `globalThis.fetch` keep working. The pinned `lookup` answers both shapes Node calls it with — the address *list* it uses for Happy Eyeballs, and the single `(address, family)` it uses when `autoSelectFamily` is off — since answering in the wrong shape is rejected as an invalid address. - A fail-closed peer assertion runs on the Python side before a byte is written to the socket — redundant while pinning works, and a tripwire if a urllib3 upgrade ever breaks it. - `workerd` is unchanged: it already fails closed for user-supplied URLs. Redirect *validation* already existed in both SDKs (`safe_request` / `ssrfSafeFetch`); what was missing was re-pinning each hop. ## Tests The existing suites could not express this bug: they mock both the resolver and the HTTP client, so check and use are the same mock. The new tests use real sockets. - `python/tests/test_url_safety_pinning.py` — two loopback servers and a resolver that answers the first lookup with one endpoint and every later one with another, which is what a short-TTL rebinding record does. Asserts the rebound endpoint receives **zero** connections, and that `Host` still carries the hostname. Both tests fail on `next` and pass here. - `ts/packages/core/test/utils/pinnedDispatcher.node.test.ts` — a real server plus a hostname under `.invalid`, which RFC 2606 guarantees never resolves. A request that arrives proves the connect used the pinned address and never consulted DNS. The third case shows the contrast: unpinned, the same fetch cannot resolve at all. - `ssrfGuard.test.ts` gains assertions that each hop is pinned to that hop's own validated address. - `pinnedDispatcher.node.test.ts` also pins with `setDefaultAutoSelectFamily(false)`, which is the branch Node takes for the single-address callback. ## Notes - Supersedes #4157, which diagnosed this correctly. Its post-response peer check turned out not to hold: with an HTTP/1.0 or `Connection: close` server, urllib3 detaches the socket (`conn.sock is None`) while `r.content` still returns the full body, so the check fails open exactly where exfiltration succeeds. That is why the assertion here runs at connect time instead. - The Python package now declares `urllib3>=2` directly. `url_safety` imports it for `NameResolutionError`, which only exists from 2.0, and the pinning adapter reaches into 2.x connection internals; `requests` alone allows 1.x, where `import composio` would have failed outright. - `@composio/core` gains an `undici` dependency, pinned to `^7`: undici 8 dispatchers are rejected by the `fetch` in every Node version this package supports (22/24/25, verified). The real-socket test runs on the full CI matrix, so a future incompatibility fails loudly instead of silently un-pinning. - `undici` is imported on first pinned request rather than at module load: importing it installs a process-wide global dispatcher, which would have handed the host application's own unrelated `fetch` calls this package's undici merely because it imported `@composio/core`. - Residuals, now documented in the modules: - Requests routed through an environment proxy keep the pre-flight check only. The proxy resolves the hostname itself and the SDK cannot see or pin that resolution. - A process that does perform a pinned fetch still ends up on this package's `Agent` if nothing had claimed the global dispatcher slot yet. undici defines that slot non-configurable, so it cannot be handed back — assigning `undefined` leaves the runtime's own `fetch` asserting on a missing dispatcher.

Alberto Schiabel committed Aug 20, 2026 at 13:34 UTC d544006a25c4575685e5899b7b2595be003f736b
21 files changed +1059 -159
.changeset/ssrf-pin-validated-address.md new
+5
@@ -0,0 +1,5 @@
1 +---
2 +'@composio/core': patch
3 +---
4 +
5 +Close a DNS-rebinding window in the SSRF guard: the address validated by `assertSafeFetchTarget` is now the address `ssrfSafeFetch` connects to, so a hostname is no longer resolved a second time between the check and the connection. Each redirect hop is re-validated and re-pinned. The request still carries the original hostname in `Host` and TLS SNI, so certificate verification is unchanged. Hops whose effective dispatcher is a configured route — a caller-supplied `dispatcher`, a global `ProxyAgent`/`EnvHttpProxyAgent`, or `NODE_USE_ENV_PROXY` env-proxy mode — keep the pre-flight check only, mirroring the Python guard's documented proxy residual.
pnpm-lock.yaml
+6
@@ -1394,6 +1394,9 @@ importers:
1394 semver:
1395 specifier: ^7.8.5
1396 version: 7.8.5
1397 + undici:
1398 + specifier: ^7.29.0
1399 + version: 7.29.0
1400 zod-to-json-schema:
1401 specifier: 'catalog:'
1402 version: 3.25.2(zod@4.4.3)
@@ -1738,6 +1741,9 @@ importers:
1741 semver:
1742 specifier: ^7.8.5
1743 version: 7.8.5
1744 + undici:
1745 + specifier: ^7.29.0
1746 + version: 7.29.0
1747 zod:
1748 specifier: '>=3.25.76 <5'
1749 version: 4.4.3
python/composio/core/models/_files.py
+10 -11
@@ -27,8 +27,8 @@ from composio.utils import mimetypes
27 from composio.utils.json_schema import dereference_json_schema
28 from composio.utils.safe_path import secure_basename_join, secure_join
29 from composio.utils.url_safety import (
30 - assert_safe_fetch_target,
30 parse_content_length,
31 + safe_get,
32 safe_request,
33 )
34 from composio.utils.sensitive_file_upload_paths import (
@@ -398,14 +398,12 @@ def _fetch_file_from_url(
398 ResponseTooLargeError: If response exceeds max_size
399 ErrorUploadingFile: If fetch fails for other reasons
400 """
401 - assert_safe_fetch_target(url)
402 -
403 - # Make request without following redirects
401 + # `safe_get` validates the target, connects to the address it validated
402 + # (so DNS cannot rebind between the two), and never follows redirects.
403 try:
405 - response = requests.get(
404 + response = safe_get(
405 url,
406 stream=True, # Enable streaming for size limiting
408 - allow_redirects=False, # Disable redirects for security
407 timeout=(_CONNECT_TIMEOUT, _READ_TIMEOUT),
408 )
409 except requests.exceptions.Timeout:
@@ -696,13 +694,10 @@ class FileDownloadable(BaseModel):
694 outfile = secure_basename_join(outdir, self.name, root=root)
695 except UnsafePathComponentError as e:
696 raise ErrorDownloadingFile(str(e)) from e
699 - assert_safe_fetch_target(self.s3url)
700 - outdir.mkdir(exist_ok=True, parents=True)
697 try:
702 - response = requests.get(
703 - url=self.s3url,
698 + response = safe_get(
699 + self.s3url,
700 stream=True,
705 - allow_redirects=False,
701 timeout=(_CONNECT_TIMEOUT, _READ_TIMEOUT),
702 )
703 except requests.exceptions.RequestException as e:
@@ -717,6 +712,10 @@ class FileDownloadable(BaseModel):
712 )
713
714 try:
715 + # Only once the fetch is validated and connected, so a blocked URL
716 + # leaves no directory behind — and inside the `try`, so a failure
717 + # here still closes the response.
718 + outdir.mkdir(exist_ok=True, parents=True)
719 with outfile.open("wb") as fd:
720 for chunk in response.iter_content(chunk_size=chunk_size):
721 fd.write(chunk)
python/composio/core/models/tool_router_session_files.py
+2 -4
@@ -30,8 +30,8 @@ from composio.exceptions import (
30 from composio.utils.mimetypes import get_extension_from_mime_type
31 from composio.utils.safe_path import secure_basename_join
32 from composio.utils.url_safety import (
33 - assert_safe_fetch_target,
33 parse_content_length,
34 + safe_get,
35 safe_request,
36 )
37 from composio.utils.uuid import generate_short_id
@@ -90,12 +90,10 @@ def _fetch_url_bytes(url: str) -> t.Tuple[bytes, str]:
90
91 Returns (content, mimetype). Raises :class:`_UrlFetchError`.
92 """
93 - assert_safe_fetch_target(url)
93 try:
95 - response = requests.get(
94 + response = safe_get(
95 url,
96 stream=True,
98 - allow_redirects=False,
97 timeout=(_CONNECT_TIMEOUT, _READ_TIMEOUT),
98 )
99 except requests.exceptions.RequestException as e:
python/composio/utils/url_safety.py
+198 -21
@@ -8,13 +8,16 @@ SDK at loopback, RFC1918 space, or a link-local cloud-metadata endpoint
8 (``169.254.169.254``) and turn it into a request proxy for internal
9 infrastructure.
10
11 -Known residual — DNS rebinding: the host is resolved here with
12 -``getaddrinfo``, and the HTTP client resolves it again independently when it
13 -connects. A short-TTL record that alternates a public and a private answer can
14 -pass this check and still connect to the private address. Closing that window
15 -requires pinning the validated address and connecting to it directly (a custom
16 -``requests`` transport adapter), which is out of scope for this module; the
17 -TypeScript guard carries the same limitation.
11 +Validating a hostname is not enough on its own, because the hostname is
12 +resolved twice: once here, and once by the HTTP client when it opens the
13 +socket. A short-TTL record can answer with a public address for the first
14 +lookup and an internal one for the second — a time-of-check/time-of-use
15 +window better known as DNS rebinding. So the address validated here is also
16 +the address connected to: :func:`safe_get` and :func:`safe_request` pin it
17 +onto the connection, keeping the original hostname for the ``Host`` header
18 +and TLS SNI/certificate verification. Every fetch in the SDK goes through one
19 +of those two, so no call site can reintroduce the gap by calling
20 +``requests.get`` next to a bare check. The TypeScript guard pins the same way.
21
22 Also parses response headers that gate how much of a body the SDK reads:
23 ``parse_content_length`` treats ``Content-Length`` as the untrusted hint it
@@ -30,9 +33,20 @@ import typing as t
33 from urllib.parse import urljoin, urlparse
34
35 import requests
36 +import urllib3.exceptions
37
38 from composio.exceptions import BlockedInternalUrlError
39
40 +# urllib3 rewraps connect failures into its own hierarchy, none of which
41 +# inherits from OSError, so a bare `except OSError` would never see them and
42 +# the address fallback below would never run.
43 +_CONNECT_ERRORS = (
44 + OSError,
45 + urllib3.exceptions.NewConnectionError,
46 + urllib3.exceptions.ConnectTimeoutError,
47 + urllib3.exceptions.NameResolutionError,
48 +)
49 +
50 _REDIRECT_STATUS_CODES = frozenset({301, 302, 303, 307, 308})
51 _MAX_REDIRECTS = 5
52
@@ -57,11 +71,15 @@ def is_blocked_ip(value: str) -> bool:
71 return not address.is_global
72
73
60 -def assert_safe_fetch_target(url: str) -> None:
74 +def assert_safe_fetch_target(url: str) -> t.List[str]:
75 """Refuse non-HTTP(S) URLs and hosts that resolve to internal addresses.
76
77 Parse the URL after Requests prepares it so validation uses the same
78 canonical hostname that the eventual connection will use.
79 +
80 + :returns: The validated addresses to connect to, in resolver order.
81 + Callers must connect to *these* rather than re-resolving the hostname;
82 + see :func:`safe_get`.
83 """
84 try:
85 prepared_url = requests.Request(method="GET", url=url).prepare().url
@@ -76,14 +94,17 @@ def assert_safe_fetch_target(url: str) -> None:
94 ) from None
95
96 try:
79 - addresses: set[str] = set()
97 + # Resolver order is kept: it encodes the system's address preference
98 + # (RFC 6724), and connecting walks it the way urllib3 would.
99 + addresses: t.List[str] = []
100 for result in socket.getaddrinfo(parsed.hostname, None):
101 address = result[4][0]
102 if not isinstance(address, str):
103 raise BlockedInternalUrlError(
104 f'Could not resolve host "{parsed.hostname}"'
105 )
86 - addresses.add(address)
106 + if address not in addresses:
107 + addresses.append(address)
108 except socket.gaierror as error:
109 raise BlockedInternalUrlError(
110 f'Could not resolve host "{parsed.hostname}"'
@@ -95,6 +116,11 @@ def assert_safe_fetch_target(url: str) -> None:
116 f'Refusing to fetch "{parsed.hostname}" because it resolves to a non-public address'
117 )
118
119 + if not addresses:
120 + raise BlockedInternalUrlError(f'Could not resolve host "{parsed.hostname}"')
121 +
122 + return addresses
123 +
124
125 def parse_content_length(value: t.Optional[str]) -> t.Optional[int]:
126 """Parse a ``Content-Length`` header into a non-negative ``int``.
@@ -126,15 +152,15 @@ def safe_request(
152 ) -> requests.Response:
153 """Send a request, validating the target before *every* hop.
154
129 - Redirects are followed manually so each new location is validated too.
130 - Validating only the first URL is not enough: a target that passes the check
131 - and then answers ``302 Location: http://169.254.169.254/`` would have the
132 - redirect followed by ``requests`` with no further validation. Mirrors
133 - ``ssrfSafeFetch`` in the TypeScript SDK.
155 + Redirects are followed manually so each new location is validated — and
156 + pinned — too. Validating only the first URL is not enough: a target that
157 + passes the check and then answers ``302 Location: http://169.254.169.254/``
158 + would have the redirect followed by ``requests`` with no further
159 + validation. Mirrors ``ssrfSafeFetch`` in the TypeScript SDK.
160
161 Use this where redirects are legitimate (S3 can answer a PUT with a 307
136 - region redirect). Call sites that require a direct URL should instead call
137 - :func:`assert_safe_fetch_target` and pass ``allow_redirects=False``.
162 + region redirect). Call sites that require a direct URL should use
163 + :func:`safe_get`, which rejects nothing but simply does not follow them.
164
165 :param max_redirects: Hops to follow before giving up.
166 :raises BlockedInternalUrlError: If any hop fails validation, or the
@@ -144,10 +170,7 @@ def safe_request(
170 current_url = url
171
172 for _ in range(max_redirects + 1):
147 - assert_safe_fetch_target(current_url)
148 - response = requests.request(
149 - method, current_url, allow_redirects=False, **kwargs
150 - )
173 + response = _pinned_request(method, current_url, **kwargs)
174
175 location = response.headers.get("Location")
176 if response.status_code not in _REDIRECT_STATUS_CODES or location is None:
@@ -165,3 +188,157 @@ def safe_request(
188 raise BlockedInternalUrlError(
189 f"Refusing to fetch: too many redirects (max {max_redirects})"
190 )
191 +
192 +
193 +class _PinnedAddressAdapter(requests.adapters.HTTPAdapter):
194 + """Transport adapter that connects to a pre-validated address.
195 +
196 + The hostname is left untouched on the connection, so the ``Host`` header
197 + and the TLS SNI/certificate check still use it; only the address the
198 + socket dials is replaced. Doing it the other way round — rewriting
199 + ``conn._dns_host`` for the whole connection — would also rewrite
200 + ``conn.host``, which urllib3 derives from it, and the request would go out
201 + with an IP in ``Host`` and an IP in SNI, failing certificate verification
202 + against every real origin.
203 +
204 + This reaches into two urllib3 internals, ``HTTPConnection._new_conn`` and
205 + ``HTTPConnection._dns_host``. ``test_url_safety_pinning.py`` asserts both
206 + exist so a urllib3 upgrade that removes them fails loudly rather than
207 + silently un-pinning the connection.
208 + """
209 +
210 + def __init__(self, addresses: t.Sequence[str], **kwargs: t.Any) -> None:
211 + self._addresses = list(addresses)
212 + super().__init__(**kwargs)
213 +
214 + def get_connection_with_tls_context(
215 + self,
216 + request: requests.PreparedRequest,
217 + verify: t.Union[bool, str, None],
218 + proxies: t.Optional[t.Mapping[str, str]] = None,
219 + cert: t.Union[str, t.Tuple[str, str], None] = None,
220 + ) -> t.Any:
221 + # `Any`, because the pinning below reaches for urllib3 internals that
222 + # the typed `ConnectionPool` surface does not expose.
223 + pool: t.Any = super().get_connection_with_tls_context(
224 + request, verify, proxies=proxies, cert=cert
225 + )
226 + addresses = self._addresses
227 + build_connection = pool._new_conn
228 +
229 + def _new_conn() -> t.Any:
230 + connection = build_connection()
231 + open_socket = connection._new_conn
232 +
233 + def _pinned_new_conn() -> t.Any:
234 + # Swap the resolution target for the duration of the socket
235 + # connect only. urllib3 reads `self.host` for SNI *after*
236 + # `_new_conn()` returns, and `http.client` reads it later
237 + # still for the `Host` header, so both see the hostname.
238 + #
239 + # Every validated address is tried in resolver order, the way
240 + # urllib3 would have: pinning one address of a dual-stack host
241 + # would strand callers whose network cannot reach that family.
242 + hostname = connection._dns_host
243 + last_error: t.Optional[BaseException] = None
244 + for address in addresses:
245 + connection._dns_host = address
246 + try:
247 + sock = open_socket()
248 + except _CONNECT_ERRORS as error:
249 + last_error = error
250 + continue
251 + finally:
252 + connection._dns_host = hostname
253 + _assert_pinned_peer(sock, address, hostname)
254 + return sock
255 +
256 + assert last_error is not None
257 + raise last_error
258 +
259 + connection._new_conn = _pinned_new_conn
260 + return connection
261 +
262 + pool._new_conn = _new_conn
263 + return pool
264 +
265 +
266 +def _assert_pinned_peer(sock: t.Any, address: str, hostname: str) -> None:
267 + """Fail closed if the open socket is not connected to the pinned address.
268 +
269 + A redundancy check on the pinning above, run before a single byte is
270 + written to the socket — a post-response check would be too late and
271 + unreliable, because urllib3 detaches the socket as soon as the server
272 + signals ``Connection: close``, while the body stays readable.
273 + """
274 + try:
275 + peer = sock.getpeername()[0]
276 + except (AttributeError, OSError, IndexError):
277 + return
278 +
279 + try:
280 + connected_to_pinned = ipaddress.ip_address(peer) == ipaddress.ip_address(
281 + address
282 + )
283 + except ValueError:
284 + connected_to_pinned = peer == address
285 +
286 + if connected_to_pinned:
287 + return
288 +
289 + sock.close()
290 + raise BlockedInternalUrlError(
291 + f'Refusing to talk to "{hostname}": the connection was established to '
292 + f"{peer}, not to the validated address {address}"
293 + )
294 +
295 +
296 +def _proxy_applies(url: str, proxies: t.Optional[t.Mapping[str, str]]) -> bool:
297 + """Whether Requests would send ``url`` through a proxy.
298 +
299 + Requests honours ``HTTP_PROXY``/``HTTPS_PROXY``/``ALL_PROXY`` (minus
300 + ``NO_PROXY``) by default. Through a proxy the socket is dialled to the
301 + *proxy*, so pinning the target address would connect to the wrong host
302 + entirely.
303 +
304 + Residual: proxied requests keep only the pre-flight check, because the
305 + proxy resolves the hostname itself and the SDK cannot see or pin that
306 + resolution. A rebinding window therefore remains for callers that run
307 + behind a proxy — including one inherited from the environment.
308 + """
309 + try:
310 + environment_proxies = requests.utils.get_environ_proxies(url)
311 + merged = {**environment_proxies, **(proxies or {})}
312 + return requests.utils.select_proxy(url, merged) is not None
313 + except Exception: # pragma: no cover - platform proxy discovery can fail
314 + # Unknowable, so assume a proxy rather than pinning onto a connection
315 + # that may not go where we think it goes.
316 + return True
317 +
318 +
319 +def safe_get(url: str, **kwargs: t.Any) -> requests.Response:
320 + """Validate a URL and fetch it without re-resolving its hostname.
321 +
322 + Redirects are never followed: call sites that need a direct URL treat a
323 + 3xx as an error, and call sites where redirects are legitimate use
324 + :func:`safe_request`.
325 + """
326 + return _pinned_request("GET", url, **kwargs)
327 +
328 +
329 +def _pinned_request(method: str, url: str, **kwargs: t.Any) -> requests.Response:
330 + addresses = assert_safe_fetch_target(url)
331 +
332 + session = requests.Session()
333 + if not _proxy_applies(url, kwargs.get("proxies")):
334 + # A fresh Session per request, so a connection pinned to one address is
335 + # never reused for a request validated against another.
336 + adapter = _PinnedAddressAdapter(addresses)
337 + session.mount("http://", adapter)
338 + session.mount("https://", adapter)
339 +
340 + response = session.request(method, url, allow_redirects=False, **kwargs)
341 + # The response may still be streaming, and closing the session would close
342 + # the pool holding its connection, so tie the session's lifetime to it.
343 + response._composio_session = session # type: ignore[attr-defined]
344 + return response
python/pyproject.toml
+8
@@ -12,6 +12,14 @@ dependencies = [
12 "openai>=2.48.0",
13 "json-schema-to-pydantic>=0.4.11",
14 "jsonschema>=4.23.0",
15 + # The pinning adapter mounts through `get_connection_with_tls_context`,
16 + # which `HTTPAdapter.send` only calls on requests >= 2.32.2; older
17 + # versions silently skip it and the pinning never engages.
18 + "requests>=2.32.2",
19 + # `url_safety` imports urllib3 directly: `NameResolutionError` only exists
20 + # from 2.0, and the pinning adapter reaches into 2.x connection internals.
21 + # requests alone allows 1.x, which would fail at import time.
22 + "urllib3>=2",
23 ]
24 classifiers = [
25 "Programming Language :: Python :: 3",
python/setup.py
+6
@@ -30,6 +30,12 @@ setup(
30 "openai>=2.48.0",
31 "json-schema-to-pydantic>=0.4.11",
32 "jsonschema>=4.23.0",
33 + # `get_connection_with_tls_context` (the pinning adapter's mount
34 + # point) is only called by `HTTPAdapter.send` on requests >= 2.32.2.
35 + "requests>=2.32.2",
36 + # `url_safety` imports urllib3 directly and needs 2.x: 1.x has no
37 + # `NameResolutionError` and different connection internals.
38 + "urllib3>=2",
39 ],
40 include_package_data=True,
41 )
python/tests/test_file_upload_robustness.py
+4 -10
@@ -186,11 +186,8 @@ class TestMalformedContentLength:
186 @pytest.mark.parametrize(
187 "header", ["abc", "12.5", "1,024", "1e3", "0x10", "100 200", ""]
188 )
189 - @patch("composio.core.models._files.assert_safe_fetch_target")
190 - @patch("composio.core.models._files.requests.get")
191 - def test_malformed_header_does_not_raise(
192 - self, mock_get: MagicMock, _mock_assert: MagicMock, header: str
193 - ):
189 + @patch("composio.core.models._files.safe_get")
190 + def test_malformed_header_does_not_raise(self, mock_get: MagicMock, header: str):
191 """A malformed header must not surface a raw ValueError to the caller."""
192 mock_get.return_value = _stream_response(
193 {"content-type": "image/jpeg", "Content-Length": header}
@@ -204,11 +201,8 @@ class TestMalformedContentLength:
201 assert content == b"payload"
202 assert mimetype == "image/jpeg"
203
207 - @patch("composio.core.models._files.assert_safe_fetch_target")
208 - @patch("composio.core.models._files.requests.get")
209 - def test_negative_header_still_enforced_while_streaming(
210 - self, mock_get: MagicMock, _mock_assert: MagicMock
211 - ):
204 + @patch("composio.core.models._files.safe_get")
205 + def test_negative_header_still_enforced_while_streaming(self, mock_get: MagicMock):
206 """A negative header means unknown size, not a trusted "small" value."""
207 mock_get.return_value = _stream_response(
208 {"Content-Length": "-1"},
python/tests/test_files.py
+62 -63
@@ -1560,7 +1560,7 @@ class TestUrlHelperFunctions:
1560 class TestFetchFileFromUrl:
1561 """Test cases for _fetch_file_from_url function."""
1562
1563 - @patch("composio.core.models._files.requests.get")
1563 + @patch("composio.core.models._files.safe_get")
1564 def test_fetch_file_from_url_success(self, mock_get):
1565 """Test successful file fetch from URL."""
1566 mock_response = MagicMock()
@@ -1581,11 +1581,10 @@ class TestFetchFileFromUrl:
1581 mock_get.assert_called_once_with(
1582 "https://example.com/image.jpg",
1583 stream=True,
1584 - allow_redirects=False,
1584 timeout=(5, 60),
1585 )
1586
1588 - @patch("composio.core.models._files.requests.get")
1587 + @patch("composio.core.models._files.safe_get")
1588 def test_fetch_file_from_url_with_charset_in_content_type(self, mock_get):
1589 """Test that charset is stripped from content-type."""
1590 mock_response = MagicMock()
@@ -1602,7 +1601,7 @@ class TestFetchFileFromUrl:
1601
1602 assert mimetype == "text/html"
1603
1605 - @patch("composio.core.models._files.requests.get")
1604 + @patch("composio.core.models._files.safe_get")
1605 def test_fetch_file_from_url_generates_filename_when_missing(self, mock_get):
1606 """Test filename generation when URL has no filename."""
1607 mock_response = MagicMock()
@@ -1618,7 +1617,7 @@ class TestFetchFileFromUrl:
1617 assert filename.startswith("file_")
1618 assert filename.endswith(".png")
1619
1621 - @patch("composio.core.models._files.requests.get")
1620 + @patch("composio.core.models._files.safe_get")
1621 def test_fetch_file_from_url_generates_filename_when_no_extension(self, mock_get):
1622 """Test filename generation when URL filename has no extension."""
1623 mock_response = MagicMock()
@@ -1636,7 +1635,7 @@ class TestFetchFileFromUrl:
1635 assert filename.startswith("file_")
1636 assert filename.endswith(".pdf")
1637
1639 - @patch("composio.core.models._files.requests.get")
1638 + @patch("composio.core.models._files.safe_get")
1639 def test_fetch_file_from_url_failure(self, mock_get):
1640 """Test error handling when URL fetch fails."""
1641 mock_response = MagicMock()
@@ -1651,7 +1650,7 @@ class TestFetchFileFromUrl:
1650 assert "Failed to fetch file from URL" in str(exc_info.value)
1651 assert "404" in str(exc_info.value)
1652
1654 - @patch("composio.core.models._files.requests.get")
1653 + @patch("composio.core.models._files.safe_get")
1654 def test_fetch_file_from_url_decodes_percent_encoded_filename(self, mock_get):
1655 """Test that percent-encoded characters in URL filenames are decoded."""
1656 mock_response = MagicMock()
@@ -1672,7 +1671,7 @@ class TestFetchFileFromUrl:
1671 assert content == b"document content"
1672 assert mimetype == "application/pdf"
1673
1675 - @patch("composio.core.models._files.requests.get")
1674 + @patch("composio.core.models._files.safe_get")
1675 def test_fetch_file_from_url_decodes_unicode_filename(self, mock_get):
1676 """Test that percent-encoded unicode characters in URL filenames are decoded."""
1677 mock_response = MagicMock()
@@ -1691,7 +1690,7 @@ class TestFetchFileFromUrl:
1690 # Filename should be decoded to unicode
1691 assert filename == "ファイル.jpg"
1692
1694 - @patch("composio.core.models._files.requests.get")
1693 + @patch("composio.core.models._files.safe_get")
1694 def test_fetch_file_from_url_handles_plus_sign_in_filename(self, mock_get):
1695 """Test that plus signs in URL paths are preserved (not converted to spaces)."""
1696 mock_response = MagicMock()
@@ -2215,9 +2214,8 @@ class TestTruncateFilename:
2214 class TestFetchFileFromUrlWithTruncation:
2215 """Test cases for _fetch_file_from_url with filename truncation."""
2216
2218 - @patch("composio.core.models._files.assert_safe_fetch_target")
2219 - @patch("composio.core.models._files.requests.get")
2220 - def test_fetch_truncates_long_filename(self, mock_get, _mock_safe_fetch_target):
2217 + @patch("composio.core.models._files.safe_get")
2218 + def test_fetch_truncates_long_filename(self, mock_get):
2219 """Long filenames from URLs should be truncated."""
2220 mock_response = MagicMock()
2221 mock_response.ok = True
@@ -2239,7 +2237,7 @@ class TestFetchFileFromUrlWithTruncation:
2237 assert filename.endswith(".pdf")
2238 assert content == b"test content"
2239
2242 - @patch("composio.core.models._files.requests.get")
2240 + @patch("composio.core.models._files.safe_get")
2241 def test_fetch_preserves_short_filename(self, mock_get):
2242 """Short filenames should be preserved unchanged."""
2243 mock_response = MagicMock()
@@ -2256,11 +2254,8 @@ class TestFetchFileFromUrlWithTruncation:
2254
2255 assert filename == "photo.jpg"
2256
2259 - @patch("composio.core.models._files.assert_safe_fetch_target")
2260 - @patch("composio.core.models._files.requests.get")
2261 - def test_fetch_truncates_after_adding_extension(
2262 - self, mock_get, _mock_safe_fetch_target
2263 - ):
2257 + @patch("composio.core.models._files.safe_get")
2258 + def test_fetch_truncates_after_adding_extension(self, mock_get):
2259 """Truncation should happen after extension is appended."""
2260 mock_response = MagicMock()
2261 mock_response.ok = True
@@ -2280,7 +2275,7 @@ class TestFetchFileFromUrlWithTruncation:
2275 assert len(filename) <= _MAX_FILENAME_LENGTH
2276 assert filename.endswith(".pdf")
2277
2283 - @patch("composio.core.models._files.requests.get")
2278 + @patch("composio.core.models._files.safe_get")
2279 def test_fetch_generated_filename_not_truncated(self, mock_get):
2280 """Generated timestamped filenames (when URL has no filename) should be short enough."""
2281 mock_response = MagicMock()
@@ -2299,7 +2294,7 @@ class TestFetchFileFromUrlWithTruncation:
2294 assert filename.endswith(".png")
2295 assert len(filename) < 50 # Timestamped names are short
2296
2302 - @patch("composio.core.models._files.requests.get")
2297 + @patch("composio.core.models._files.safe_get")
2298 def test_fetch_long_real_world_url(self, mock_get):
2299 """Long real-world URLs should be handled correctly."""
2300 mock_response = MagicMock()
@@ -2326,7 +2321,7 @@ class TestFetchFileFromUrlWithTruncation:
2321 class TestResponseSizeLimit:
2322 """Test response size limiting."""
2323
2329 - @patch("composio.core.models._files.requests.get")
2324 + @patch("composio.core.models._files.safe_get")
2325 def test_rejects_oversized_content_length(self, mock_get):
2326 """Files with Content-Length > max_size should be rejected early."""
2327 mock_response = MagicMock()
@@ -2341,7 +2336,7 @@ class TestResponseSizeLimit:
2336 "https://example.com/large.zip", max_size=100 * 1024 * 1024
2337 )
2338
2344 - @patch("composio.core.models._files.requests.get")
2339 + @patch("composio.core.models._files.safe_get")
2340 def test_rejects_oversized_during_streaming(self, mock_get):
2341 """Files that exceed max_size during download should be rejected."""
2342 mock_response = MagicMock()
@@ -2360,7 +2355,7 @@ class TestResponseSizeLimit:
2355 "https://example.com/large.zip", max_size=10 * 1024 * 1024
2356 )
2357
2363 - @patch("composio.core.models._files.requests.get")
2358 + @patch("composio.core.models._files.safe_get")
2359 def test_accepts_file_within_limit(self, mock_get):
2360 """Files within size limit should be accepted."""
2361 mock_response = MagicMock()
@@ -2382,7 +2377,7 @@ class TestResponseSizeLimit:
2377 class TestRedirectHandling:
2378 """Test redirect handling (redirects should be rejected)."""
2379
2385 - @patch("composio.core.models._files.requests.get")
2380 + @patch("composio.core.models._files.safe_get")
2381 def test_rejects_redirect_302(self, mock_get):
2382 """302 redirects should be rejected with clear error message."""
2383 mock_response = MagicMock()
@@ -2394,7 +2389,7 @@ class TestRedirectHandling:
2389 with pytest.raises(ErrorUploadingFile, match="redirect"):
2390 _fetch_file_from_url("https://example.com/redirect")
2391
2397 - @patch("composio.core.models._files.requests.get")
2392 + @patch("composio.core.models._files.safe_get")
2393 def test_rejects_redirect_301(self, mock_get):
2394 """301 redirects should be rejected."""
2395 mock_response = MagicMock()
@@ -2406,7 +2401,7 @@ class TestRedirectHandling:
2401 with pytest.raises(ErrorUploadingFile, match="redirect"):
2402 _fetch_file_from_url("https://example.com/test")
2403
2409 - @patch("composio.core.models._files.requests.get")
2404 + @patch("composio.core.models._files.safe_get")
2405 def test_rejects_redirect_307(self, mock_get):
2406 """307 redirects should be rejected."""
2407 mock_response = MagicMock()
@@ -2418,7 +2413,7 @@ class TestRedirectHandling:
2413 with pytest.raises(ErrorUploadingFile, match="redirect"):
2414 _fetch_file_from_url("https://example.com/test")
2415
2421 - @patch("composio.core.models._files.requests.get")
2416 + @patch("composio.core.models._files.safe_get")
2417 def test_rejects_redirect_308(self, mock_get):
2418 """308 redirects should be rejected."""
2419 mock_response = MagicMock()
@@ -2540,8 +2535,7 @@ class TestFileDownloadablePathTraversal:
2535
2536 @pytest.fixture(autouse=True)
2537 def _allow_fetch_target(self):
2543 - with patch("composio.core.models._files.assert_safe_fetch_target"):
2544 - yield
2538 + yield
2539
2540 def _mock_response(self, content: bytes = b"data") -> MagicMock:
2541 response = MagicMock()
@@ -2559,7 +2553,7 @@ class TestFileDownloadablePathTraversal:
2553 s3url="https://example.com/file",
2554 )
2555 with patch(
2562 - "composio.core.models._files.requests.get",
2556 + "composio.core.models._files.safe_get",
2557 return_value=self._mock_response(b"#!/bin/sh\n"),
2558 ):
2559 written = f.download(outdir, root=outdir)
@@ -2578,7 +2572,7 @@ class TestFileDownloadablePathTraversal:
2572 s3url="https://example.com/file",
2573 )
2574 with patch(
2581 - "composio.core.models._files.requests.get",
2575 + "composio.core.models._files.safe_get",
2576 return_value=self._mock_response(b"x"),
2577 ):
2578 written = f.download(outdir, root=outdir)
@@ -2599,7 +2593,7 @@ class TestFileDownloadablePathTraversal:
2593 s3url="https://example.com/file",
2594 )
2595 with patch(
2602 - "composio.core.models._files.requests.get",
2596 + "composio.core.models._files.safe_get",
2597 return_value=self._mock_response(),
2598 ):
2599 with pytest.raises(ErrorDownloadingFile, match="Path traversal detected"):
@@ -2615,7 +2609,7 @@ class TestFileDownloadablePathTraversal:
2609 s3url="https://example.com/file",
2610 )
2611 with patch(
2618 - "composio.core.models._files.requests.get",
2612 + "composio.core.models._files.safe_get",
2613 return_value=self._mock_response(b"%PDF-1.4"),
2614 ):
2615 written = f.download(outdir, root=outdir)
@@ -2631,15 +2625,14 @@ class TestFileDownloadablePathTraversal:
2625 s3url="https://example.com/file",
2626 )
2627 with patch(
2634 - "composio.core.models._files.requests.get",
2628 + "composio.core.models._files.safe_get",
2629 return_value=self._mock_response(b"%PDF-1.4"),
2630 ) as mock_get:
2631 f.download(outdir, root=outdir)
2632
2633 mock_get.assert_called_once_with(
2640 - url="https://example.com/file",
2634 + "https://example.com/file",
2635 stream=True,
2642 - allow_redirects=False,
2636 timeout=(5, 60),
2637 )
2638
@@ -2651,7 +2644,7 @@ class TestFileDownloadablePathTraversal:
2644 s3url="https://example.com/file?token=abc",
2645 )
2646 with patch(
2654 - "composio.core.models._files.requests.get",
2647 + "composio.core.models._files.safe_get",
2648 side_effect=requests.exceptions.Timeout(
2649 "Max retries exceeded with url: /file?token=abc"
2650 ),
@@ -2676,7 +2669,7 @@ class TestFileDownloadablePathTraversal:
2669 s3url="https://example.com/file?token=abc",
2670 )
2671 with patch(
2679 - "composio.core.models._files.requests.get",
2672 + "composio.core.models._files.safe_get",
2673 return_value=response,
2674 ):
2675 with pytest.raises(ErrorDownloadingFile) as exc_info:
@@ -2696,7 +2689,7 @@ class TestFileDownloadablePathTraversal:
2689 s3url="https://example.com/file?token=abc",
2690 )
2691 with patch(
2699 - "composio.core.models._files.requests.get",
2692 + "composio.core.models._files.safe_get",
2693 return_value=response,
2694 ):
2695 with pytest.raises(ErrorDownloadingFile) as exc_info:
@@ -2720,7 +2713,7 @@ class TestFileDownloadablePathTraversal:
2713 s3url="https://example.com/file",
2714 )
2715 with patch(
2723 - "composio.core.models._files.requests.get",
2716 + "composio.core.models._files.safe_get",
2717 return_value=self._mock_response(),
2718 ):
2719 with pytest.raises(ErrorDownloadingFile, match="Path traversal detected"):
@@ -2747,7 +2740,7 @@ class TestFileDownloadablePathTraversal:
2740 s3url="https://example.com/file",
2741 )
2742 with patch(
2750 - "composio.core.models._files.requests.get",
2743 + "composio.core.models._files.safe_get",
2744 return_value=self._mock_response(b"x"),
2745 ):
2746 with pytest.raises(ErrorDownloadingFile, match="no usable basename"):
@@ -2783,7 +2776,7 @@ class TestFileDownloadablePathTraversal:
2776 s3url="https://example.com/file",
2777 )
2778 with patch(
2786 - "composio.core.models._files.requests.get",
2779 + "composio.core.models._files.safe_get",
2780 return_value=self._mock_response(b"x"),
2781 ):
2782 with pytest.raises(ErrorDownloadingFile, match=reason):
@@ -2838,7 +2831,7 @@ class TestDownloadDirSlugTraversal:
2831 response.status_code = 200
2832 response.iter_content = lambda chunk_size: [content]
2833 response.close = MagicMock()
2841 - return patch("composio.core.models._files.requests.get", return_value=response)
2834 + return patch("composio.core.models._files.safe_get", return_value=response)
2835
2836 @pytest.mark.parametrize("slug", TRAVERSALS)
2837 def test_hostile_tool_slug_is_rejected(self, slug, tmp_path):
@@ -3069,15 +3062,14 @@ class TestResponseDerivedUrlsAreGuarded:
3062 s3url="https://s3.example.com/file",
3063 )
3064 with patch(
3072 - "composio.core.models._files.assert_safe_fetch_target"
3073 - ) as mock_assert:
3074 - with patch(
3075 - "composio.core.models._files.requests.get",
3076 - return_value=self._download_response(),
3077 - ):
3078 - f.download(tmp_path / "out", root=tmp_path / "out")
3065 + "composio.core.models._files.safe_get",
3066 + return_value=self._download_response(),
3067 + ) as mock_get:
3068 + f.download(tmp_path / "out", root=tmp_path / "out")
3069
3080 - mock_assert.assert_called_once_with("https://s3.example.com/file")
3070 + # `safe_get` is the guard: it validates the target and connects to the
3071 + # address it validated, rather than re-resolving the hostname.
3072 + assert mock_get.call_args.args == ("https://s3.example.com/file",)
3073
3074 def test_download_blocked_url_never_reaches_the_network(self, tmp_path):
3075 outdir = tmp_path / "out"
@@ -3087,14 +3079,16 @@ class TestResponseDerivedUrlsAreGuarded:
3079 s3url="http://169.254.169.254/latest/meta-data",
3080 )
3081 with patch(
3090 - "composio.core.models._files.assert_safe_fetch_target",
3082 + "composio.utils.url_safety.assert_safe_fetch_target",
3083 side_effect=BlockedInternalUrlError("blocked"),
3084 ):
3093 - with patch("composio.core.models._files.requests.get") as mock_get:
3085 + with patch(
3086 + "composio.utils.url_safety.requests.Session.request"
3087 + ) as mock_send:
3088 with pytest.raises(BlockedInternalUrlError):
3089 f.download(outdir, root=outdir)
3090
3097 - mock_get.assert_not_called()
3091 + mock_send.assert_not_called()
3092 assert not outdir.exists()
3093
3094 def test_download_refuses_to_follow_redirects(self, tmp_path):
@@ -3103,14 +3097,15 @@ class TestResponseDerivedUrlsAreGuarded:
3097 mimetype="application/pdf",
3098 s3url="https://s3.example.com/file",
3099 )
3106 - with patch("composio.core.models._files.assert_safe_fetch_target"):
3107 - with patch(
3108 - "composio.core.models._files.requests.get",
3109 - return_value=self._download_response(),
3110 - ) as mock_get:
3111 - f.download(tmp_path / "out", root=tmp_path / "out")
3100 + with patch(
3101 + "composio.core.models._files.safe_get",
3102 + return_value=self._download_response(),
3103 + ) as mock_get:
3104 + f.download(tmp_path / "out", root=tmp_path / "out")
3105
3113 - assert mock_get.call_args.kwargs["allow_redirects"] is False
3106 + # `safe_get` never follows redirects, and passing `allow_redirects`
3107 + # through to it would be a way to turn that off.
3108 + assert "allow_redirects" not in mock_get.call_args.kwargs
3109
3110 def test_upload_bytes_to_s3_goes_through_safe_request(self):
3111 client = self._s3_client("https://s3.example.com/upload")
@@ -3136,7 +3131,9 @@ class TestResponseDerivedUrlsAreGuarded:
3131 "composio.utils.url_safety.assert_safe_fetch_target",
3132 side_effect=BlockedInternalUrlError("blocked"),
3133 ):
3139 - with patch("composio.utils.url_safety.requests.request") as mock_request:
3134 + with patch(
3135 + "composio.utils.url_safety.requests.Session.request"
3136 + ) as mock_request:
3137 with pytest.raises(BlockedInternalUrlError):
3138 _upload_bytes_to_s3(
3139 client=client,
@@ -3170,7 +3167,9 @@ class TestResponseDerivedUrlsAreGuarded:
3167 "composio.utils.url_safety.assert_safe_fetch_target",
3168 side_effect=BlockedInternalUrlError("blocked"),
3169 ):
3173 - with patch("composio.utils.url_safety.requests.request") as mock_request:
3170 + with patch(
3171 + "composio.utils.url_safety.requests.Session.request"
3172 + ) as mock_request:
3173 with pytest.raises(BlockedInternalUrlError):
3174 upload(url="http://127.0.0.1:9000/upload", file=source)
3175
python/tests/test_tool_router_session_files.py
+22 -22
@@ -18,7 +18,9 @@ from composio.exceptions import (
18
19 MODULE = "composio.core.models.tool_router_session_files"
20 SAFE_REQUEST = f"{MODULE}.safe_request"
21 -ASSERT_SAFE_FETCH_TARGET = f"{MODULE}.assert_safe_fetch_target"
21 +SAFE_GET = f"{MODULE}.safe_get"
22 +ASSERT_SAFE_FETCH_TARGET = "composio.utils.url_safety.assert_safe_fetch_target"
23 +SESSION_REQUEST = "composio.utils.url_safety.requests.Session.request"
24
25
26 def mock_stream_response(
@@ -214,14 +216,13 @@ class TestRemoteFile:
216 download_url="https://example.com/file",
217 )
218 with patch(ASSERT_SAFE_FETCH_TARGET):
217 - with patch("requests.get", return_value=mock_stream_response()) as mock_get:
219 + with patch(SAFE_GET, return_value=mock_stream_response()) as mock_get:
220 result = rf.buffer()
221
222 assert result == b"file content"
223 mock_get.assert_called_once_with(
224 "https://example.com/file",
225 stream=True,
224 - allow_redirects=False,
226 timeout=(5, 60),
227 )
228
@@ -234,9 +235,7 @@ class TestRemoteFile:
235 download_url="https://example.com/file",
236 )
237 with patch(ASSERT_SAFE_FETCH_TARGET):
237 - with patch(
238 - "requests.get", return_value=mock_stream_response(status_code=404)
239 - ):
238 + with patch(SAFE_GET, return_value=mock_stream_response(status_code=404)):
239 with pytest.raises(RemoteFileDownloadError) as exc_info:
240 rf.buffer()
241
@@ -252,9 +251,7 @@ class TestRemoteFile:
251 download_url="https://example.com/file",
252 )
253 with patch(ASSERT_SAFE_FETCH_TARGET):
255 - with patch(
256 - "requests.get", side_effect=requests.exceptions.Timeout("timeout")
257 - ):
254 + with patch(SAFE_GET, side_effect=requests.exceptions.Timeout("timeout")):
255 with pytest.raises(RemoteFileDownloadError) as exc_info:
256 rf.buffer()
257
@@ -344,11 +341,12 @@ class TestResponseDerivedUrlsAreGuarded:
341 def test_buffer_validates_download_url(self):
342 rf = self._remote_file("https://s3.example.com/download")
343
347 - with patch(ASSERT_SAFE_FETCH_TARGET) as mock_assert:
348 - with patch("requests.get", return_value=mock_stream_response()):
349 - rf.buffer()
344 + with patch(SAFE_GET, return_value=mock_stream_response()) as mock_get:
345 + rf.buffer()
346
351 - mock_assert.assert_called_once_with("https://s3.example.com/download")
347 + # `safe_get` is the guard: it validates the target and then connects to
348 + # the address it validated instead of re-resolving the hostname.
349 + assert mock_get.call_args.args == ("https://s3.example.com/download",)
350
351 def test_buffer_blocked_url_never_reaches_the_network(self):
352 rf = self._remote_file("http://169.254.169.254/latest/meta-data")
@@ -357,11 +355,11 @@ class TestResponseDerivedUrlsAreGuarded:
355 ASSERT_SAFE_FETCH_TARGET,
356 side_effect=BlockedInternalUrlError("blocked"),
357 ):
360 - with patch("requests.get") as mock_get:
358 + with patch(SESSION_REQUEST) as mock_send:
359 with pytest.raises(BlockedInternalUrlError):
360 rf.buffer()
361
364 - mock_get.assert_not_called()
362 + mock_send.assert_not_called()
363
364 def test_buffer_rejects_redirects(self):
365 """A validated URL must not be able to bounce the fetch elsewhere."""
@@ -369,12 +367,14 @@ class TestResponseDerivedUrlsAreGuarded:
367
368 with patch(ASSERT_SAFE_FETCH_TARGET):
369 with patch(
372 - "requests.get", return_value=mock_stream_response(status_code=302)
370 + SAFE_GET, return_value=mock_stream_response(status_code=302)
371 ) as mock_get:
372 with pytest.raises(RemoteFileDownloadError, match="redirect"):
373 rf.buffer()
374
377 - assert mock_get.call_args.kwargs["allow_redirects"] is False
375 + # `safe_get` never follows redirects; passing `allow_redirects` through
376 + # to it would be a way to turn that off.
377 + assert "allow_redirects" not in mock_get.call_args.kwargs
378
379 def test_buffer_tolerates_malformed_content_length(self):
380 """A malformed `Content-Length` means unknown size, not a crash.
@@ -391,7 +391,7 @@ class TestResponseDerivedUrlsAreGuarded:
391 }
392
393 with patch(ASSERT_SAFE_FETCH_TARGET):
394 - with patch("requests.get", return_value=malformed):
394 + with patch(SAFE_GET, return_value=malformed):
395 assert rf.buffer() == b"file content"
396
397 def test_buffer_caps_response_size(self):
@@ -404,7 +404,7 @@ class TestResponseDerivedUrlsAreGuarded:
404 }
405
406 with patch(ASSERT_SAFE_FETCH_TARGET):
407 - with patch("requests.get", return_value=oversized):
407 + with patch(SAFE_GET, return_value=oversized):
408 with pytest.raises(RemoteFileDownloadError, match="exceeds maximum"):
409 rf.buffer()
410
@@ -416,13 +416,13 @@ class TestResponseDerivedUrlsAreGuarded:
416 ASSERT_SAFE_FETCH_TARGET,
417 side_effect=BlockedInternalUrlError("blocked"),
418 ):
419 - with patch("requests.get") as mock_get:
419 + with patch(SESSION_REQUEST) as mock_send:
420 with pytest.raises(BlockedInternalUrlError):
421 rf.text()
422 with pytest.raises(BlockedInternalUrlError):
423 rf.save(str(tmp_path / "out.txt"))
424
425 - mock_get.assert_not_called()
425 + mock_send.assert_not_called()
426 assert not (tmp_path / "out.txt").exists()
427
428 def test_upload_blocked_url_sends_nothing(self, files_mount):
@@ -430,7 +430,7 @@ class TestResponseDerivedUrlsAreGuarded:
430 "composio.utils.url_safety.assert_safe_fetch_target",
431 side_effect=BlockedInternalUrlError("blocked"),
432 ):
433 - with patch("composio.utils.url_safety.requests.request") as mock_request:
433 + with patch(SESSION_REQUEST) as mock_request:
434 with pytest.raises(BlockedInternalUrlError):
435 files_mount.upload(
436 b"hello world",
python/tests/test_url_safety.py
+7 -7
@@ -109,7 +109,7 @@ def _response(status_code: int, location: str | None = None) -> MagicMock:
109 return response
110
111
112 -@patch("composio.utils.url_safety.requests.request")
112 +@patch("composio.utils.url_safety.requests.Session.request")
113 @patch("composio.utils.url_safety.assert_safe_fetch_target")
114 def test_safe_request_validates_before_sending(mock_assert, mock_request) -> None:
115 mock_assert.side_effect = BlockedInternalUrlError("blocked")
@@ -121,7 +121,7 @@ def test_safe_request_validates_before_sending(mock_assert, mock_request) -> Non
121 mock_request.assert_not_called()
122
123
124 -@patch("composio.utils.url_safety.requests.request")
124 +@patch("composio.utils.url_safety.requests.Session.request")
125 @patch("composio.utils.url_safety.assert_safe_fetch_target")
126 def test_safe_request_disables_automatic_redirects(mock_assert, mock_request) -> None:
127 mock_request.return_value = _response(200)
@@ -135,11 +135,11 @@ def test_safe_request_disables_automatic_redirects(mock_assert, mock_request) ->
135 assert mock_request.call_args.kwargs["allow_redirects"] is False
136
137
138 -@patch("composio.utils.url_safety.requests.request")
138 +@patch("composio.utils.url_safety.requests.Session.request")
139 @patch("composio.utils.url_safety.assert_safe_fetch_target")
140 def test_safe_request_revalidates_each_redirect_hop(mock_assert, mock_request) -> None:
141 """A public URL that redirects into private space must be caught at the hop."""
142 - mock_assert.side_effect = [None, BlockedInternalUrlError("blocked")]
142 + mock_assert.side_effect = [["93.184.216.34"], BlockedInternalUrlError("blocked")]
143 mock_request.return_value = _response(
144 307, "http://169.254.169.254/latest/meta-data"
145 )
@@ -155,7 +155,7 @@ def test_safe_request_revalidates_each_redirect_hop(mock_assert, mock_request) -
155 assert mock_request.call_count == 1
156
157
158 -@patch("composio.utils.url_safety.requests.request")
158 +@patch("composio.utils.url_safety.requests.Session.request")
159 @patch("composio.utils.url_safety.assert_safe_fetch_target")
160 def test_safe_request_follows_validated_redirect(mock_assert, mock_request) -> None:
161 """S3 can answer a PUT with a 307 region redirect; that must still work."""
@@ -177,7 +177,7 @@ def test_safe_request_follows_validated_redirect(mock_assert, mock_request) -> N
177 assert body.read() == b"payload"
178
179
180 -@patch("composio.utils.url_safety.requests.request")
180 +@patch("composio.utils.url_safety.requests.Session.request")
181 @patch("composio.utils.url_safety.assert_safe_fetch_target")
182 def test_safe_request_relative_redirect_is_resolved(mock_assert, mock_request) -> None:
183 mock_request.side_effect = [_response(302, "/elsewhere"), _response(200)]
@@ -187,7 +187,7 @@ def test_safe_request_relative_redirect_is_resolved(mock_assert, mock_request) -
187 assert mock_assert.call_args_list[1] == call("https://files.example.com/elsewhere")
188
189
190 -@patch("composio.utils.url_safety.requests.request")
190 +@patch("composio.utils.url_safety.requests.Session.request")
191 @patch("composio.utils.url_safety.assert_safe_fetch_target")
192 def test_safe_request_rejects_endless_redirects(mock_assert, mock_request) -> None:
193 mock_request.return_value = _response(302, "https://s3.example.com/upload")
python/tests/test_url_safety_pinning.py new
+276
@@ -0,0 +1,276 @@
1 +"""Regression tests for the DNS-rebinding window in the SSRF guard.
2 +
3 +The guard resolves a hostname to decide whether a URL is safe. If the HTTP
4 +client then resolves that hostname a *second* time to open the socket, an
5 +attacker who controls the authoritative DNS can answer with a public address
6 +for the check and an internal one for the connect — the check and the use are
7 +about different addresses (issue #4151).
8 +
9 +These tests run against real sockets on loopback rather than mocking the HTTP
10 +client, because a mocked client cannot resolve anything twice and so cannot
11 +express the bug at all. The resolver is the only thing faked: it answers the
12 +first lookup of the hostname with one endpoint and every later lookup with
13 +another, which is exactly what a short-TTL rebinding record does.
14 +"""
15 +
16 +from __future__ import annotations
17 +
18 +import ipaddress
19 +import os
20 +import socket
21 +import threading
22 +import typing as t
23 +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
24 +from unittest.mock import MagicMock, patch
25 +
26 +import pytest
27 +import requests
28 +import urllib3.connection
29 +
30 +from composio.exceptions import BlockedInternalUrlError
31 +from composio.utils.url_safety import _assert_pinned_peer, safe_get
32 +
33 +HOSTNAME = "rebind.test"
34 +
35 +
36 +def _is_ip_literal(host: str) -> bool:
37 + try:
38 + ipaddress.ip_address(host)
39 + except ValueError:
40 + return False
41 + return True
42 +
43 +
44 +class _RecordingServer:
45 + """A loopback HTTP server that records what actually reached it."""
46 +
47 + def __init__(self, body: bytes) -> None:
48 + self.hits: t.List[str] = []
49 + recorder = self
50 +
51 + class Handler(BaseHTTPRequestHandler):
52 + protocol_version = "HTTP/1.1"
53 +
54 + def do_GET(self) -> None: # noqa: N802 - stdlib naming
55 + recorder.hits.append(self.headers.get("Host", ""))
56 + self.send_response(200)
57 + self.send_header("Content-Type", "text/plain")
58 + self.send_header("Content-Length", str(len(body)))
59 + self.end_headers()
60 + self.wfile.write(body)
61 +
62 + def log_message(self, *args: t.Any) -> None:
63 + pass
64 +
65 + self._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
66 + # Keep-alive connections would otherwise hold `shutdown()` open.
67 + self._server.daemon_threads = True
68 + self.port = self._server.server_address[1]
69 + self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
70 + self._thread.start()
71 +
72 + def close(self) -> None:
73 + self._server.shutdown()
74 + self._server.server_close()
75 +
76 +
77 +@pytest.fixture
78 +def validated_server() -> t.Iterator[_RecordingServer]:
79 + server = _RecordingServer(b"public payload")
80 + yield server
81 + server.close()
82 +
83 +
84 +@pytest.fixture
85 +def rebound_server() -> t.Iterator[_RecordingServer]:
86 + server = _RecordingServer(b"internal secret")
87 + yield server
88 + server.close()
89 +
90 +
91 +@pytest.fixture
92 +def no_inherited_proxy(monkeypatch: pytest.MonkeyPatch) -> None:
93 + """Make ``_proxy_applies`` deterministic: no proxy, from env or system.
94 +
95 + An inherited ``HTTP_PROXY`` (or a macOS/Windows system proxy, which
96 + ``getproxies`` also reads) would disable the pinning adapter, and the
97 + request would dial the proxy's answer to the hostname instead of the
98 + validated address — failing these tests on machines that carry one.
99 + """
100 + monkeypatch.setattr(
101 + "requests.utils.get_environ_proxies",
102 + lambda url: {},
103 + )
104 +
105 +
106 +@pytest.fixture
107 +def rebinding_dns(
108 + no_inherited_proxy: None,
109 + validated_server: _RecordingServer,
110 + rebound_server: _RecordingServer,
111 +) -> t.Iterator[None]:
112 + """Answer the first lookup of the hostname benignly, later ones with the victim.
113 +
114 + Both endpoints live on 127.0.0.1 because a test cannot bind a public
115 + address, so they are told apart by port; `is_blocked_ip` is stubbed out for
116 + the same reason. Neither substitution touches the property under test,
117 + which is whether the fetch connects to the endpoint that was validated or
118 + re-resolves and connects somewhere else. The blocklist itself is covered by
119 + ``test_url_safety.py``.
120 + """
121 + lookups: t.List[str] = []
122 + real_getaddrinfo = socket.getaddrinfo
123 +
124 + def fake_getaddrinfo(host, port, *args, **kwargs): # type: ignore[no-untyped-def]
125 + # A real resolver hands an IP literal straight back, which is what the
126 + # fix relies on when it connects to the address it already validated.
127 + if _is_ip_literal(host):
128 + return real_getaddrinfo(host, port, *args, **kwargs)
129 +
130 + lookups.append(host)
131 + victim = rebound_server if len(lookups) > 1 else validated_server
132 + return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("127.0.0.1", victim.port))]
133 +
134 + with patch("socket.getaddrinfo", side_effect=fake_getaddrinfo):
135 + with patch("composio.utils.url_safety.is_blocked_ip", return_value=False):
136 + yield
137 +
138 +
139 +def test_proxy_environment_keeps_only_the_pre_flight_check(
140 + validated_server: _RecordingServer,
141 +) -> None:
142 + """Behind a proxy the SDK cannot pin: the proxy resolves the hostname.
143 +
144 + Documented residual — ``_proxy_applies`` disables the pinning adapter, so
145 + the request dials the proxy. A proxy nothing can reach makes that
146 + observable: the request must fail by trying the proxy, never by dialling
147 + the validated address. (This test deliberately does not use
148 + ``no_inherited_proxy`` — it needs ``HTTP_PROXY`` visible — so it patches
149 + the resolver inline instead.)
150 + """
151 + real_getaddrinfo = socket.getaddrinfo
152 +
153 + def fake_getaddrinfo(host, port, *args, **kwargs): # type: ignore[no-untyped-def]
154 + if _is_ip_literal(host):
155 + return real_getaddrinfo(host, port, *args, **kwargs)
156 + return [
157 + (
158 + socket.AF_INET,
159 + socket.SOCK_STREAM,
160 + 6,
161 + "",
162 + ("127.0.0.1", validated_server.port),
163 + )
164 + ]
165 +
166 + # Empty NO_PROXY so no machine-level bypass list can cover the hostname.
167 + env = {"HTTP_PROXY": "http://127.0.0.1:9", "NO_PROXY": "", "no_proxy": ""}
168 + with patch("socket.getaddrinfo", side_effect=fake_getaddrinfo):
169 + with patch("composio.utils.url_safety.is_blocked_ip", return_value=False):
170 + with patch("composio.utils.url_safety._PinnedAddressAdapter") as adapter:
171 + with patch.dict(os.environ, env):
172 + with pytest.raises(requests.exceptions.RequestException):
173 + safe_get(
174 + f"http://{HOSTNAME}:{validated_server.port}/payload",
175 + timeout=(1, 1),
176 + )
177 +
178 + adapter.assert_not_called()
179 + assert validated_server.hits == []
180 +
181 +
182 +@pytest.mark.usefixtures("rebinding_dns")
183 +def test_fetch_connects_to_the_address_it_validated(
184 + validated_server: _RecordingServer, rebound_server: _RecordingServer
185 +) -> None:
186 + response = safe_get(
187 + f"http://{HOSTNAME}:{validated_server.port}/payload", timeout=(5, 5)
188 + )
189 +
190 + assert response.content == b"public payload"
191 + assert len(validated_server.hits) == 1
192 + # The whole point: the rebound endpoint is never even connected to, so
193 + # neither its body nor a bare TCP connection to it is available to the
194 + # attacker.
195 + assert rebound_server.hits == []
196 +
197 +
198 +@pytest.mark.usefixtures("rebinding_dns")
199 +def test_pinning_keeps_the_hostname_on_the_wire(
200 + validated_server: _RecordingServer,
201 +) -> None:
202 + """The address is pinned; the hostname is not replaced by it.
203 +
204 + Pinning by rewriting the connection's host would send ``Host: 127.0.0.1``
205 + and offer the IP as TLS SNI, which fails certificate verification against
206 + every real origin.
207 + """
208 + safe_get(f"http://{HOSTNAME}:{validated_server.port}/payload", timeout=(5, 5))
209 +
210 + assert validated_server.hits == [f"{HOSTNAME}:{validated_server.port}"]
211 +
212 +
213 +def test_pinning_still_has_the_urllib3_internals_it_relies_on() -> None:
214 + """Fail loudly if a urllib3 upgrade removes what the adapter hooks into.
215 +
216 + Silently losing either of these would silently un-pin every connection.
217 + """
218 + connection = urllib3.connection.HTTPConnection("example.com")
219 +
220 + assert hasattr(connection, "_new_conn")
221 + assert hasattr(connection, "_dns_host")
222 +
223 +
224 +def test_peer_mismatch_fails_closed() -> None:
225 + """The last line of defence, checked before a byte is written to the socket."""
226 + sock = MagicMock()
227 + sock.getpeername.return_value = ("169.254.169.254", 80)
228 +
229 + with pytest.raises(BlockedInternalUrlError, match="169.254.169.254"):
230 + _assert_pinned_peer(sock, "93.184.216.34", HOSTNAME)
231 +
232 + sock.close.assert_called_once()
233 +
234 +
235 +def test_connect_falls_back_to_the_next_validated_address(
236 + no_inherited_proxy: None,
237 + validated_server: _RecordingServer,
238 +) -> None:
239 + """A dual-stack host must not be stranded on its first answer.
240 +
241 + ``::1`` is validated but nothing listens there, so the connect has to move
242 + on to the second answer the way the HTTP client would have. urllib3 rewraps
243 + connect failures into its own exception hierarchy, none of which inherits
244 + from ``OSError``, so this also pins down which exceptions the fallback has
245 + to catch.
246 + """
247 + real_getaddrinfo = socket.getaddrinfo
248 +
249 + def fake_getaddrinfo(host, port, *args, **kwargs): # type: ignore[no-untyped-def]
250 + if _is_ip_literal(host):
251 + return real_getaddrinfo(host, port, *args, **kwargs)
252 + return [
253 + (
254 + socket.AF_INET6,
255 + socket.SOCK_STREAM,
256 + 6,
257 + "",
258 + ("::1", validated_server.port, 0, 0),
259 + ),
260 + (
261 + socket.AF_INET,
262 + socket.SOCK_STREAM,
263 + 6,
264 + "",
265 + ("127.0.0.1", validated_server.port),
266 + ),
267 + ]
268 +
269 + with patch("socket.getaddrinfo", side_effect=fake_getaddrinfo):
270 + with patch("composio.utils.url_safety.is_blocked_ip", return_value=False):
271 + response = safe_get(
272 + f"http://{HOSTNAME}:{validated_server.port}/payload", timeout=(5, 5)
273 + )
274 +
275 + assert response.content == b"public payload"
276 + assert len(validated_server.hits) == 1
ts/packages/core/package.json
+1
@@ -119,6 +119,7 @@
119 "picocolors": "catalog:",
120 "pusher-js": "^8.6.0",
121 "semver": "^7.8.5",
122 + "undici": "^7.29.0",
123 "zod-to-json-schema": "catalog:"
124 },
125 "gitHead": "4fae6e54d5c150fba955cc5fa314281da5a1e064",
ts/packages/core/src/utils/pinnedDispatcher.node.ts new
+118
@@ -0,0 +1,118 @@
1 +import { isIP } from 'node:net';
2 +import type { Agent } from 'undici';
3 +
4 +/**
5 + * The `globalThis` symbols that hold the process-wide dispatcher `fetch` falls
6 + * back to. Every undici in the process shares them — including the one inside
7 + * the Node runtime — and `setGlobalDispatcher` writes both, so reading them
8 + * directly sees the same dispatcher the runtime's own `fetch` would use.
9 + *
10 + * Read rather than imported: importing `undici` installs a dispatcher of its
11 + * own as an import side effect (see {@link loadUndici}), which is exactly what
12 + * this module must not do at load time.
13 + */
14 +const GLOBAL_DISPATCHER_SYMBOLS = [
15 + Symbol.for('undici.globalDispatcher.1'),
16 + Symbol.for('undici.globalDispatcher.2'),
17 +] as const;
18 +
19 +/**
20 + * `undici`, imported on first use rather than at module load.
21 + *
22 + * Importing it runs `if (getGlobalDispatcher() === undefined) setGlobalDispatcher(new Agent())`
23 + * as a side effect, which would hand every unrelated `fetch` in the host
24 + * application this package's undici instead of the runtime's own — a global
25 + * change no caller asked for by importing `@composio/core`. Deferring the
26 + * import to the first pinned request keeps that side effect on the code path
27 + * that actually needs undici, and behind the `hasCustomGlobalDispatcher` check
28 + * that reads the slot's true, pre-import state.
29 + *
30 + * A process that does make a pinned request still ends up on this package's
31 + * `Agent` if nothing had claimed the slot yet. That much is not reversible:
32 + * undici defines the slot non-configurable and writable, so it cannot be
33 + * cleared back to empty — assigning `undefined` leaves the runtime's own
34 + * `fetch` asserting on a missing dispatcher.
35 + */
36 +let undici: Promise<typeof import('undici')> | undefined;
37 +const loadUndici = (): Promise<typeof import('undici')> => (undici ??= import('undici'));
38 +
39 +/**
40 + * A `fetch` dispatcher that connects to `addresses` instead of resolving the
41 + * hostname again.
42 + *
43 + * The SSRF guard resolves a hostname to decide whether a URL is safe to fetch.
44 + * `fetch` would then resolve that hostname a second time when it opens the
45 + * socket, and the two answers need not match: a short-TTL record can answer
46 + * with a public address for the check and an internal one for the connect
47 + * (DNS rebinding). Pinning the connect to the address that was validated
48 + * removes the second lookup, so there is nothing left to rebind.
49 + *
50 + * Only the connect target is replaced. The request still carries the original
51 + * hostname in `Host` and offers it as TLS SNI, so certificate verification is
52 + * unchanged — connecting by IP with the IP as SNI would fail against every
53 + * origin whose certificate names a hostname.
54 + *
55 + * The dispatcher is handed to the runtime's own `fetch`, so callers that stub
56 + * `globalThis.fetch` keep working. That does tie the `undici` major to the one
57 + * Node's `fetch` speaks — undici 8 dispatchers are rejected by the Node
58 + * versions this package supports — which is why the dependency is pinned to
59 + * `^7`, and why `pinnedDispatcher.node.test.ts` exercises a real socket on
60 + * every Node version in CI rather than mocking the transport.
61 + */
62 +export const createPinnedDispatcher = async (addresses: ReadonlyArray<string>): Promise<Agent> => {
63 + const { Agent } = await loadUndici();
64 +
65 + return new Agent({
66 + connect: {
67 + lookup: (_hostname, options, callback) => {
68 + // Node calls a lookup in one of two shapes and picks between them at
69 + // connect time: `(err, addresses)` when it is doing Happy Eyeballs,
70 + // and `(err, address, family)` otherwise — when `autoSelectFamily` is
71 + // off, or a `family`/`localAddress` was requested. Answering in the
72 + // wrong shape is rejected as an invalid address, which would fail
73 + // every pinned fetch in such a process.
74 + if (options.all) {
75 + // All of them, in resolver order: handing over a single address of a
76 + // dual-stack host would strand callers whose network cannot reach
77 + // that family, where the runtime would otherwise have fallen back.
78 + callback(
79 + null,
80 + addresses.map(address => ({ address, family: isIP(address) }))
81 + );
82 + return;
83 + }
84 +
85 + const [address] = addresses;
86 + callback(null, address, isIP(address));
87 + },
88 + },
89 + });
90 +};
91 +
92 +/**
93 + * Whether the process-wide dispatcher `fetch` falls back to (when a request
94 + * carries no `dispatcher` of its own) is something other than a stock `Agent`.
95 + *
96 + * A `ProxyAgent`, `EnvHttpProxyAgent`, or custom dispatcher installed through
97 + * `setGlobalDispatcher` is visible here, and pinning stands down for it: the
98 + * connect would dial the validated origin instead of that route's next hop.
99 + *
100 + * Nothing installed yet is *not* custom. The dispatcher slot is empty until
101 + * either the runtime's `fetch` or an undici import fills it, so an absent
102 + * dispatcher means no caller has configured a route — treating it as custom
103 + * would disable pinning for every process that had not yet made a request.
104 + *
105 + * Stock-ness is compared by constructor *name*, deliberately not with
106 + * `instanceof Agent`: an `Agent` created by the runtime's internal undici is a
107 + * different class from this package's `Agent`, so once the runtime's own
108 + * `fetch` has initialized its default dispatcher, an `instanceof` check would
109 + * report a stock setup as custom and silently disable pinning.
110 + */
111 +export const hasCustomGlobalDispatcher = (): boolean => {
112 + const slots = globalThis as unknown as Record<symbol, { constructor?: { name?: string } }>;
113 + const dispatcher = GLOBAL_DISPATCHER_SYMBOLS.map(symbol => slots[symbol]).find(
114 + installed => installed !== undefined
115 + );
116 +
117 + return dispatcher !== undefined && dispatcher.constructor?.name !== 'Agent';
118 +};
ts/packages/core/src/utils/ssrfGuard.node.ts
+89 -11
@@ -1,6 +1,7 @@
1 import { lookup } from 'node:dns/promises'; // we're in a Node.js-specific module
2 import { isIP } from 'node:net';
3 import { ComposioBlockedInternalUrlError } from '../errors/SsrfErrors';
4 +import { createPinnedDispatcher, hasCustomGlobalDispatcher } from './pinnedDispatcher.node';
5
6 /**
7 * SSRF guard for user-supplied URL file inputs.
@@ -16,14 +17,43 @@ import { ComposioBlockedInternalUrlError } from '../errors/SsrfErrors';
17 * which defeats decimal/octal/hex IP obfuscation) before every fetch, and
18 * follows redirects manually so each hop is re-validated.
19 *
19 - * Known residual: a DNS name could resolve to a public address here and rebind
20 - * to an internal one before the underlying `fetch` connects (a TOCTOU / DNS
21 - * rebinding window). Closing that fully requires pinning the validated IP at
22 - * connect time (a custom dispatcher), which is out of scope for this guard.
20 + * The address that was validated is also the address connected to: `fetch` gets
21 + * a dispatcher pinned to it (see {@link createPinnedDispatcher}), so the
22 + * hostname is never resolved a second time. Without that, a name could resolve
23 + * to a public address during validation and rebind to an internal one before
24 + * the connect — a TOCTOU window the Python guard closes the same way.
25 + *
26 + * Residual: a hop whose effective dispatcher is a configured route — a
27 + * caller-supplied `init.dispatcher`, a non-stock global dispatcher (a
28 + * `ProxyAgent` or `EnvHttpProxyAgent` installed via `setGlobalDispatcher`), or
29 + * the runtime's env-proxy mode (`NODE_USE_ENV_PROXY`) — keeps only the
30 + * pre-flight validation. Pinning would dial the validated address instead of
31 + * the proxy, and the proxy resolves the hostname itself where the SDK cannot
32 + * see or pin that resolution. The Python guard carries the same residual for
33 + * the same reason (`_proxy_applies`).
34 */
35
36 const MAX_REDIRECTS = 5;
37
38 +/**
39 + * Whether the runtime would route `url` through an env proxy the SDK cannot
40 + * see into. Only meaningful when `NODE_USE_ENV_PROXY` opts the built-in
41 + * `fetch` into env proxies (Node >= 24); without that flag the runtime's
42 + * `fetch` ignores `HTTP_PROXY`/`HTTPS_PROXY` entirely, so mirroring Python's
43 + * bare env-var check would drop pinning for users whose fetch never proxied.
44 + *
45 + * `NO_PROXY=*` is the one bypass honored precisely (nothing is proxied, so
46 + * pinning is safe again); per-host `NO_PROXY` entries are treated
47 + * conservatively as proxied rather than parsed.
48 + */
49 +const envProxyApplies = (url: URL): boolean => {
50 + const useEnvProxy = ['1', 'true'].includes((process.env.NODE_USE_ENV_PROXY ?? '').toLowerCase());
51 + if (!useEnvProxy) return false;
52 + if ((process.env.NO_PROXY ?? process.env.no_proxy ?? '') === '*') return false;
53 + const schemeVar = url.protocol === 'https:' ? 'HTTPS_PROXY' : 'HTTP_PROXY';
54 + return Boolean(process.env[schemeVar] ?? process.env[schemeVar.toLowerCase()]);
55 +};
56 +
57 const ipv4ToLong = (ip: string): number => {
58 const [a, b, c, d] = ip.split('.').map(Number);
59 return (((a << 24) >>> 0) | (b << 16) | (c << 8) | d) >>> 0;
@@ -133,8 +163,12 @@ export const isBlockedIp = (ip: string): boolean => {
163 * Validate a single URL: it must be http(s), its host must resolve, and every
164 * resolved address must be publicly routable. Throws
165 * {@link ComposioBlockedInternalUrlError} otherwise.
166 + *
167 + * @returns the validated addresses to connect to, in resolver order. Callers
168 + * must connect to *those* rather than let the client resolve the hostname
169 + * again; see {@link ssrfSafeFetch}.
170 */
137 -export const assertSafeFetchTarget = async (rawUrl: string): Promise<void> => {
171 +export const assertSafeFetchTarget = async (rawUrl: string): Promise<string[]> => {
172 let url: URL;
173 try {
174 url = new URL(rawUrl);
@@ -158,6 +192,10 @@ export const assertSafeFetchTarget = async (rawUrl: string): Promise<void> => {
192 throw new ComposioBlockedInternalUrlError(`Could not resolve host "${host}"`, { url: rawUrl });
193 }
194
195 + if (resolved.length === 0) {
196 + throw new ComposioBlockedInternalUrlError(`Could not resolve host "${host}"`, { url: rawUrl });
197 + }
198 +
199 for (const { address } of resolved) {
200 if (isBlockedIp(address)) {
201 throw new ComposioBlockedInternalUrlError(
@@ -166,13 +204,22 @@ export const assertSafeFetchTarget = async (rawUrl: string): Promise<void> => {
204 );
205 }
206 }
207 +
208 + // Every answer was validated, so all of them are safe to connect to, and
209 + // resolver order is the system's own address preference.
210 + return resolved.map(({ address }) => address);
211 };
212
213 /**
172 - * Drop-in replacement for `fetch` that blocks SSRF. Validates the target before
173 - * connecting and re-validates every redirect hop (redirects are followed
174 - * manually up to {@link MAX_REDIRECTS}). Intermediate redirect bodies are
175 - * cancelled; non-redirect responses are returned unchanged.
214 + * Drop-in replacement for `fetch` that blocks SSRF. Validates the target, then
215 + * connects to the address it validated, and re-validates and re-pins every
216 + * redirect hop (redirects are followed manually up to {@link MAX_REDIRECTS}).
217 + * Intermediate redirect bodies are cancelled; non-redirect responses are
218 + * returned unchanged.
219 + *
220 + * A hop whose effective dispatcher is a configured route (caller-supplied
221 + * `init.dispatcher`, non-stock global dispatcher, env-proxy mode) is *not*
222 + * pinned — see the module residual above.
223 */
224 export const ssrfSafeFetch = async (
225 rawUrl: string,
@@ -182,9 +229,40 @@ export const ssrfSafeFetch = async (
229 let currentUrl = rawUrl;
230
231 for (let hop = 0; hop <= maxRedirects; hop++) {
185 - await assertSafeFetchTarget(currentUrl);
232 + const addresses = await assertSafeFetchTarget(currentUrl);
233 +
234 + // Pinning replaces the connect target; through a configured route that
235 + // would dial the validated origin instead of the route's next hop (the
236 + // proxy), so those hops keep the pre-flight check only.
237 + const callerDispatcher = (init as RequestInit & { dispatcher?: unknown }).dispatcher;
238 + const respectConfiguredRoute =
239 + callerDispatcher !== undefined ||
240 + envProxyApplies(new URL(currentUrl)) ||
241 + hasCustomGlobalDispatcher();
242 +
243 + const dispatcher = respectConfiguredRoute ? undefined : await createPinnedDispatcher(addresses);
244
187 - const response = await fetch(currentUrl, { ...init, redirect: 'manual' });
245 + let response: Response;
246 + try {
247 + // `dispatcher` is a Node-only extension to `RequestInit`.
248 + response = await fetch(
249 + currentUrl,
250 + (dispatcher === undefined
251 + ? { ...init, redirect: 'manual' }
252 + : { ...init, redirect: 'manual', dispatcher }) as RequestInit
253 + );
254 + } catch (error) {
255 + if (dispatcher !== undefined) {
256 + await dispatcher.close().catch(() => undefined);
257 + }
258 + throw error;
259 + }
260 +
261 + // `close()` is graceful: it waits for the in-flight request — including a
262 + // body still being streamed to the caller — before releasing the socket.
263 + if (dispatcher !== undefined) {
264 + void dispatcher.close().catch(() => undefined);
265 + }
266
267 const isRedirect =
268 response.status >= 300 && response.status < 400 && response.headers.has('location');
ts/packages/core/test/models/RemoteFile.test.ts
+7 -4
@@ -98,10 +98,13 @@ describe('RemoteFile', () => {
98 const result = await file.buffer();
99
100 // `redirect: 'manual'` is the guard following redirects itself so it can
101 - // re-validate each hop, rather than letting fetch follow them unchecked.
102 - expect(globalThis.fetch).toHaveBeenCalledWith(validCamelCaseData.downloadUrl, {
103 - redirect: 'manual',
104 - });
101 + // re-validate each hop, rather than letting fetch follow them unchecked;
102 + // the `dispatcher` alongside it pins the connection to the address the
103 + // guard validated.
104 + expect(globalThis.fetch).toHaveBeenCalledWith(
105 + validCamelCaseData.downloadUrl,
106 + expect.objectContaining({ redirect: 'manual' })
107 + );
108 expect(result).toBeInstanceOf(Uint8Array);
109 expect(result).toEqual(content);
110 });
ts/packages/core/test/utils/fileUtils.test.ts
+5 -4
@@ -118,10 +118,11 @@ describe('fileUtils', () => {
118 });
119
120 expect(result.name).toBe('document.pdf');
121 - expect(mockFetch).toHaveBeenCalledWith(urlWithQuery, {
122 - signal: undefined,
123 - redirect: 'manual',
124 - });
121 + expect(mockFetch).toHaveBeenCalledWith(
122 + urlWithQuery,
123 + // Plus a `dispatcher` pinned to the address the guard validated.
124 + expect.objectContaining({ signal: undefined, redirect: 'manual' })
125 + );
126 });
127
128 it('should generate filename when URL has no filename', async () => {
ts/packages/core/test/utils/pinnedDispatcher.node.test.ts new
+124
@@ -0,0 +1,124 @@
1 +import { describe, it, expect, afterEach } from 'vitest';
2 +import { createServer, type Server } from 'node:http';
3 +import { getDefaultAutoSelectFamily, setDefaultAutoSelectFamily, type AddressInfo } from 'node:net';
4 +import { Agent, ProxyAgent, getGlobalDispatcher, setGlobalDispatcher } from 'undici';
5 +import {
6 + createPinnedDispatcher,
7 + hasCustomGlobalDispatcher,
8 +} from '../../src/utils/pinnedDispatcher.node';
9 +
10 +/**
11 + * The SSRF guard resolves a hostname to validate it. If `fetch` then resolves
12 + * that hostname again to open the socket, the two answers can differ — a
13 + * short-TTL record can answer publicly for the check and internally for the
14 + * connect (DNS rebinding, issue #4151).
15 + *
16 + * These tests use a real server and a hostname under `.invalid`, which RFC 2606
17 + * guarantees never resolves. A request that arrives at the server therefore
18 + * proves the connect used the pinned address and never consulted DNS — which is
19 + * exactly the property that closes the rebinding window.
20 + */
21 +describe('createPinnedDispatcher', () => {
22 + let server: Server | undefined;
23 + const hosts: string[] = [];
24 +
25 + const startServer = async (): Promise<number> => {
26 + server = createServer((request, response) => {
27 + hosts.push(request.headers.host ?? '');
28 + response.writeHead(200, { 'content-type': 'text/plain' });
29 + response.end('pinned payload');
30 + });
31 + await new Promise<void>(resolve => server!.listen(0, '127.0.0.1', resolve));
32 + return (server!.address() as AddressInfo).port;
33 + };
34 +
35 + afterEach(async () => {
36 + hosts.length = 0;
37 + if (server) {
38 + await new Promise<void>(resolve => server!.close(() => resolve()));
39 + server = undefined;
40 + }
41 + });
42 +
43 + it('connects to the pinned address instead of resolving the hostname', async () => {
44 + const port = await startServer();
45 + const dispatcher = await createPinnedDispatcher(['127.0.0.1']);
46 +
47 + const response = await fetch(`http://pinned.invalid:${port}/payload`, {
48 + dispatcher,
49 + } as RequestInit);
50 +
51 + expect(response.status).toBe(200);
52 + expect(await response.text()).toBe('pinned payload');
53 + await dispatcher.close();
54 + });
55 +
56 + it('leaves the hostname on the wire, so Host and TLS SNI are unchanged', async () => {
57 + const port = await startServer();
58 + const dispatcher = await createPinnedDispatcher(['127.0.0.1']);
59 +
60 + await fetch(`http://pinned.invalid:${port}/payload`, { dispatcher } as RequestInit);
61 +
62 + // Pinning by rewriting the URL to the IP would send `Host: 127.0.0.1` and
63 + // offer the IP as SNI, failing certificate verification against every
64 + // origin whose certificate names a hostname.
65 + expect(hosts).toEqual([`pinned.invalid:${port}`]);
66 + await dispatcher.close();
67 + });
68 +
69 + it('is the only reason the request lands: unpinned, the hostname does not resolve', async () => {
70 + const port = await startServer();
71 +
72 + await expect(fetch(`http://pinned.invalid:${port}/payload`)).rejects.toThrow();
73 + expect(hosts).toEqual([]);
74 + });
75 +
76 + it('pins just as well with Happy Eyeballs off, which asks for one address', async () => {
77 + // Node calls the lookup with `all: false` here and expects a single
78 + // address rather than a list. A host that disables auto-select-family —
79 + // a common workaround for Happy Eyeballs stalls — would otherwise get
80 + // `ERR_INVALID_IP_ADDRESS` on every SDK fetch.
81 + const port = await startServer();
82 + const previous = getDefaultAutoSelectFamily();
83 + setDefaultAutoSelectFamily(false);
84 +
85 + try {
86 + const dispatcher = await createPinnedDispatcher(['127.0.0.1']);
87 + const response = await fetch(`http://pinned.invalid:${port}/payload`, {
88 + dispatcher,
89 + } as RequestInit);
90 +
91 + expect(response.status).toBe(200);
92 + expect(await response.text()).toBe('pinned payload');
93 + await dispatcher.close();
94 + } finally {
95 + setDefaultAutoSelectFamily(previous);
96 + }
97 + });
98 +});
99 +
100 +describe('hasCustomGlobalDispatcher', () => {
101 + it('reports the stock dispatcher as not custom', () => {
102 + expect(hasCustomGlobalDispatcher()).toBe(false);
103 + });
104 +
105 + it('reports a configured proxy dispatcher as custom', () => {
106 + const previous = getGlobalDispatcher();
107 + setGlobalDispatcher(new ProxyAgent({ uri: 'http://127.0.0.1:9' }));
108 + try {
109 + expect(hasCustomGlobalDispatcher()).toBe(true);
110 + } finally {
111 + setGlobalDispatcher(previous);
112 + }
113 + });
114 +
115 + it('still calls a plain Agent stock, even when created by this package', () => {
116 + const previous = getGlobalDispatcher();
117 + setGlobalDispatcher(new Agent());
118 + try {
119 + expect(hasCustomGlobalDispatcher()).toBe(false);
120 + } finally {
121 + setGlobalDispatcher(previous);
122 + }
123 + });
124 +});
ts/packages/core/test/utils/ssrfGuard.test.ts
+104 -2
@@ -6,6 +6,19 @@ vi.mock('node:dns/promises', () => ({
6 lookup: vi.fn(),
7 }));
8
9 +vi.mock('../../src/utils/pinnedDispatcher.node', () => ({
10 + createPinnedDispatcher: vi.fn(() => Promise.resolve({ close: () => Promise.resolve() })),
11 + hasCustomGlobalDispatcher: vi.fn(() => false),
12 +}));
13 +
14 +import {
15 + createPinnedDispatcher,
16 + hasCustomGlobalDispatcher,
17 +} from '../../src/utils/pinnedDispatcher.node';
18 +
19 +const mockCreatePinnedDispatcher = vi.mocked(createPinnedDispatcher);
20 +const mockHasCustomGlobalDispatcher = vi.mocked(hasCustomGlobalDispatcher);
21 +
22 // eslint-disable-next-line no-restricted-imports
23 import { lookup } from 'node:dns/promises';
24
@@ -97,9 +110,11 @@ describe('assertSafeFetchTarget', () => {
110 );
111 });
112
100 - it('allows a host that resolves only to public addresses', async () => {
113 + it('allows a host that resolves only to public addresses, returning the address to connect to', async () => {
114 resolvesTo('93.184.216.34');
102 - await expect(assertSafeFetchTarget('https://example.com/file.pdf')).resolves.toBeUndefined();
115 + await expect(assertSafeFetchTarget('https://example.com/file.pdf')).resolves.toEqual([
116 + '93.184.216.34',
117 + ]);
118 });
119
120 it('blocks an IPv6 literal loopback host', async () => {
@@ -116,10 +131,97 @@ describe('ssrfSafeFetch', () => {
131 beforeEach(() => {
132 mockLookup.mockReset();
133 mockFetch.mockReset();
134 + mockCreatePinnedDispatcher.mockClear();
135 + mockHasCustomGlobalDispatcher.mockReturnValue(false);
136 + // Deterministic even on machines that carry the runtime env-proxy opt-in.
137 + vi.stubEnv('NODE_USE_ENV_PROXY', '');
138 vi.stubGlobal('fetch', mockFetch);
139 });
140 afterEach(() => vi.unstubAllGlobals());
141
142 + it('connects to the address it validated, rather than resolving the host again', async () => {
143 + resolvesTo('93.184.216.34');
144 + mockFetch.mockResolvedValue(new Response('data', { status: 200 }));
145 +
146 + await ssrfSafeFetch('https://example.com/file.pdf');
147 +
148 + // Without this the host is resolved twice — once to validate, once to
149 + // connect — and a short-TTL record can answer those two lookups
150 + // differently (DNS rebinding, issue #4151).
151 + expect(mockCreatePinnedDispatcher).toHaveBeenCalledWith(['93.184.216.34']);
152 + expect(mockFetch.mock.calls[0][1].dispatcher).toBeDefined();
153 + });
154 +
155 + it("re-pins each redirect hop to that hop's own validated address", async () => {
156 + mockLookup
157 + .mockResolvedValueOnce([{ address: '93.184.216.34', family: 4 }] as never)
158 + .mockResolvedValueOnce([{ address: '151.101.1.140', family: 4 }] as never);
159 + mockFetch
160 + .mockResolvedValueOnce(
161 + new Response(null, { status: 302, headers: { location: 'https://cdn.example.com/f' } })
162 + )
163 + .mockResolvedValueOnce(new Response('data', { status: 200 }));
164 +
165 + await ssrfSafeFetch('https://example.com/file.pdf');
166 +
167 + expect(mockCreatePinnedDispatcher.mock.calls).toEqual([
168 + [['93.184.216.34']],
169 + [['151.101.1.140']],
170 + ]);
171 + });
172 +
173 + it('respects a caller-supplied dispatcher instead of pinning', async () => {
174 + resolvesTo('93.184.216.34');
175 + mockFetch.mockResolvedValue(new Response('data', { status: 200 }));
176 + const callerDispatcher = { close: () => Promise.resolve() };
177 +
178 + await ssrfSafeFetch('https://example.com/file.pdf', {
179 + dispatcher: callerDispatcher,
180 + } as RequestInit);
181 +
182 + // An explicit dispatcher is a routing choice by the caller (e.g. a proxy);
183 + // overriding it would dial the validated origin instead of that route.
184 + expect(mockCreatePinnedDispatcher).not.toHaveBeenCalled();
185 + expect(mockFetch.mock.calls[0][1].dispatcher).toBe(callerDispatcher);
186 + });
187 +
188 + it('does not pin when a non-stock global dispatcher is configured', async () => {
189 + resolvesTo('93.184.216.34');
190 + mockFetch.mockResolvedValue(new Response('data', { status: 200 }));
191 + mockHasCustomGlobalDispatcher.mockReturnValue(true);
192 +
193 + await ssrfSafeFetch('https://example.com/file.pdf');
194 +
195 + // The configured route (a ProxyAgent and friends) resolves the hostname
196 + // itself; fetch falls back to it when no dispatcher is passed.
197 + expect(mockCreatePinnedDispatcher).not.toHaveBeenCalled();
198 + expect(mockFetch.mock.calls[0][1].dispatcher).toBeUndefined();
199 + });
200 +
201 + it('does not pin while the runtime env-proxy mode is active', async () => {
202 + resolvesTo('93.184.216.34');
203 + mockFetch.mockResolvedValue(new Response('data', { status: 200 }));
204 + vi.stubEnv('NODE_USE_ENV_PROXY', '1');
205 + vi.stubEnv('HTTPS_PROXY', 'http://proxy.example:3128');
206 +
207 + await ssrfSafeFetch('https://example.com/file.pdf');
208 +
209 + expect(mockCreatePinnedDispatcher).not.toHaveBeenCalled();
210 + expect(mockFetch.mock.calls[0][1].dispatcher).toBeUndefined();
211 + });
212 +
213 + it('pins again when NO_PROXY=* bypasses every host', async () => {
214 + resolvesTo('93.184.216.34');
215 + mockFetch.mockResolvedValue(new Response('data', { status: 200 }));
216 + vi.stubEnv('NODE_USE_ENV_PROXY', '1');
217 + vi.stubEnv('HTTPS_PROXY', 'http://proxy.example:3128');
218 + vi.stubEnv('NO_PROXY', '*');
219 +
220 + await ssrfSafeFetch('https://example.com/file.pdf');
221 +
222 + expect(mockCreatePinnedDispatcher).toHaveBeenCalledWith(['93.184.216.34']);
223 + });
224 +
225 it('validates and fetches a public URL', async () => {
226 resolvesTo('93.184.216.34');
227 const ok = new Response('data', { status: 200 });
ts/packages/slim/package.json
+1
@@ -112,6 +112,7 @@
112 "picocolors": "catalog:",
113 "pusher-js": "^8.6.0",
114 "semver": "^7.8.5",
115 + "undici": "^7.29.0",
116 "zod-to-json-schema": "catalog:"
117 },
118 "repository": {
uv.lock
+4
@@ -723,7 +723,9 @@ dependencies = [
723 { name = "openai" },
724 { name = "pydantic" },
725 { name = "pysher" },
726 + { name = "requests" },
727 { name = "typing-extensions" },
728 + { name = "urllib3" },
729 ]
730
731 [package.dev-dependencies]
@@ -751,7 +753,9 @@ requires-dist = [
753 { name = "openai", specifier = ">=2.48.0" },
754 { name = "pydantic", specifier = ">=2.11.9" },
755 { name = "pysher", specifier = ">=1.0.8" },
756 + { name = "requests", specifier = ">=2.32.2" },
757 { name = "typing-extensions", specifier = ">=4.16.0" },
758 + { name = "urllib3", specifier = ">=2" },
759 ]
760
761 [package.metadata.requires-dev]