@devries-cameron20 / composio Mirror · read-only / commits / aeeb471c6
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 19:45 UTC

fix(py): resolve $ref/$defs so file flags behind a reference aren't dropped (#3653)

## Problem Tools whose parameter schemas express their file fields through a `$ref`/`$defs` indirection lose auto file handling. With `dangerously_allow_auto_upload_download_files=True`, a `file_downloadable` output behind a `$ref` is never downloaded (the raw `{name, mimetype, s3url}` object is passed through), and a `file_uploadable` input behind a `$ref` is never staged — silently, with no error. `GMAIL_GET_ATTACHMENT` is the canonical shape — the flag sits two `$ref` hops deep: ``` data → $ref → #/$defs/GetAttachmentResponse → file → $ref → #/$defs/FileDownloadable (file_downloadable: true) ``` **Root cause:** the hand-rolled walkers in `composio/core/models/_files.py` (`_has_file_property`, `_substitute_file_upload_value`, `_substitute_file_download_value`, …) recurse through `properties`/`anyOf`/`oneOf`/`allOf`/`items` but **never dereference `$ref`**, so a flagged node reachable only through a reference is invisible to them. Fixes https://github.com/ComposioHQ/composio/issues/3506 ## Approach Rather than teach each walker to dereference — which means threading a root schema through ~10 methods, re-deriving `$ref` handling in each, and risking unbounded recursion on cyclic schemas — this introduces a single **`dereference_json_schema`** utility and inlines references **once at the `FileHelper` boundary** (`process_file_uploadable_schema`, `substitute_file_uploads`, `substitute_file_downloads`). The walkers stay reference-agnostic. This mirrors the TypeScript SDK's `dereferenceJsonSchema` (the counterpart fix in #3566 for the TS twin of this bug, #3307) so both SDKs share one behavioral contract. Resolving once at the boundary also gives every walker `$ref` support for free — including keywords they never special-cased (`additionalProperties`, `patternProperties`, `prefixItems`, `not`), because the resolver walks containers reflectively. ## What's in the utility (`composio/utils/json_schema.py`) Faithful port of the TS guards: - **Cycle safety** — breaks both `$ref` cycles and live-object identity cycles with a permissive `{type: object, additionalProperties: true}` sentinel. (A per-node lazy resolver without threaded cycle tracking infinite-loops on a self-referential schema; there's a regression test for exactly this.) - **Depth caps** — `MAX_REF_CHAIN_DEPTH=100` / `MAX_NODE_DEPTH=512` raise a typed `JSONSchemaRefResolutionError` instead of exhausting the interpreter (with a `RecursionError` backstop converted to the same typed error). - **Sibling-keyword merge** — Draft 2020-12 semantics (siblings win on collision), so a `description`/`default` next to a `$ref` survives. - **External-ref passthrough** — `http(s)://` refs are left untouched and logged once for audit. - **`sentinel` mode** — a dangling `$ref` (an API schema that emits `$ref` with no `$defs` block, https://github.com/ComposioHQ/composio/issues/3307) degrades to the permissive sentinel + an LLM-visible hint, instead of aborting the tool call. The `FileHelper` uses this mode for API-sourced schemas. - **Non-mutating** — returns a new schema; `$defs`/`definitions` are stripped from the inlined root. ## Tests - `python/tests/test_json_schema.py` — 30 cases mirroring the TS `jsonSchema.test.ts` contract (chains, reflective container walk, legacy `definitions`, sibling merge, cycles, depth caps, pointer-escape decoding, array-index pointers, full `sentinel`-mode suite). - `python/tests/test_files.py` — end-to-end regressions through the public methods: download/upload behind `$ref`/`$defs` (the #3506 repro), the LLM-facing input transform, graceful degradation of a dangling `$ref`, and a **self-referential schema that must not recurse infinitely**. - `nox -s chk` (ruff + mypy) green; `nox -s tst` for the file/execution suites green. ## Notes Supersedes #3545 (third-party PR that fixed the same bug by threading `root_schema` through each walker). That diagnosis was correct; this takes the centralized route consistent with the TS SDK and adds the cycle/depth guards. No changeset — Python-package change.

Alberto Schiabel committed Jun 25, 2026 at 00:53 UTC aeeb471c634101abc65c33d2249f9d3c7937bf66
5 files changed +931 -3
python/composio/core/models/_files.py
+28 -3
@@ -24,6 +24,7 @@ from composio.exceptions import (
24 SDKFileNotFoundError,
25 )
26 from composio.utils import mimetypes
27 +from composio.utils.json_schema import dereference_json_schema
28 from composio.utils.sensitive_file_upload_paths import (
29 assert_safe_local_file_upload_path,
30 )
@@ -813,10 +814,23 @@ class FileHelper(WithLogger):
814 Should only be called when the caller opted in via
815 `dangerously_allow_auto_upload_download_files=True`.
816 Recursively handles anyOf, oneOf, allOf, nested properties, and array items.
817 +
818 + ``$ref``/``$defs`` indirection is inlined first so file_uploadable flags
819 + reachable only through a reference are visible to the transform below
820 + (https://github.com/ComposioHQ/composio/issues/3506). Tool schemas come
821 + from the API, which may emit a ``$ref`` without a ``$defs`` block
822 + (https://github.com/ComposioHQ/composio/issues/3307); ``"sentinel"``
823 + degrades that gracefully instead of raising.
824 """
817 - if "properties" not in schema:
825 + resolved = dereference_json_schema(schema, on_unresolved="sentinel")
826 + if "properties" not in resolved:
827 return schema
828
829 + # Preserve the original object's identity (callers such as
830 + # ``process_schema_recursively`` mutate in place) while adopting the
831 + # dereferenced, ``$defs``-stripped shape.
832 + schema.clear()
833 + schema.update(resolved)
834 schema["properties"] = {
835 key: self._transform_schema_for_file_upload(prop)
836 for key, prop in schema["properties"].items()
@@ -1042,7 +1056,12 @@ class FileHelper(WithLogger):
1056 """
1057 return self._substitute_file_uploads_recursively(
1058 tool=tool,
1045 - schema=tool.input_parameters,
1059 + # Inline $ref/$defs once at the boundary so the walker sees
1060 + # file_uploadable flags hidden behind a reference. Non-mutating:
1061 + # ``tool.input_parameters`` is left untouched.
1062 + schema=dereference_json_schema(
1063 + tool.input_parameters, on_unresolved="sentinel"
1064 + ),
1065 request=request,
1066 before_file_upload=before_file_upload,
1067 )
@@ -1160,7 +1179,13 @@ class FileHelper(WithLogger):
1179 "ToolExecutionResponse",
1180 self._substitute_file_downloads_recursively(
1181 tool=tool,
1163 - schema=tool.output_parameters,
1182 + # Inline $ref/$defs once at the boundary so the walker sees
1183 + # file_downloadable flags hidden behind a reference (e.g.
1184 + # GMAIL_GET_ATTACHMENT). Non-mutating: ``tool.output_parameters``
1185 + # is left untouched.
1186 + schema=dereference_json_schema(
1187 + tool.output_parameters, on_unresolved="sentinel"
1188 + ),
1189 request=t.cast(dict, response),
1190 ),
1191 )
python/composio/exceptions.py
+19
@@ -79,6 +79,25 @@ class ValidationError(ComposioError):
79 pass
80
81
82 +class JSONSchemaRefResolutionError(ValidationError):
83 + """Raised when an internal JSON Schema ``$ref`` cannot be inlined.
84 +
85 + Covers malformed pointers, missing ``$defs``/``definitions`` targets
86 + (strict mode only), and ``$ref`` chains or node nesting that exceed the
87 + safety caps in :func:`composio.utils.json_schema.dereference_json_schema`.
88 + """
89 +
90 + def __init__(
91 + self,
92 + message: str,
93 + *args: t.Any,
94 + meta: t.Optional[t.Dict[str, t.Any]] = None,
95 + delegate: bool = False,
96 + ) -> None:
97 + super().__init__(message, *args, delegate=delegate)
98 + self.meta = meta or {}
99 +
100 +
101 class ToolkitError(ComposioError):
102 pass
103
python/composio/utils/json_schema.py new
+257
@@ -0,0 +1,257 @@
1 +"""Inline internal JSON Schema ``$ref`` pointers.
2 +
3 +Python counterpart of the TypeScript SDK's ``dereferenceJsonSchema``
4 +(``ts/packages/core/src/utils/jsonSchema.ts``). Hand-rolled schema walkers in
5 +the SDK (e.g. the file-upload/download substitution in
6 +:mod:`composio.core.models._files`) recurse through ``properties``/``anyOf``/
7 +``oneOf``/``allOf``/``items`` but do not dereference ``$ref``. The Composio API
8 +emits tool parameter schemas with ``$ref``/``$defs`` indirection (``GMAIL_GET_
9 +ATTACHMENT`` is the canonical shape), so a ``file_uploadable``/``file_
10 +downloadable`` flag reachable only through a reference is silently missed.
11 +
12 +Resolving references *once* at the boundary — rather than teaching every walker
13 +to dereference — keeps the walkers reference-agnostic and gives all of them
14 +``$ref`` support for free, including keywords they never special-cased
15 +(``additionalProperties``, ``patternProperties``, ``prefixItems``, ``not``, …),
16 +because :func:`dereference_json_schema` walks containers reflectively.
17 +"""
18 +
19 +from __future__ import annotations
20 +
21 +import typing as t
22 +
23 +from composio.exceptions import JSONSchemaRefResolutionError
24 +from composio.utils.logging import get as get_logger
25 +
26 +logger = get_logger()
27 +
28 +# A ``$ref`` chain longer than this, or object nesting deeper than this, is
29 +# treated as pathological (cyclic data, a billion-laughs-style schema) and
30 +# raises rather than exhausting the interpreter's recursion limit. The caps
31 +# fire in both strict and lenient modes — leniency applies to *unresolvable*
32 +# refs, not to runaway expansion.
33 +MAX_REF_CHAIN_DEPTH = 100
34 +MAX_NODE_DEPTH = 512
35 +
36 +# In-band hint attached to the sentinel when lenient mode stands in for an
37 +# unresolvable ``$ref``. Makes the degradation visible to an LLM reading the
38 +# schema instead of leaving it a bare permissive object. Overridden by a
39 +# caller-provided ``description`` sibling (Draft 2020-12 sibling-keyword merge).
40 +UNRESOLVED_REF_DESCRIPTION = (
41 + "Schema shape unresolved at the source — validate loosely. "
42 + "See https://github.com/ComposioHQ/composio/issues/3307."
43 +)
44 +
45 +# Strategy when an internal ``$ref`` cannot be resolved.
46 +# - ``"throw"`` (default): raise ``JSONSchemaRefResolutionError``. Right for
47 +# first-party / custom-tool schemas where a dangling ``$ref`` is a bug.
48 +# - ``"sentinel"``: replace the node with a permissive object. Right for
49 +# schemas from an upstream service the caller cannot edit (the Composio API
50 +# ships some ``output_parameters`` with a ``$ref`` into ``#/$defs/...`` but no
51 +# ``$defs`` block — see https://github.com/ComposioHQ/composio/issues/3307).
52 +UnresolvedRefStrategy = t.Literal["throw", "sentinel"]
53 +UnresolvedRefReason = t.Literal["missing-target", "malformed-pointer"]
54 +
55 +# Callback invoked once per replaced node in ``"sentinel"`` mode.
56 +OnReplace = t.Callable[[str, "UnresolvedRefReason"], None]
57 +
58 +
59 +def _cycle_break_sentinel() -> t.Dict[str, t.Any]:
60 + """A fresh permissive object used to break cycles / stand in for danglers."""
61 + return {"type": "object", "additionalProperties": True}
62 +
63 +
64 +def _is_mapping(value: t.Any) -> bool:
65 + return isinstance(value, dict)
66 +
67 +
68 +def _decode_pointer_segment(segment: str) -> str:
69 + # Order matters: ``~01`` must decode to ``~1`` (literal), not ``/``.
70 + return segment.replace("~1", "/").replace("~0", "~")
71 +
72 +
73 +class _Resolution(t.NamedTuple):
74 + ok: bool
75 + value: t.Any = None
76 + reason: t.Optional[UnresolvedRefReason] = None
77 + failed_at: t.Optional[str] = None
78 +
79 +
80 +def _try_resolve_pointer(root: t.Dict[str, t.Any], pointer: str) -> _Resolution:
81 + """Walk a local JSON Pointer (``#/a/b/0``) against ``root``.
82 +
83 + Returns a tagged result so the caller can distinguish a malformed pointer
84 + from a structurally valid pointer whose target is absent.
85 + """
86 + if pointer in ("#", ""):
87 + return _Resolution(ok=True, value=root)
88 + if not pointer.startswith("#/"):
89 + return _Resolution(ok=False, reason="malformed-pointer")
90 +
91 + cursor: t.Any = root
92 + for raw_segment in pointer[2:].split("/"):
93 + segment = _decode_pointer_segment(raw_segment)
94 + if isinstance(cursor, list):
95 + try:
96 + index = int(segment)
97 + except ValueError:
98 + return _Resolution(ok=False, reason="missing-target", failed_at=segment)
99 + if index < 0 or index >= len(cursor):
100 + return _Resolution(ok=False, reason="missing-target", failed_at=segment)
101 + cursor = cursor[index]
102 + elif isinstance(cursor, dict):
103 + if segment not in cursor:
104 + return _Resolution(ok=False, reason="missing-target", failed_at=segment)
105 + cursor = cursor[segment]
106 + else:
107 + return _Resolution(ok=False, reason="missing-target", failed_at=segment)
108 + return _Resolution(ok=True, value=cursor)
109 +
110 +
111 +def _raise_resolution_error(pointer: str, resolution: _Resolution) -> t.NoReturn:
112 + if resolution.reason == "malformed-pointer":
113 + raise JSONSchemaRefResolutionError(
114 + f"Unsupported $ref pointer: {pointer}",
115 + meta={"ref": pointer},
116 + )
117 + meta: t.Dict[str, t.Any] = {"ref": pointer}
118 + if resolution.failed_at is not None:
119 + meta["failed_at"] = resolution.failed_at
120 + raise JSONSchemaRefResolutionError(
121 + f"Cannot resolve $ref {pointer}",
122 + meta=meta,
123 + )
124 +
125 +
126 +def dereference_json_schema(
127 + schema: t.Any,
128 + *,
129 + on_unresolved: UnresolvedRefStrategy = "throw",
130 + on_replace: t.Optional[OnReplace] = None,
131 +) -> t.Any:
132 + """Inline internal ``$ref`` pointers (``#/$defs/...`` and legacy
133 + ``#/definitions/...``), returning a new schema; the input is never mutated.
134 +
135 + External refs (``http://``, ``https://``, …) are left untouched (and logged
136 + once for audit, since a downstream resolver fetching them could enable SSRF
137 + or local-file disclosure). Cycles — both ``$ref`` cycles and JS-object-style
138 + identity cycles — are broken with a permissive ``{"type": "object",
139 + "additionalProperties": True}`` sentinel. ``$defs``/``definitions`` are
140 + stripped from the returned root once everything is inlined.
141 +
142 + :param on_unresolved: see :data:`UnresolvedRefStrategy`.
143 + :param on_replace: invoked once per replaced node in ``"sentinel"`` mode,
144 + with the original pointer and the reason it could not be resolved.
145 + :raises JSONSchemaRefResolutionError: on malformed pointers or missing
146 + targets (strict mode only), or chains/nesting past the safety caps
147 + (both modes).
148 + """
149 + if not _is_mapping(schema):
150 + return schema
151 +
152 + root = t.cast(t.Dict[str, t.Any], schema)
153 + # Identity-based cycle guard for the *current* walk path. Python doesn't
154 + # suffer JS-style prototype pollution, so unlike the TS port there's no
155 + # need to filter ``__proto__``/``constructor`` keys while cloning.
156 + visiting: t.Set[int] = set()
157 +
158 + # ``walk`` uses explicit loops (not comprehensions) so each level of schema
159 + # nesting costs exactly one Python stack frame. That keeps ``MAX_NODE_DEPTH``
160 + # the binding limit under the interpreter's default recursion ceiling, so a
161 + # pathologically deep schema fails with our typed error rather than a bare
162 + # ``RecursionError`` (which is also caught at the call site as a backstop).
163 + def walk(
164 + node: t.Any,
165 + visited_refs: t.FrozenSet[str],
166 + chain_depth: int,
167 + node_depth: int,
168 + ) -> t.Any:
169 + if node_depth >= MAX_NODE_DEPTH:
170 + raise JSONSchemaRefResolutionError(
171 + f"JSON Schema node depth exceeded cap ({MAX_NODE_DEPTH})"
172 + )
173 +
174 + if isinstance(node, list):
175 + node_id = id(node)
176 + if node_id in visiting:
177 + return _cycle_break_sentinel()
178 + visiting.add(node_id)
179 + try:
180 + cloned_list: t.List[t.Any] = []
181 + for item in node:
182 + cloned_list.append(
183 + walk(item, visited_refs, chain_depth, node_depth + 1)
184 + )
185 + return cloned_list
186 + finally:
187 + visiting.discard(node_id)
188 +
189 + if not _is_mapping(node):
190 + return node
191 +
192 + node_id = id(node)
193 + if node_id in visiting:
194 + return _cycle_break_sentinel()
195 + visiting.add(node_id)
196 + try:
197 + ref = node.get("$ref")
198 + ref = ref if isinstance(ref, str) else None
199 +
200 + # External refs and non-``$ref`` nodes both pass through the same
201 + # reflective clone path.
202 + if ref is None or not ref.startswith("#"):
203 + if ref is not None:
204 + logger.warning("Leaving external $ref untouched: %s", ref)
205 + cloned: t.Dict[str, t.Any] = {}
206 + for key, value in node.items():
207 + cloned[key] = walk(value, visited_refs, chain_depth, node_depth + 1)
208 + return cloned
209 +
210 + if chain_depth >= MAX_REF_CHAIN_DEPTH:
211 + raise JSONSchemaRefResolutionError(
212 + f"JSON Schema $ref chain exceeded depth cap "
213 + f"({MAX_REF_CHAIN_DEPTH}): {ref}",
214 + meta={"ref": ref},
215 + )
216 + if ref in visited_refs:
217 + return _cycle_break_sentinel()
218 +
219 + resolution = _try_resolve_pointer(root, ref)
220 + if resolution.ok:
221 + target: t.Any = resolution.value
222 + elif on_unresolved == "sentinel":
223 + if on_replace is not None and resolution.reason is not None:
224 + on_replace(ref, resolution.reason)
225 + target = {
226 + **_cycle_break_sentinel(),
227 + "description": UNRESOLVED_REF_DESCRIPTION,
228 + }
229 + else:
230 + _raise_resolution_error(ref, resolution)
231 +
232 + next_refs = visited_refs | {ref}
233 + resolved = walk(target, next_refs, chain_depth + 1, node_depth + 1)
234 +
235 + # Shallow-merge sibling keywords next to ``$ref`` (Draft 2020-12:
236 + # siblings win on collision). Draft 7 ignores them, but the Composio
237 + # tool surface admits both, so we honor siblings for safety.
238 + siblings = {key: value for key, value in node.items() if key != "$ref"}
239 + if not siblings or not _is_mapping(resolved):
240 + return resolved
241 + merged = dict(resolved)
242 + for key, value in siblings.items():
243 + merged[key] = walk(value, visited_refs, chain_depth, node_depth + 1)
244 + return merged
245 + finally:
246 + visiting.discard(node_id)
247 +
248 + try:
249 + out = walk(root, frozenset(), 0, 0)
250 + except RecursionError as exc: # pragma: no cover - depth cap normally fires first
251 + raise JSONSchemaRefResolutionError(
252 + "JSON Schema nesting too deep to dereference"
253 + ) from exc
254 + if _is_mapping(out):
255 + out.pop("$defs", None)
256 + out.pop("definitions", None)
257 + return out
python/tests/test_files.py
+201
@@ -336,6 +336,207 @@ class TestFileHelperSchemaHandling:
336 assert result == {"data": {"value": "test"}}
337
338
339 +class TestFileHelperRefDefsResolution:
340 + """File flags reachable only through a $ref/$defs indirection.
341 +
342 + Regression coverage for
343 + https://github.com/ComposioHQ/composio/issues/3506. References are inlined
344 + once at the public-method boundary (``dereference_json_schema``), so the
345 + walkers themselves stay reference-agnostic.
346 + """
347 +
348 + @patch("composio.core.models._files.FileDownloadable.download")
349 + def test_download_resolves_ref_defs(self, mock_download, file_helper, mock_tool):
350 + """GMAIL_GET_ATTACHMENT shape: file_downloadable two $ref hops deep."""
351 + mock_download.return_value = "/tmp/invoice.pdf"
352 + mock_tool.output_parameters = {
353 + "type": "object",
354 + "$defs": {
355 + "FileDownloadable": {
356 + "type": "object",
357 + "properties": {
358 + "name": {"type": "string"},
359 + "mimetype": {"type": "string"},
360 + "s3url": {"type": "string"},
361 + },
362 + "required": ["name", "mimetype", "s3url"],
363 + "file_downloadable": True,
364 + },
365 + "GetAttachmentResponse": {
366 + "type": "object",
367 + "properties": {
368 + "file": {"$ref": "#/$defs/FileDownloadable"},
369 + "display_url": {"type": "string"},
370 + },
371 + "required": ["file"],
372 + },
373 + },
374 + "properties": {
375 + "data": {"$ref": "#/$defs/GetAttachmentResponse"},
376 + "successful": {"type": "boolean"},
377 + },
378 + "required": ["data", "successful"],
379 + }
380 + response = {
381 + "data": {
382 + "file": {
383 + "name": "invoice.pdf",
384 + "mimetype": "application/pdf",
385 + "s3url": "https://s3.example.com/invoice.pdf",
386 + },
387 + "display_url": "https://mail.google.com/...",
388 + },
389 + "successful": True,
390 + }
391 +
392 + result = file_helper.substitute_file_downloads(
393 + tool=mock_tool,
394 + response=response,
395 + )
396 +
397 + assert result["data"]["file"] == "/tmp/invoice.pdf"
398 + assert result["data"]["display_url"] == "https://mail.google.com/..."
399 + assert result["successful"] is True
400 + mock_download.assert_called_once()
401 +
402 + @patch("composio.core.models._files.FileUploadable.from_path")
403 + def test_upload_resolves_ref_defs(self, mock_from_path, file_helper, mock_tool):
404 + """$ref/$defs input schema exposes a nested file_uploadable leaf."""
405 + upload = MagicMock()
406 + upload.model_dump.return_value = {
407 + "name": "invoice.pdf",
408 + "mimetype": "application/pdf",
409 + "s3key": "uploads/invoice.pdf",
410 + }
411 + mock_from_path.return_value = upload
412 + mock_tool.input_parameters = {
413 + "type": "object",
414 + "$defs": {
415 + "FileUploadable": {"type": "string", "file_uploadable": True},
416 + "AttachmentInput": {
417 + "type": "object",
418 + "properties": {
419 + "file": {"$ref": "#/$defs/FileUploadable"},
420 + "name": {"type": "string"},
421 + },
422 + "required": ["file"],
423 + },
424 + },
425 + "properties": {"attachment": {"$ref": "#/$defs/AttachmentInput"}},
426 + "required": ["attachment"],
427 + }
428 + request = {"attachment": {"file": "/local/invoice.pdf", "name": "invoice.pdf"}}
429 +
430 + result = file_helper.substitute_file_uploads(
431 + tool=mock_tool,
432 + request=request,
433 + )
434 +
435 + assert result["attachment"]["file"] == {
436 + "name": "invoice.pdf",
437 + "mimetype": "application/pdf",
438 + "s3key": "uploads/invoice.pdf",
439 + }
440 + assert result["attachment"]["name"] == "invoice.pdf"
441 + mock_from_path.assert_called_once()
442 +
443 + def test_process_file_uploadable_schema_resolves_ref_defs(self, file_helper):
444 + """The LLM-facing input transform inlines $ref and rewrites the leaf."""
445 + schema = {
446 + "type": "object",
447 + "$defs": {
448 + "FileUploadable": {"type": "string", "file_uploadable": True},
449 + "AttachmentInput": {
450 + "type": "object",
451 + "properties": {"file": {"$ref": "#/$defs/FileUploadable"}},
452 + },
453 + },
454 + "properties": {"attachment": {"$ref": "#/$defs/AttachmentInput"}},
455 + }
456 +
457 + result = file_helper.process_file_uploadable_schema(schema)
458 +
459 + leaf = result["properties"]["attachment"]["properties"]["file"]
460 + assert leaf["type"] == "string"
461 + assert leaf["format"] == "path"
462 + assert leaf["file_uploadable"] is True
463 + # $defs is inlined away; no dangling references remain.
464 + assert "$defs" not in result
465 + assert "$ref" not in result["properties"]["attachment"]
466 +
467 + @patch("composio.core.models._files.FileDownloadable.download")
468 + def test_dangling_ref_download_degrades_gracefully(
469 + self, mock_download, file_helper, mock_tool
470 + ):
471 + """A $ref with no $defs block (issue #3307) must not raise (sentinel)."""
472 + mock_tool.output_parameters = {
473 + "type": "object",
474 + "properties": {
475 + "data": {"$ref": "#/$defs/Missing"},
476 + "successful": {"type": "boolean"},
477 + },
478 + }
479 + response = {"data": {"value": "passthrough"}, "successful": True}
480 +
481 + # No $defs block at all: strict resolution would raise and abort the
482 + # tool call. Sentinel mode keeps execution working — the unresolved
483 + # branch is simply not treated as a file.
484 + result = file_helper.substitute_file_downloads(
485 + tool=mock_tool,
486 + response=response,
487 + )
488 +
489 + assert result == {"data": {"value": "passthrough"}, "successful": True}
490 + mock_download.assert_not_called()
491 +
492 + @patch("composio.core.models._files.FileDownloadable.download")
493 + def test_recursive_ref_schema_does_not_recurse_infinitely(
494 + self, mock_download, file_helper, mock_tool
495 + ):
496 + """A self-referential $ref schema must not blow the stack.
497 +
498 + A per-node lazy resolver without threaded cycle tracking infinite-loops
499 + here; the centralized walker breaks the cycle with a sentinel.
500 + """
501 + mock_download.return_value = "/tmp/node.bin"
502 + mock_tool.output_parameters = {
503 + "type": "object",
504 + "$defs": {
505 + "Node": {
506 + "type": "object",
507 + "properties": {
508 + "file": {
509 + "type": "object",
510 + "properties": {"s3url": {"type": "string"}},
511 + "file_downloadable": True,
512 + },
513 + "child": {"$ref": "#/$defs/Node"},
514 + },
515 + },
516 + },
517 + "properties": {"root": {"$ref": "#/$defs/Node"}},
518 + }
519 + response = {
520 + "root": {
521 + "file": {
522 + "name": "n.bin",
523 + "mimetype": "application/octet-stream",
524 + "s3url": "https://s3/x",
525 + },
526 + "child": {"value": "leaf"},
527 + }
528 + }
529 +
530 + # Must complete without RecursionError.
531 + result = file_helper.substitute_file_downloads(
532 + tool=mock_tool,
533 + response=response,
534 + )
535 +
536 + assert result["root"]["file"] == "/tmp/node.bin"
537 + assert result["root"]["child"] == {"value": "leaf"}
538 +
539 +
540 class TestFileUploadableInUnionTypes:
541 """Test cases for file_uploadable detection in anyOf, oneOf, and allOf schemas."""
542
python/tests/test_json_schema.py new
+426
@@ -0,0 +1,426 @@
1 +"""Tests for :func:`composio.utils.json_schema.dereference_json_schema`.
2 +
3 +Mirrors the TypeScript SDK's ``jsonSchema.test.ts`` so the two SDKs share one
4 +behavioral contract for inlining JSON Schema ``$ref`` pointers.
5 +"""
6 +
7 +import typing as t
8 +from unittest.mock import patch
9 +
10 +import pytest
11 +
12 +from composio.exceptions import JSONSchemaRefResolutionError
13 +from composio.utils import json_schema
14 +from composio.utils.json_schema import (
15 + MAX_REF_CHAIN_DEPTH,
16 + UNRESOLVED_REF_DESCRIPTION,
17 + dereference_json_schema,
18 +)
19 +
20 +
21 +def _contains_ref(value: t.Any) -> bool:
22 + if isinstance(value, dict):
23 + if "$ref" in value:
24 + return True
25 + return any(_contains_ref(v) for v in value.values())
26 + if isinstance(value, list):
27 + return any(_contains_ref(v) for v in value)
28 + return False
29 +
30 +
31 +PERMISSIVE: t.Dict[str, t.Any] = {"type": "object", "additionalProperties": True}
32 +PERMISSIVE_WITH_HINT: t.Dict[str, t.Any] = {
33 + **PERMISSIVE,
34 + "description": UNRESOLVED_REF_DESCRIPTION,
35 +}
36 +
37 +
38 +class TestDereferenceJsonSchema:
39 + def test_inlines_single_internal_ref(self):
40 + out = dereference_json_schema(
41 + {
42 + "type": "object",
43 + "properties": {"user": {"$ref": "#/$defs/User"}},
44 + "required": ["user"],
45 + "$defs": {"User": {"type": "string"}},
46 + }
47 + )
48 + assert out == {
49 + "type": "object",
50 + "properties": {"user": {"type": "string"}},
51 + "required": ["user"],
52 + }
53 + assert _contains_ref(out) is False
54 +
55 + def test_resolves_chain_a_b_c(self):
56 + out = dereference_json_schema(
57 + {
58 + "type": "object",
59 + "properties": {"v": {"$ref": "#/$defs/A"}},
60 + "$defs": {
61 + "A": {"$ref": "#/$defs/B"},
62 + "B": {"$ref": "#/$defs/C"},
63 + "C": {"type": "integer"},
64 + },
65 + }
66 + )
67 + assert _contains_ref(out) is False
68 + assert out["properties"]["v"] == {"type": "integer"}
69 +
70 + def test_walks_containers_reflectively(self):
71 + """Refs are resolved in keywords the file walkers never special-cased."""
72 + out = dereference_json_schema(
73 + {
74 + "type": "object",
75 + "properties": {
76 + "a": {"type": "array", "items": {"$ref": "#/$defs/Leaf"}},
77 + "b": {"oneOf": [{"$ref": "#/$defs/Leaf"}, {"type": "null"}]},
78 + "c": {"anyOf": [{"$ref": "#/$defs/Leaf"}]},
79 + "d": {"allOf": [{"$ref": "#/$defs/Leaf"}]},
80 + "e": {"not": {"$ref": "#/$defs/Leaf"}},
81 + "f": {
82 + "type": "object",
83 + "additionalProperties": {"$ref": "#/$defs/Leaf"},
84 + },
85 + "g": {
86 + "type": "object",
87 + "patternProperties": {"^x_": {"$ref": "#/$defs/Leaf"}},
88 + },
89 + "h": {"type": "array", "prefixItems": [{"$ref": "#/$defs/Leaf"}]},
90 + },
91 + "$defs": {"Leaf": {"type": "string"}},
92 + }
93 + )
94 + assert _contains_ref(out) is False
95 +
96 + def test_resolves_legacy_definitions(self):
97 + out = dereference_json_schema(
98 + {
99 + "type": "object",
100 + "properties": {"name": {"$ref": "#/definitions/Name"}},
101 + "definitions": {"Name": {"type": "string", "minLength": 1}},
102 + }
103 + )
104 + assert _contains_ref(out) is False
105 + assert out["properties"]["name"] == {"type": "string", "minLength": 1}
106 +
107 + def test_mixes_defs_and_definitions_transitively(self):
108 + out = dereference_json_schema(
109 + {
110 + "type": "object",
111 + "properties": {"v": {"$ref": "#/definitions/A"}},
112 + "definitions": {"A": {"$ref": "#/$defs/B"}},
113 + "$defs": {"B": {"type": "boolean"}},
114 + }
115 + )
116 + assert _contains_ref(out) is False
117 + assert out["properties"]["v"] == {"type": "boolean"}
118 +
119 + def test_sibling_keywords_win_on_collision(self):
120 + out = dereference_json_schema(
121 + {
122 + "type": "object",
123 + "properties": {
124 + "v": {
125 + "$ref": "#/$defs/Foo",
126 + "description": "caller override",
127 + "default": None,
128 + },
129 + },
130 + "$defs": {
131 + "Foo": {"type": "string", "description": "target description"}
132 + },
133 + }
134 + )
135 + v = out["properties"]["v"]
136 + assert v["type"] == "string"
137 + assert v["description"] == "caller override"
138 + assert v["default"] is None
139 + assert _contains_ref(out) is False
140 +
141 + def test_breaks_recursive_ref_cycles_with_sentinel(self):
142 + out = dereference_json_schema(
143 + {
144 + "$ref": "#/$defs/Tree",
145 + "$defs": {
146 + "Tree": {
147 + "type": "object",
148 + "properties": {
149 + "children": {
150 + "type": "array",
151 + "items": {"$ref": "#/$defs/Tree"},
152 + },
153 + },
154 + },
155 + },
156 + }
157 + )
158 + assert out["type"] == "object"
159 + assert out["properties"]["children"]["items"] == PERMISSIVE
160 + assert _contains_ref(out) is False
161 +
162 + def test_strips_defs_and_definitions_from_root(self):
163 + out = dereference_json_schema(
164 + {
165 + "type": "object",
166 + "properties": {"x": {"$ref": "#/$defs/Foo"}},
167 + "$defs": {"Foo": {"type": "integer"}},
168 + "definitions": {"Bar": {"type": "string"}},
169 + }
170 + )
171 + assert "$defs" not in out
172 + assert "definitions" not in out
173 +
174 + def test_throws_when_target_missing(self):
175 + with pytest.raises(JSONSchemaRefResolutionError):
176 + dereference_json_schema(
177 + {
178 + "type": "object",
179 + "properties": {"v": {"$ref": "#/$defs/Missing"}},
180 + "$defs": {},
181 + }
182 + )
183 +
184 + def test_throws_when_chain_depth_exceeds_cap(self):
185 + defs: t.Dict[str, t.Any] = {}
186 + for i in range(200):
187 + target = f"#/$defs/A{i + 1}" if i + 1 < 200 else "#/$defs/A0"
188 + defs[f"A{i}"] = {"$ref": target}
189 + with pytest.raises(JSONSchemaRefResolutionError):
190 + dereference_json_schema(
191 + {
192 + "type": "object",
193 + "properties": {"v": {"$ref": "#/$defs/A0"}},
194 + "$defs": defs,
195 + }
196 + )
197 +
198 + def test_throws_on_pathologically_deep_nesting(self):
199 + leaf: t.Dict[str, t.Any] = {"type": "string"}
200 + for _ in range(1000):
201 + leaf = {"type": "object", "properties": {"x": leaf}}
202 + with pytest.raises(JSONSchemaRefResolutionError):
203 + dereference_json_schema(leaf)
204 +
205 + def test_breaks_identity_cycles_without_ref(self):
206 + """A live-object cycle that does not flow through a ``$ref``."""
207 + root: t.Dict[str, t.Any] = {"type": "object", "properties": {}}
208 + root["properties"]["self"] = root
209 + out = dereference_json_schema(root)
210 + assert out["properties"]["self"] == PERMISSIVE
211 +
212 + def test_leaves_external_ref_untouched_and_warns(self):
213 + with patch.object(json_schema.logger, "warning") as warn:
214 + out = dereference_json_schema(
215 + {
216 + "type": "object",
217 + "properties": {"v": {"$ref": "https://example.com/Foo"}},
218 + }
219 + )
220 + assert out["properties"]["v"]["$ref"] == "https://example.com/Foo"
221 + warn.assert_called_once()
222 + assert "https://example.com/Foo" in str(warn.call_args)
223 +
224 + def test_does_not_mutate_input(self):
225 + import copy
226 +
227 + schema = {
228 + "type": "object",
229 + "properties": {"v": {"$ref": "#/$defs/Foo"}},
230 + "$defs": {"Foo": {"type": "string"}},
231 + }
232 + snapshot = copy.deepcopy(schema)
233 + dereference_json_schema(schema)
234 + assert schema == snapshot
235 +
236 + def test_decodes_json_pointer_escapes(self):
237 + out = dereference_json_schema(
238 + {
239 + "type": "object",
240 + "properties": {
241 + "a": {"$ref": "#/$defs/with~1slash"},
242 + "b": {"$ref": "#/$defs/with~0tilde"},
243 + "c": {"$ref": "#/$defs/tilde-then-slash~01"},
244 + },
245 + "$defs": {
246 + "with/slash": {"type": "string"},
247 + "with~tilde": {"type": "integer"},
248 + "tilde-then-slash~1": {"type": "boolean"},
249 + },
250 + }
251 + )
252 + props = out["properties"]
253 + assert props["a"]["type"] == "string"
254 + assert props["b"]["type"] == "integer"
255 + assert props["c"]["type"] == "boolean"
256 +
257 + def test_resolves_array_index_pointers(self):
258 + out = dereference_json_schema(
259 + {
260 + "type": "object",
261 + "properties": {"v": {"$ref": "#/$defs/foo/oneOf/0"}},
262 + "$defs": {"foo": {"oneOf": [{"type": "string"}, {"type": "number"}]}},
263 + }
264 + )
265 + assert out["properties"]["v"]["type"] == "string"
266 +
267 + def test_passthrough_for_non_dict_input(self):
268 + assert dereference_json_schema("not-a-schema") == "not-a-schema"
269 + assert dereference_json_schema(None) is None
270 +
271 +
272 +class TestSentinelMode:
273 + def test_replaces_dangling_ref_with_no_defs_block(self):
274 + """Mirrors GMAIL_FETCH_EMAILS: a ``$ref`` into ``#/$defs`` with no defs."""
275 + out = dereference_json_schema(
276 + {
277 + "type": "object",
278 + "properties": {
279 + "data": {"$ref": "#/$defs/FetchEmailsResponse"},
280 + "error": {"type": "string"},
281 + "successful": {"type": "boolean"},
282 + },
283 + "required": ["data", "successful"],
284 + "title": "FetchEmailsResponseWrapper",
285 + },
286 + on_unresolved="sentinel",
287 + )
288 + assert out["properties"]["data"] == PERMISSIVE_WITH_HINT
289 + assert out["properties"]["error"] == {"type": "string"}
290 + assert out["properties"]["successful"] == {"type": "boolean"}
291 + assert out["required"] == ["data", "successful"]
292 + assert _contains_ref(out) is False
293 +
294 + def test_replaces_ref_into_empty_defs(self):
295 + out = dereference_json_schema(
296 + {
297 + "type": "object",
298 + "properties": {"v": {"$ref": "#/$defs/Missing"}},
299 + "$defs": {},
300 + },
301 + on_unresolved="sentinel",
302 + )
303 + assert out["properties"]["v"] == PERMISSIVE_WITH_HINT
304 + assert _contains_ref(out) is False
305 +
306 + def test_on_replace_called_once_per_replacement(self):
307 + calls: t.List[t.Tuple[str, str]] = []
308 + dereference_json_schema(
309 + {
310 + "type": "object",
311 + "properties": {
312 + "a": {"$ref": "#/$defs/Missing"},
313 + "b": {"$ref": "#/$defs/Missing"},
314 + "c": {"$ref": "#/$defs/AlsoMissing"},
315 + },
316 + },
317 + on_unresolved="sentinel",
318 + on_replace=lambda ref, reason: calls.append((ref, reason)),
319 + )
320 + assert calls == [
321 + ("#/$defs/Missing", "missing-target"),
322 + ("#/$defs/Missing", "missing-target"),
323 + ("#/$defs/AlsoMissing", "missing-target"),
324 + ]
325 +
326 + def test_on_replace_malformed_pointer(self):
327 + calls: t.List[t.Tuple[str, str]] = []
328 + out = dereference_json_schema(
329 + {"type": "object", "properties": {"v": {"$ref": "#bar"}}},
330 + on_unresolved="sentinel",
331 + on_replace=lambda ref, reason: calls.append((ref, reason)),
332 + )
333 + assert calls == [("#bar", "malformed-pointer")]
334 + assert out["properties"]["v"] == PERMISSIVE_WITH_HINT
335 +
336 + def test_still_throws_on_chain_depth_cap(self):
337 + defs: t.Dict[str, t.Any] = {}
338 + for i in range(200):
339 + target = f"#/$defs/A{i + 1}" if i + 1 < 200 else "#/$defs/A0"
340 + defs[f"A{i}"] = {"$ref": target}
341 + with pytest.raises(JSONSchemaRefResolutionError):
342 + dereference_json_schema(
343 + {
344 + "type": "object",
345 + "properties": {"v": {"$ref": "#/$defs/A0"}},
346 + "$defs": defs,
347 + },
348 + on_unresolved="sentinel",
349 + )
350 +
351 + def test_still_throws_on_node_depth_cap(self):
352 + leaf: t.Dict[str, t.Any] = {"type": "string"}
353 + for _ in range(1000):
354 + leaf = {"type": "object", "properties": {"x": leaf}}
355 + with pytest.raises(JSONSchemaRefResolutionError):
356 + dereference_json_schema(leaf, on_unresolved="sentinel")
357 +
358 + def test_explicit_throw_matches_default(self):
359 + with pytest.raises(JSONSchemaRefResolutionError):
360 + dereference_json_schema(
361 + {
362 + "type": "object",
363 + "properties": {"v": {"$ref": "#/$defs/Missing"}},
364 + "$defs": {},
365 + },
366 + on_unresolved="throw",
367 + )
368 +
369 + def test_does_not_call_on_replace_in_strict_mode(self):
370 + calls: t.List[t.Any] = []
371 + with pytest.raises(JSONSchemaRefResolutionError):
372 + dereference_json_schema(
373 + {
374 + "type": "object",
375 + "properties": {"v": {"$ref": "#/$defs/Missing"}},
376 + "$defs": {},
377 + },
378 + on_unresolved="throw",
379 + on_replace=lambda ref, reason: calls.append((ref, reason)),
380 + )
381 + assert calls == []
382 +
383 + def test_injects_default_description_when_no_sibling(self):
384 + out = dereference_json_schema(
385 + {"type": "object", "properties": {"v": {"$ref": "#/$defs/Missing"}}},
386 + on_unresolved="sentinel",
387 + )
388 + description = out["properties"]["v"]["description"]
389 + assert "Schema shape unresolved at the source" in description
390 + assert "https://github.com/ComposioHQ/composio/issues/3307" in description
391 +
392 + def test_preserves_caller_description_sibling(self):
393 + out = dereference_json_schema(
394 + {
395 + "type": "object",
396 + "properties": {
397 + "v": {
398 + "$ref": "#/$defs/Missing",
399 + "description": "caller-supplied prose context",
400 + },
401 + },
402 + },
403 + on_unresolved="sentinel",
404 + )
405 + assert out["properties"]["v"]["description"] == "caller-supplied prose context"
406 + assert out["properties"]["v"]["type"] == "object"
407 +
408 + def test_preserves_resolvable_defs_while_replacing_dangling_branch(self):
409 + out = dereference_json_schema(
410 + {
411 + "type": "object",
412 + "properties": {
413 + "resolved": {"$ref": "#/$defs/Real"},
414 + "dangling": {"$ref": "#/$defs/Ghost"},
415 + },
416 + "$defs": {"Real": {"type": "integer"}},
417 + },
418 + on_unresolved="sentinel",
419 + )
420 + assert out["properties"]["resolved"] == {"type": "integer"}
421 + assert out["properties"]["dangling"] == PERMISSIVE_WITH_HINT
422 +
423 +
424 +def test_max_ref_chain_depth_is_exposed():
425 + """Guard against accidental removal of the safety cap constant."""
426 + assert MAX_REF_CHAIN_DEPTH == 100