@devries-cameron20 / composio-1 / commits / d9f6291f4
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

fix(sdk): run schema regex patterns on a linear-time engine (#4765)

## Summary Fixes a ReDoS in both SDKs. Tool schemas and the tool arguments matched against them are both untrusted, but a schema `pattern` or `patternProperties` key such as `^(a+)+$` used to run on a backtracking regex engine. A short argument could therefore freeze the worker. The ask was to make schema regex evaluation safe in both the TypeScript and Python SDKs. Designated PR reviewer: [@jkomyno](mailto:alberto@composio.dev). ## Changes **TypeScript (`@composio/json-schema-to-zod`)** - New `utils/linear-pattern.ts` translates ECMA `u`-mode patterns to RE2 syntax and compiles them with `re2js`, a pure-JS RE2 that runs in linear time. - `compilePattern` returns a `SchemaPatternRegExp`. Its `test`/`exec` run on RE2, while `.source` is unchanged so JSON-schema emission still works. - Patterns RE2 can't run (look-around, backreferences) fall back to native `RegExp` only when `isSafeForBacktracking` passes, and only on input of 1000 characters or fewer. Otherwise the pattern is not enforced. Widening is safe because a hostile schema could simply omit the pattern. - `patternProperties` keys must compile on RE2. Otherwise conversion throws `InvalidPatternError` with the new reason `'unsupported'`. - The cfworker whole-schema validator compiles `patternProperties` keys natively on every validation. It now gets per-call spellings: each key is rewritten as an anchored literal trie of the instance keys that RE2 matches. A schema regex therefore never reaches the native engine. - New dependency: `re2js`. **Python (`composio.utils.schema_converter`)** - Every schema regex goes through pydantic-core's linear-time Rust regex. The guarded `re` fallback uses the same static check as TS, and `patternProperties` keys follow the same linear-only rule (unsupported keys raise `ValueError`). - Paths covered: - the converter's own matchers - object policies - typeless constraints - the jsonschema `pattern`, `patternProperties`, `additionalProperties` and `unevaluatedProperties` keywords - subschemas that switch dialect via `$schema` **The shared static check behind the fallback** rejects: - backreferences - quantified groups that contain a quantifier or alternation - nested look-around - polynomial degree above 2 - a product of small-quantifier spans and alternation widths above 64 ## Behavior changes - Look-around or backreference `pattern`s that fail the safety check are no longer enforced (Python logs a warning). Safe ones are enforced on input up to 1000 chars. - Look-around or backreference `patternProperties` keys now fail conversion. - TS: astral characters now match as single code points (u-mode semantics). ## Type of change - [x] Bug fix - [ ] New feature - [ ] Refactor/Chore - [ ] Documentation - [ ] Breaking change ## How Has This Been Tested? **TypeScript** - `pnpm --filter @composio/json-schema-to-zod test`: all pass, including the new `linear-pattern.test.ts` and the existing AJV property-semantics and regression suites. - The new tests include: - fast-check parity of RE2 against native `u`-mode over tricky patterns and code units - accept/reject tables for the static check - `<1s` timing tests for `^(a+)+$` and `^(a|a)*$`, through the native, `$ref` guard-only and `propertyNames` paths - Package typecheck, oxlint and the tsdown build (attw/publint) pass. `@composio/core` utils tests and typecheck pass, and so do the effect-schema tests. **Python** - `pytest tests/test_schema*.py`: 563 passed. - The new `tests/test_schema_pattern_redos.py` has 35 tests. They cover timing for every sink, enforcement versus non-enforcement of the fallback, and the static-check tables. - Full suite: one failure, `test_path_join_guardrail.py::test_module_constant_exemption_requires_a_literal`, which also fails on the base commit without this change. - `make chk` (ruff + mypy) passes. ## Checklist - [ ] I have read the Code of Conduct and this PR adheres to it - [ ] I have read and understand every line of this diff, including any LLM-generated code - [ ] Any third-party docs links or promotional content follow the [third-party links policy](https://github.com/ComposioHQ/composio/blob/next/CONTRIBUTING.md#third-party-links-in-docs) - [x] I ran linters/tests locally and they passed - [ ] I updated documentation as needed - [x] I added tests or explain why not applicable - [x] I added a changeset if this change affects published packages ## Additional context Follow-up: the private `json-schema-to-effect-schema` CLI package still evaluates schema patterns with native regexes. It isn't covered here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Srujan (in pst) [zen] <srujan@composio.dev> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: jkomyno <alberto@composio.dev>

composio-zen[bot] committed Oct 8, 2026 at 16:21 UTC d9f6291f43a1fd2635cfef4d87f638e29f94bf0b
9 files changed +2209 -181
.changeset/schema-pattern-redos.md new
+6
@@ -0,0 +1,6 @@
1 +---
2 +'@composio/json-schema-to-zod': patch
3 +'@composio/core': patch
4 +---
5 +
6 +Tool schema `pattern` and `patternProperties` regexes now run on a linear-time engine (RE2, via `re2js`), so a hostile pattern such as `^(a+)+$` can no longer hang argument validation. A `pattern` that needs lookaround still runs on the native engine when a static check shows it cannot backtrack catastrophically, on input up to 1000 characters; otherwise it is left unenforced. A `patternProperties` key that needs lookaround or a backreference now fails conversion with an `InvalidPatternError` (reason `unsupported`). Patterns now match astral characters (emoji) as single characters, as JSON Schema's Unicode regex dialect does.
pnpm-lock.yaml
+9
@@ -1480,6 +1480,9 @@ importers:
1480 dequal:
1481 specifier: ^2.0.3
1482 version: 2.0.3
1483 + re2js:
1484 + specifier: ^2.8.6
1485 + version: 2.8.6
1486 devDependencies:
1487 '@types/json-schema':
1488 specifier: ^7.0.15
@@ -5942,6 +5945,10 @@ packages:
5945 resolution: {integrity: sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==}
5946 engines: {node: '>= 0.10'}
5947
5948 + re2js@2.8.6:
5949 + resolution: {integrity: sha512-xLgQil4kIUCrAzVk9fRSkxkFNwmygLFjVxXrLc65aE1F0+Zsb8rxumFBy4XKyvgMCTL6kilDq3EZ0piE2dP/Dg==}
5950 + engines: {node: '>=18.0.0'}
5951 +
5952 remark-gfm@4.0.1:
5953 resolution: {integrity: sha512-1quofZ2RQ9EWdeN34S79+KExV1764+wCUGop5CPL1WGdD0ocPpu91lzPGbwWMECpEpd42kJGQwzRfyov9j4yNg==}
5954
@@ -10977,6 +10984,8 @@ snapshots:
10984 iconv-lite: 0.7.3
10985 unpipe: 1.0.0
10986
10987 + re2js@2.8.6: {}
10988 +
10989 remark-gfm@4.0.1:
10990 dependencies:
10991 '@types/mdast': 4.0.4
python/composio/utils/schema_converter.py
+568 -102
@@ -10,6 +10,8 @@ We handle those by pre-filtering them before passing to the library.
10 """
11
12 import decimal
13 +import functools
14 +import importlib
15 import re
16 import types
17 import typing as t
@@ -435,7 +437,9 @@ class _ObjectPolicy(t.NamedTuple):
437 """
438
439 declared: t.FrozenSet[str]
438 - patterns: t.Tuple[t.Tuple[t.Pattern[str], _DynamicKeyValidator], ...]
440 + # Key matchers come from `_linear_pattern_matcher`, never Python `re`: keys
441 + # are attacker-influenced and the schema's patterns are untrusted.
442 + patterns: t.Tuple[t.Tuple[t.Callable[[str], bool], _DynamicKeyValidator], ...]
443 additional: t.Optional[_DynamicKeyValidator]
444 rejects_unmatched: bool
445
@@ -813,13 +817,32 @@ def _dynamic_key_validator(
817 )
818
819
816 -def _compile_pattern_property(pattern: str) -> t.Pattern[str]:
820 +def _compile_pattern_property(pattern: str) -> t.Callable[[str], bool]:
821 + """Compile a `patternProperties` key on the linear-time engine only.
822 +
823 + A key pattern decides which subschema validates a value, so it cannot be
824 + dropped the way an unenforceable `pattern` constraint is. Patterns that
825 + engine rejects (look-around, backreferences) fail conversion instead of
826 + running on Python's backtracking `re` against attacker-chosen keys.
827 + """
828 + matcher = _linear_pattern_matcher(pattern)
829 + if matcher is not None:
830 + return matcher
831 try:
818 - return re.compile(pattern)
819 - except re.error as exc:
832 + re.compile(pattern)
833 + except (re.error, OverflowError, RecursionError, TypeError) as exc:
834 raise ValueError(
835 f"Invalid patternProperties regular expression: {pattern!r}"
836 ) from exc
837 + raise ValueError(_unsupported_pattern_property_message(pattern))
838 +
839 +
840 +def _unsupported_pattern_property_message(pattern: t.Any) -> str:
841 + return (
842 + f"Unsupported patternProperties regular expression: {pattern!r}; "
843 + "only patterns the linear-time regex engine accepts are supported "
844 + "(look-around and backreferences can backtrack catastrophically)"
845 + )
846
847
848 def _validate_object_policy_schema(
@@ -827,9 +850,11 @@ def _validate_object_policy_schema(
850 root_schema: t.Dict[str, t.Any],
851 ) -> None:
852 """Validate a dynamic object policy without materializing defaults."""
830 - validator_type = jsonschema_validators.validator_for(
831 - root_schema,
832 - default=jsonschema_validators.Draft7Validator,
853 + validator_type = _with_safe_regex_keywords(
854 + jsonschema_validators.validator_for(
855 + root_schema,
856 + default=jsonschema_validators.Draft7Validator,
857 + )
858 )
859 root_validator = validator_type(root_schema)
860
@@ -853,9 +878,11 @@ def _compile_object_policy(
878 document_root = root_schema if root_schema is not None else schema
879 declared = frozenset(schema.get("properties") or {})
880 additional = schema.get("additionalProperties", _MISSING)
856 - validator_type = jsonschema_validators.validator_for(
857 - document_root,
858 - default=jsonschema_validators.Draft7Validator,
881 + validator_type = _with_safe_regex_keywords(
882 + jsonschema_validators.validator_for(
883 + document_root,
884 + default=jsonschema_validators.Draft7Validator,
885 + )
886 )
887 root_validator = validator_type(document_root)
888
@@ -888,18 +915,32 @@ def _compile_object_policy(
915 def _iter_reachable_schemas(
916 schema: t.Any,
917 root_schema: t.Dict[str, t.Any],
891 - visited: t.Optional[t.Set[int]] = None,
918 + visited: t.Optional[t.Set[t.Tuple[int, t.Any]]] = None,
919 + validator_class: t.Any = None,
920 ) -> t.Iterator[t.Dict[str, t.Any]]:
893 - """Yield schema nodes reached through Draft 7 schema-valued keywords."""
921 + """Yield schema nodes and local targets under their effective dialect."""
922 if visited is None:
923 visited = set()
924 + if validator_class is None:
925 + validator_class = jsonschema_validators.validator_for(
926 + root_schema, default=jsonschema_validators.Draft7Validator
927 + )
928 if isinstance(schema, list):
929 for item in schema:
898 - yield from _iter_reachable_schemas(item, root_schema, visited)
930 + yield from _iter_reachable_schemas(
931 + item, root_schema, visited, validator_class
932 + )
933 return
900 - if not isinstance(schema, dict) or id(schema) in visited:
934 + if not isinstance(schema, dict):
935 return
902 - visited.add(id(schema))
936 + if "$schema" in schema:
937 + validator_class = jsonschema_validators.validator_for(
938 + schema, default=validator_class
939 + )
940 + visit_key = (id(schema), validator_class)
941 + if visit_key in visited:
942 + return
943 + visited.add(visit_key)
944 yield schema
945
946 reference = schema.get("$ref")
@@ -912,33 +953,52 @@ def _iter_reachable_schemas(
953 # validated.
954 resolved = None
955 if isinstance(resolved, (dict, list)):
915 - yield from _iter_reachable_schemas(resolved, root_schema, visited)
956 + yield from _iter_reachable_schemas(
957 + resolved, root_schema, visited, validator_class
958 + )
959
917 - for keyword in ("properties", "patternProperties"):
960 + for keyword in ("properties", "patternProperties", "dependentSchemas"):
961 + if keyword == "dependentSchemas" and keyword not in validator_class.VALIDATORS:
962 + continue
963 values = schema.get(keyword)
964 if isinstance(values, dict):
965 for child in values.values():
921 - yield from _iter_reachable_schemas(child, root_schema, visited)
966 + yield from _iter_reachable_schemas(
967 + child, root_schema, visited, validator_class
968 + )
969
923 - for keyword in _SCHEMA_VALUED_KEYWORDS:
970 + value_keywords = _SCHEMA_VALUED_KEYWORDS | (
971 + {"unevaluatedItems", "unevaluatedProperties"}
972 + & validator_class.VALIDATORS.keys()
973 + )
974 + for keyword in value_keywords:
975 child = schema.get(keyword)
976 if isinstance(child, (dict, list)):
926 - yield from _iter_reachable_schemas(child, root_schema, visited)
977 + yield from _iter_reachable_schemas(
978 + child, root_schema, visited, validator_class
979 + )
980
928 - for keyword in _SCHEMA_LIST_KEYWORDS:
981 + list_keywords = _SCHEMA_LIST_KEYWORDS | (
982 + {"prefixItems"} & validator_class.VALIDATORS.keys()
983 + )
984 + for keyword in list_keywords:
985 children = schema.get(keyword)
986 if isinstance(children, list):
931 - yield from _iter_reachable_schemas(children, root_schema, visited)
987 + yield from _iter_reachable_schemas(
988 + children, root_schema, visited, validator_class
989 + )
990
991 items = schema.get("items")
992 if isinstance(items, (dict, list)):
935 - yield from _iter_reachable_schemas(items, root_schema, visited)
993 + yield from _iter_reachable_schemas(items, root_schema, visited, validator_class)
994
995 dependencies = schema.get("dependencies")
996 if isinstance(dependencies, dict):
997 for child in dependencies.values():
998 if isinstance(child, (dict, bool)):
941 - yield from _iter_reachable_schemas(child, root_schema, visited)
999 + yield from _iter_reachable_schemas(
1000 + child, root_schema, visited, validator_class
1001 + )
1002
1003
1004 def _has_dynamic_object_policy(schema: t.Dict[str, t.Any]) -> bool:
@@ -1040,8 +1100,8 @@ class _DynamicObjectModel(BaseModel):
1100 unrecognized = []
1101 for key, item in value.items():
1102 matched = False
1043 - for regex, validator in policy.patterns:
1044 - if regex.search(key):
1103 + for matches_key, validator in policy.patterns:
1104 + if matches_key(key):
1105 matched = True
1106 validator.validate(item)
1107
@@ -1074,8 +1134,8 @@ class _DynamicObjectModel(BaseModel):
1134
1135 for key, value in list(extra.items()):
1136 matched = False
1077 - for regex, validator in policy.patterns:
1078 - if regex.search(key):
1137 + for matches_key, validator in policy.patterns:
1138 + if matches_key(key):
1139 matched = True
1140 extra[key] = validator.materialize(extra[key])
1141
@@ -1347,17 +1407,6 @@ def _multiple_of_validator(multiple: t.Any) -> t.Callable[[t.Any], t.Any]:
1407 return validate
1408
1409
1350 -def _compile_pattern(pattern: str) -> t.Optional[t.Pattern[str]]:
1351 - try:
1352 - return re.compile(pattern)
1353 - except re.error:
1354 - logger.warning(
1355 - "Ignoring JSON Schema pattern %r: not a valid Python regular expression",
1356 - pattern,
1357 - )
1358 - return None
1359 -
1360 -
1410 def _with_string_constraints(annotation: t.Any, schema: t.Dict[str, t.Any]) -> t.Any:
1411 field_constraints: t.Dict[str, t.Any] = {}
1412 if "minLength" in schema:
@@ -1372,9 +1421,10 @@ def _with_string_constraints(annotation: t.Any, schema: t.Dict[str, t.Any]) -> t
1421
1422
1423 def _with_pattern_constraint(annotation: t.Any, pattern: str) -> t.Any:
1375 - """Prefer pydantic's native pattern; fall back to Python `re` when the Rust
1376 - regex crate rejects an ECMA construct such as look-around (legal in
1377 - Draft 7, whose regex dialect is ECMA-262)."""
1424 + """Prefer pydantic's native pattern; otherwise use the guarded matcher,
1425 + which covers ECMA constructs such as look-around (legal in Draft 7, whose
1426 + regex dialect is ECMA-262) only when they cannot backtrack catastrophically.
1427 + """
1428 candidate = t.Annotated[annotation, Field(pattern=pattern)]
1429 try:
1430 TypeAdapter(candidate)
@@ -1382,12 +1432,12 @@ def _with_pattern_constraint(annotation: t.Any, pattern: str) -> t.Any:
1432 except Exception: # noqa: BLE001 - pydantic raises SchemaError subclasses
1433 pass
1434
1385 - compiled = _compile_pattern(pattern)
1386 - if compiled is None:
1435 + matcher = _string_pattern_matcher(pattern)
1436 + if matcher is None:
1437 return annotation
1438
1439 def validate(value: t.Any) -> t.Any:
1390 - if isinstance(value, str) and compiled.search(value) is None:
1440 + if isinstance(value, str) and not matcher(value):
1441 raise ValueError(f"string does not match pattern {pattern!r}")
1442 return value
1443
@@ -1453,8 +1503,8 @@ def _typeless_constraint_validator(
1503 if not _SCALAR_CONSTRAINT_KEYWORDS.intersection(schema):
1504 return None
1505
1456 - compiled_pattern = (
1457 - _compile_pattern(schema["pattern"]) if "pattern" in schema else None
1506 + pattern_matcher = (
1507 + _string_pattern_matcher(schema["pattern"]) if "pattern" in schema else None
1508 )
1509
1510 def validate(value: t.Any) -> t.Any:
@@ -1467,7 +1517,7 @@ def _typeless_constraint_validator(
1517 raise ValueError(
1518 f"string must contain at most {schema['maxLength']} characters"
1519 )
1470 - if compiled_pattern is not None and compiled_pattern.search(value) is None:
1520 + if pattern_matcher is not None and not pattern_matcher(value):
1521 raise ValueError(f"string does not match pattern {schema['pattern']!r}")
1522 if _is_numeric(value):
1523 _check_numeric_keywords(schema, value)
@@ -1805,36 +1855,247 @@ def _exact_validation_multiple_of(
1855 )
1856
1857
1808 -_PATTERN_MATCHER_CACHE: t.Dict[str, t.Optional[t.Callable[[str], bool]]] = {}
1858 +# Schema-supplied regular expressions.
1859 +#
1860 +# Tool schemas are third-party input, and so are the strings matched against
1861 +# their patterns: LLM tool arguments, including object keys. Python's `re`
1862 +# backtracks, so a schema pattern such as `^(a+)+$` can freeze a worker on a
1863 +# short argument. Every schema regex is evaluated through one of two helpers:
1864 +#
1865 +# - `_linear_pattern_matcher`: pydantic-core's Rust regex engine, which runs in
1866 +# time linear in the input. `patternProperties` keys use only this, because a
1867 +# key pattern decides routing and cannot be dropped.
1868 +# - `_string_pattern_matcher`: the same engine, plus a guarded Python `re`
1869 +# fallback for the ECMA constructs the Rust engine rejects (look-around,
1870 +# backreferences). The fallback runs only when `_fallback_pattern_is_safe`
1871 +# proves the pattern cannot backtrack catastrophically, and only on input of
1872 +# at most `_BACKTRACKING_FALLBACK_MAX_INPUT` characters. Anything else is not
1873 +# enforced. Widening is safe: a hostile schema could omit the pattern.
1874 +#
1875 +# The static check matches the TypeScript SDK's fallback check.
1876 +
1877 +_PATTERN_CACHE_SIZE = 1024
1878 +
1879 +# Longer input is treated as matching on the `re` fallback. This bounds the
1880 +# quadratic scan over start positions that unanchored search performs.
1881 +_BACKTRACKING_FALLBACK_MAX_INPUT = 1000
1882 +
1883 +# A quantifier is "large" when it is unbounded or spans more than this many
1884 +# repetitions (`m - n` in `{n,m}`).
1885 +_LARGE_QUANTIFIER_SPAN = 16
1886 +
1887 +# Polynomial degree budget: one implicit factor for unanchored search, plus
1888 +# the large quantifiers outside look-around, plus the most in one look-around.
1889 +_MAX_FALLBACK_DEGREE = 2
1890 +
1891 +# Budget for the product of small quantifier spans and alternation widths.
1892 +_MAX_FALLBACK_CHOICES = 64
1893 +
1894 +# Absolute explicit repeat bounds also limit zero-width assertion work.
1895 +_MAX_FALLBACK_REPEAT_COUNT = 64
1896 +
1897 +
1898 +@functools.lru_cache(maxsize=_PATTERN_CACHE_SIZE)
1899 +def _linear_pattern_matcher(pattern: str) -> t.Optional[t.Callable[[str], bool]]:
1900 + """Unanchored (search) matcher on pydantic's linear-time Rust regex, or
1901 + None when that engine cannot compile `pattern`."""
1902 + try:
1903 + adapter = TypeAdapter(t.Annotated[str, Field(pattern=pattern)])
1904 + except Exception: # noqa: BLE001 - pydantic raises SchemaError subclasses
1905 + return None
1906
1907 + def match_with_pydantic(value: str) -> bool:
1908 + try:
1909 + adapter.validate_python(value, strict=True)
1910 + except ValidationError:
1911 + return False
1912 + return True
1913
1811 -def _pattern_matcher(pattern: str) -> t.Optional[t.Callable[[str], bool]]:
1812 - """Match `pattern` with pydantic's linear-time Rust regex when it can
1813 - compile it, falling back to Python `re` only for ECMA-only constructs."""
1814 - if pattern in _PATTERN_MATCHER_CACHE:
1815 - return _PATTERN_MATCHER_CACHE[pattern]
1816 - matcher: t.Optional[t.Callable[[str], bool]] = None
1817 - if _pattern_supported_by_pydantic(pattern):
1818 - adapter = TypeAdapter(t.Annotated[str, Field(pattern=pattern)])
1914 + return match_with_pydantic
1915
1820 - def match_with_pydantic(value: str) -> bool:
1821 - try:
1822 - adapter.validate_python(value, strict=True)
1823 - except ValidationError:
1824 - return False
1825 - return True
1916
1827 - matcher = match_with_pydantic
1828 - else:
1829 - compiled = _compile_pattern(pattern)
1830 - if compiled is not None:
1917 +def _pattern_supported_by_pydantic(pattern: str) -> bool:
1918 + return _linear_pattern_matcher(pattern) is not None
1919 +
1920 +
1921 +def _load_regex_parser() -> t.Tuple[t.Any, t.Any]:
1922 + try: # Python 3.11+
1923 + return (
1924 + importlib.import_module("re._parser"),
1925 + importlib.import_module("re._constants"),
1926 + )
1927 + except ImportError: # Python 3.10; deprecated aliases from 3.11 on
1928 + return (
1929 + importlib.import_module("sre_parse"),
1930 + importlib.import_module("sre_constants"),
1931 + )
1932 +
1933 +
1934 +_REGEX_PARSER, _REGEX_CONSTANTS = _load_regex_parser()
1935 +_REGEX_REPEATS = frozenset(
1936 + op
1937 + for op in (
1938 + _REGEX_CONSTANTS.MAX_REPEAT,
1939 + _REGEX_CONSTANTS.MIN_REPEAT,
1940 + getattr(_REGEX_CONSTANTS, "POSSESSIVE_REPEAT", None),
1941 + )
1942 + if op is not None
1943 +)
1944 +_REGEX_LOOKAROUNDS = frozenset({_REGEX_CONSTANTS.ASSERT, _REGEX_CONSTANTS.ASSERT_NOT})
1945 +_REGEX_ATOMS = frozenset(
1946 + {
1947 + _REGEX_CONSTANTS.ANY,
1948 + _REGEX_CONSTANTS.AT,
1949 + _REGEX_CONSTANTS.CATEGORY,
1950 + _REGEX_CONSTANTS.IN,
1951 + _REGEX_CONSTANTS.LITERAL,
1952 + _REGEX_CONSTANTS.NOT_LITERAL,
1953 + }
1954 +)
1955 +_REGEX_ATOMIC_GROUP = getattr(_REGEX_CONSTANTS, "ATOMIC_GROUP", None)
1956 +
1957 +
1958 +class _BacktrackingRisk(Exception):
1959 + """The pattern may backtrack catastrophically under Python `re`."""
1960 +
1961 +
1962 +class _FallbackPatternCheck:
1963 + """Conservative static check over a `re` parse tree.
1964 +
1965 + False positives only leave a pattern unenforced, so every construct that
1966 + is not clearly bounded counts as a risk.
1967 + """
1968 +
1969 + def __init__(self) -> None:
1970 + self.max_large_in_lookaround = 0
1971 + self.choices = 1
1972 +
1973 + def multiply_choices(self, factor: int) -> None:
1974 + self.choices *= factor
1975 + if self.choices > _MAX_FALLBACK_CHOICES:
1976 + raise _BacktrackingRisk("too many backtracking choices")
1977 +
1978 + def visit(
1979 + self,
1980 + subpattern: t.Any,
1981 + *,
1982 + in_repeat: bool,
1983 + in_lookaround: bool,
1984 + ) -> int:
1985 + """Return the large quantifiers in `subpattern` outside look-around."""
1986 + large = 0
1987 + for op, av in subpattern.data:
1988 + if op in _REGEX_REPEATS:
1989 + if in_repeat:
1990 + raise _BacktrackingRisk("quantified group contains a quantifier")
1991 + low, high, body = av
1992 + if low > _MAX_FALLBACK_REPEAT_COUNT or (
1993 + high != _REGEX_CONSTANTS.MAXREPEAT
1994 + and high > _MAX_FALLBACK_REPEAT_COUNT
1995 + ):
1996 + raise _BacktrackingRisk("repeat count exceeds the work budget")
1997 + if high == _REGEX_CONSTANTS.MAXREPEAT or (
1998 + high - low > _LARGE_QUANTIFIER_SPAN
1999 + ):
2000 + large += 1
2001 + elif high > low:
2002 + self.multiply_choices(high - low + 1)
2003 + large += self.visit(body, in_repeat=True, in_lookaround=in_lookaround)
2004 + elif op == _REGEX_CONSTANTS.BRANCH:
2005 + if in_repeat:
2006 + raise _BacktrackingRisk("quantified group contains alternation")
2007 + _, alternatives = av
2008 + self.multiply_choices(len(alternatives))
2009 + for alternative in alternatives:
2010 + large += self.visit(
2011 + alternative,
2012 + in_repeat=in_repeat,
2013 + in_lookaround=in_lookaround,
2014 + )
2015 + elif op in _REGEX_LOOKAROUNDS:
2016 + if in_lookaround:
2017 + raise _BacktrackingRisk("nested look-around")
2018 + _, body = av
2019 + self.max_large_in_lookaround = max(
2020 + self.max_large_in_lookaround,
2021 + self.visit(body, in_repeat=in_repeat, in_lookaround=True),
2022 + )
2023 + elif op == _REGEX_CONSTANTS.SUBPATTERN:
2024 + large += self.visit(
2025 + av[-1], in_repeat=in_repeat, in_lookaround=in_lookaround
2026 + )
2027 + elif _REGEX_ATOMIC_GROUP is not None and op == _REGEX_ATOMIC_GROUP:
2028 + large += self.visit(
2029 + av, in_repeat=in_repeat, in_lookaround=in_lookaround
2030 + )
2031 + elif op not in _REGEX_ATOMS:
2032 + # Backreferences (GROUPREF, GROUPREF_EXISTS) and anything this
2033 + # check does not model.
2034 + raise _BacktrackingRisk(f"unsupported regex construct {op}")
2035 + return large
2036 +
2037
1832 - def match_with_re(value: str) -> bool:
1833 - return compiled.search(value) is not None
2038 +def _fallback_pattern_is_safe(pattern: str) -> bool:
2039 + """Whether Python `re` cannot backtrack catastrophically on `pattern`.
2040 +
2041 + Rejects backreferences, quantified groups (look-around included) that
2042 + contain a quantifier or alternation, excessive explicit repeat bounds,
2043 + nested look-around, more than
2044 + `_MAX_FALLBACK_DEGREE` nested polynomial factors, and more than
2045 + `_MAX_FALLBACK_CHOICES` combined small-quantifier and alternation choices.
2046 +
2047 + :raises re.error: `pattern` is not a valid Python regular expression.
2048 + """
2049 + parsed = _REGEX_PARSER.parse(pattern)
2050 + check = _FallbackPatternCheck()
2051 + try:
2052 + large_outside = check.visit(parsed, in_repeat=False, in_lookaround=False)
2053 + except _BacktrackingRisk:
2054 + return False
2055 + anchored = (
2056 + bool(parsed.data)
2057 + and parsed.data[0]
2058 + == (
2059 + _REGEX_CONSTANTS.AT,
2060 + _REGEX_CONSTANTS.AT_BEGINNING,
2061 + )
2062 + and not parsed.state.flags & re.MULTILINE
2063 + )
2064 + degree = (0 if anchored else 1) + large_outside + check.max_large_in_lookaround
2065 + return degree <= _MAX_FALLBACK_DEGREE
2066
1835 - matcher = match_with_re
1836 - _PATTERN_MATCHER_CACHE[pattern] = matcher
1837 - return matcher
2067 +
2068 +@functools.lru_cache(maxsize=_PATTERN_CACHE_SIZE)
2069 +def _string_pattern_matcher(pattern: str) -> t.Optional[t.Callable[[str], bool]]:
2070 + """Search-semantics matcher for a `pattern` string constraint, or None
2071 + (with a warning) when it cannot be enforced in bounded time."""
2072 + linear = _linear_pattern_matcher(pattern)
2073 + if linear is not None:
2074 + return linear
2075 +
2076 + try:
2077 + safe = _fallback_pattern_is_safe(pattern)
2078 + compiled = re.compile(pattern)
2079 + except (re.error, OverflowError, RecursionError):
2080 + logger.warning(
2081 + "Ignoring JSON Schema pattern %r: not a valid Python regular expression",
2082 + pattern,
2083 + )
2084 + return None
2085 + if not safe:
2086 + logger.warning(
2087 + "Ignoring JSON Schema pattern %r: the linear-time regex engine does "
2088 + "not support it and Python's backtracking engine could stall on it",
2089 + pattern,
2090 + )
2091 + return None
2092 +
2093 + def match_with_re(value: str) -> bool:
2094 + if len(value) > _BACKTRACKING_FALLBACK_MAX_INPUT:
2095 + return True
2096 + return compiled.search(value) is not None
2097 +
2098 + return match_with_re
2099
2100
2101 def _exact_validation_pattern(
@@ -1845,13 +2106,229 @@ def _exact_validation_pattern(
2106 ) -> t.Iterator[t.Any]:
2107 if not isinstance(instance, str) or not isinstance(pattern, str):
2108 return
1848 - matcher = _pattern_matcher(pattern)
2109 + matcher = _string_pattern_matcher(pattern)
2110 if matcher is not None and not matcher(instance):
2111 yield jsonschema_exceptions.ValidationError(
2112 f"{instance!r} does not match {pattern!r}"
2113 )
2114
2115
2116 +def _safe_pattern_properties(
2117 + validator: t.Any,
2118 + pattern_properties: t.Any,
2119 + instance: t.Any,
2120 + _schema: t.Any,
2121 +) -> t.Iterator[t.Any]:
2122 + """jsonschema's `patternProperties` with keys matched in linear time."""
2123 + if not validator.is_type(instance, "object"):
2124 + return
2125 +
2126 + for pattern, subschema in pattern_properties.items():
2127 + matches_key = _linear_pattern_matcher(pattern)
2128 + if matches_key is None:
2129 + # Conversion rejects these first; fail closed if one slips past.
2130 + yield jsonschema_exceptions.ValidationError(
2131 + _unsupported_pattern_property_message(pattern)
2132 + )
2133 + continue
2134 + for key, value in instance.items():
2135 + if matches_key(key):
2136 + yield from validator.descend(
2137 + value,
2138 + subschema,
2139 + path=key,
2140 + schema_path=pattern,
2141 + )
2142 +
2143 +
2144 +def _extras_msg(extras: t.Sequence[t.Any]) -> t.Tuple[str, str]:
2145 + verb = "was" if len(extras) == 1 else "were"
2146 + return ", ".join(repr(extra) for extra in extras), verb
2147 +
2148 +
2149 +def _safe_additional_properties(
2150 + validator: t.Any,
2151 + additional: t.Any,
2152 + instance: t.Any,
2153 + schema: t.Any,
2154 +) -> t.Iterator[t.Any]:
2155 + """jsonschema's `additionalProperties` with keys matched in linear time."""
2156 + if not validator.is_type(instance, "object"):
2157 + return
2158 +
2159 + properties = schema.get("properties", {})
2160 + patterns = list(schema.get("patternProperties", {}))
2161 + matchers = []
2162 + # jsonschema searches the `|`-joined patterns, so a lone "" matches nothing.
2163 + if "|".join(patterns):
2164 + for pattern in patterns:
2165 + matches_key = _linear_pattern_matcher(pattern)
2166 + if matches_key is None:
2167 + yield jsonschema_exceptions.ValidationError(
2168 + _unsupported_pattern_property_message(pattern)
2169 + )
2170 + return
2171 + matchers.append(matches_key)
2172 + extras = {
2173 + key
2174 + for key in instance
2175 + if key not in properties
2176 + and not any(matches_key(key) for matches_key in matchers)
2177 + }
2178 +
2179 + if validator.is_type(additional, "object"):
2180 + for extra in extras:
2181 + yield from validator.descend(instance[extra], additional, path=extra)
2182 + elif not additional and extras:
2183 + if "patternProperties" in schema:
2184 + verb = "does" if len(extras) == 1 else "do"
2185 + joined = ", ".join(repr(each) for each in sorted(extras))
2186 + joined_patterns = ", ".join(
2187 + repr(each) for each in sorted(schema["patternProperties"])
2188 + )
2189 + yield jsonschema_exceptions.ValidationError(
2190 + f"{joined} {verb} not match any of the regexes: {joined_patterns}"
2191 + )
2192 + else:
2193 + error = "Additional properties are not allowed (%s %s unexpected)"
2194 + yield jsonschema_exceptions.ValidationError(
2195 + error % _extras_msg(sorted(extras, key=str))
2196 + )
2197 +
2198 +
2199 +class _LinearRegexModule:
2200 + """Stands in for `re` inside jsonschema's evaluated-property helpers, whose
2201 + only regex use is `re.search(pattern, key)` over `patternProperties`."""
2202 +
2203 + @staticmethod
2204 + def search(pattern: t.Any, string: t.Any) -> t.Optional[bool]:
2205 + matches_key = _linear_pattern_matcher(pattern)
2206 + if matches_key is None:
2207 + raise ValueError(_unsupported_pattern_property_message(pattern))
2208 + return True if matches_key(string) else None
2209 +
2210 +
2211 +def _unsupported_unevaluated_properties(
2212 + validator: t.Any,
2213 + _unevaluated: t.Any,
2214 + instance: t.Any,
2215 + _schema: t.Any,
2216 +) -> t.Iterator[t.Any]:
2217 + if validator.is_type(instance, "object"):
2218 + yield jsonschema_exceptions.ValidationError(
2219 + "unevaluatedProperties is not supported with the installed jsonschema"
2220 + )
2221 +
2222 +
2223 +def _linear_unevaluated_properties(keyword: t.Any) -> t.Any:
2224 + """Rebind jsonschema's `unevaluatedProperties` (2019-09 and 2020-12) so its
2225 + `patternProperties` key matching uses `_LinearRegexModule`.
2226 +
2227 + The keyword's semantics are long and version-specific, so the original
2228 + code objects run unchanged against copied module globals; nothing global
2229 + is patched. If the helper no longer reaches regexes through a module-level
2230 + `re`, the keyword fails closed instead of silently using `re`.
2231 + """
2232 + module_globals = getattr(keyword, "__globals__", None)
2233 + finder = (module_globals or {}).get("find_evaluated_property_keys_by_schema")
2234 + finder_globals = getattr(finder, "__globals__", None)
2235 + if (
2236 + not isinstance(keyword, types.FunctionType)
2237 + or not isinstance(finder, types.FunctionType)
2238 + or keyword.__closure__ is not None
2239 + or finder.__closure__ is not None
2240 + or finder_globals is None
2241 + or finder_globals.get("re") is not re
2242 + or "re" not in finder.__code__.co_names
2243 + or "re" in keyword.__code__.co_names
2244 + ):
2245 + return _unsupported_unevaluated_properties
2246 +
2247 + linear_finder_globals = dict(finder_globals)
2248 + linear_finder = types.FunctionType(
2249 + finder.__code__,
2250 + linear_finder_globals,
2251 + finder.__name__,
2252 + finder.__defaults__,
2253 + )
2254 + linear_finder_globals["re"] = _LinearRegexModule
2255 + linear_finder_globals["find_evaluated_property_keys_by_schema"] = linear_finder
2256 +
2257 + linear_keyword_globals = dict(keyword.__globals__)
2258 + linear_keyword_globals["re"] = _LinearRegexModule
2259 + linear_keyword_globals["find_evaluated_property_keys_by_schema"] = linear_finder
2260 + return types.FunctionType(
2261 + keyword.__code__,
2262 + linear_keyword_globals,
2263 + keyword.__name__,
2264 + keyword.__defaults__,
2265 + )
2266 +
2267 +
2268 +_SAFE_REGEX_VALIDATOR_CLASSES: t.Set[t.Any] = set()
2269 +
2270 +
2271 +@functools.lru_cache(maxsize=None)
2272 +def _with_safe_regex_keywords(validator_class: t.Any) -> t.Any:
2273 + """Extend a jsonschema validator class so no keyword evaluates a schema
2274 + regex with Python `re` on unbounded input.
2275 +
2276 + jsonschema's `evolve` (used by `descend` for every subschema) switches to
2277 + the stock class of any subschema that declares `$schema`, which would drop
2278 + these overrides. The returned class keeps the dialect switch but re-applies
2279 + the regex overrides to the class it switches to.
2280 + """
2281 + if validator_class in _SAFE_REGEX_VALIDATOR_CLASSES:
2282 + return validator_class
2283 + overrides = {
2284 + "additionalProperties": _safe_additional_properties,
2285 + "pattern": _exact_validation_pattern,
2286 + "patternProperties": _safe_pattern_properties,
2287 + }
2288 + unevaluated = validator_class.VALIDATORS.get("unevaluatedProperties")
2289 + if unevaluated is not None:
2290 + overrides["unevaluatedProperties"] = _linear_unevaluated_properties(unevaluated)
2291 + return _install_safe_regex_evolve(
2292 + jsonschema_validators.extend(validator_class, overrides)
2293 + )
2294 +
2295 +
2296 +@functools.lru_cache(maxsize=None)
2297 +def _exact_validator_class() -> t.Any:
2298 + # Align multipleOf with the decimal-scaled checks in the Zod and Effect
2299 + # converters instead of raw float modulo.
2300 + return _install_safe_regex_evolve(
2301 + jsonschema_validators.extend(
2302 + _with_safe_regex_keywords(jsonschema_validators.Draft7Validator),
2303 + {"multipleOf": _exact_validation_multiple_of},
2304 + )
2305 + )
2306 +
2307 +
2308 +def _install_safe_regex_evolve(validator_class: t.Any) -> t.Any:
2309 + stock_evolve = validator_class.evolve
2310 +
2311 + def evolve(self: t.Any, **changes: t.Any) -> t.Any:
2312 + evolved = stock_evolve(self, **changes)
2313 + evolved_class: t.Any = type(evolved)
2314 + if evolved_class in _SAFE_REGEX_VALIDATOR_CLASSES:
2315 + return evolved
2316 + # Rebuild on the safe variant, copying init fields exactly as
2317 + # jsonschema's own `evolve` does.
2318 + safe_class = _with_safe_regex_keywords(evolved_class)
2319 + return safe_class(
2320 + **{
2321 + field.alias: getattr(evolved, field.name)
2322 + for field in evolved_class.__attrs_attrs__
2323 + if field.init
2324 + }
2325 + )
2326 +
2327 + validator_class.evolve = evolve
2328 + _SAFE_REGEX_VALIDATOR_CLASSES.add(validator_class)
2329 + return validator_class
2330 +
2331 +
2332 # A bare registry resolves only references inside the schema document and
2333 # raises on anything else; jsonschema's default registry falls back to
2334 # fetching unknown URIs over the network.
@@ -1942,6 +2419,19 @@ def _close_all_of_branches(schema: t.Any) -> t.Any:
2419 return result
2420
2421
2422 +def _reject_unsupported_pattern_properties(
2423 + schema: t.Any,
2424 + root_schema: t.Dict[str, t.Any],
2425 +) -> None:
2426 + """Fail at conversion time on any reachable `patternProperties` key the
2427 + linear-time engine cannot match."""
2428 + for candidate in _iter_reachable_schemas(schema, root_schema):
2429 + pattern_properties = candidate.get("patternProperties")
2430 + if isinstance(pattern_properties, dict):
2431 + for pattern in pattern_properties:
2432 + _compile_pattern_property(pattern)
2433 +
2434 +
2435 def _validate_json_schema(
2436 schema: t.Dict[str, t.Any],
2437 root_schema: t.Dict[str, t.Any],
@@ -1958,19 +2448,11 @@ def _validate_json_schema(
2448 # The SDK contract is Draft 7 plus the OpenAPI 3.0 boolean spelling for
2449 # exclusive bounds. Falling the entire document back to Draft 4 would make
2450 # `contains`, `const`, `if`/`then`/`else`, and `propertyNames` disappear.
1961 - validator_class = jsonschema_validators.Draft7Validator
1962 - # Align multipleOf with the decimal-scaled checks in the Zod and Effect
1963 - # converters instead of raw float modulo.
1964 - # `pattern` goes through pydantic's Rust regex where possible so a
1965 - # backtracking pattern cannot stall validation (Python `re` is exponential
1966 - # on inputs such as `(a+)+$`).
1967 - validator_class = jsonschema_validators.extend(
1968 - validator_class,
1969 - {
1970 - "multipleOf": _exact_validation_multiple_of,
1971 - "pattern": _exact_validation_pattern,
1972 - },
1973 - )
2451 + # Schema regexes (`pattern`, `patternProperties`, `additionalProperties`
2452 + # key matching) never run on unbounded Python `re`, which is exponential
2453 + # on inputs such as `(a+)+$`.
2454 + _reject_unsupported_pattern_properties(validation_schema, validation_root)
2455 + validator_class = _exact_validator_class()
2456 validator = validator_class(
2457 validation_root,
2458 registry=_LOCAL_ONLY_REGISTRY,
@@ -2114,22 +2596,6 @@ _TYPE_ARRAY_CHILD_KEYWORDS_SCHEMAS = (
2596 )
2597
2598
2117 -_PYDANTIC_PATTERN_SUPPORT_CACHE: t.Dict[str, bool] = {}
2118 -
2119 -
2120 -def _pattern_supported_by_pydantic(pattern: str) -> bool:
2121 - cached = _PYDANTIC_PATTERN_SUPPORT_CACHE.get(pattern)
2122 - if cached is not None:
2123 - return cached
2124 - try:
2125 - TypeAdapter(t.Annotated[str, Field(pattern=pattern)])
2126 - supported = True
2127 - except Exception: # noqa: BLE001 - pydantic raises SchemaError subclasses
2128 - supported = False
2129 - _PYDANTIC_PATTERN_SUPPORT_CACHE[pattern] = supported
2130 - return supported
2131 -
2132 -
2599 def _normalize_schema_for_library(schema: t.Any) -> t.Any:
2600 """Rewrite constructs the json-schema-to-pydantic library mishandles.
2601
python/tests/test_schema_pattern_redos.py new
+462
@@ -0,0 +1,462 @@
1 +"""Schema-supplied regexes must not stall validation (SEC-1178).
2 +
3 +Tool schemas are third-party input, and the strings matched against their
4 +patterns (tool arguments, object keys) are attacker-influenced. Every pattern
5 +runs on pydantic's linear-time Rust regex, or on Python `re` only when a static
6 +check proves it cannot backtrack catastrophically.
7 +
8 +None of these tests may compare a catastrophic input against the stock
9 +`jsonschema` oracle: that oracle runs Python `re` and would hang.
10 +"""
11 +
12 +import time
13 +import typing as t
14 +
15 +import pytest
16 +from jsonschema import Draft201909Validator, Draft202012Validator
17 +from pydantic import TypeAdapter, ValidationError
18 +
19 +from composio.utils import schema_converter
20 +from composio.utils.schema_converter import json_schema_to_pydantic_type
21 +from composio.utils.shared import json_schema_to_model
22 +
23 +# Look-around forces the `re` fallback; the nested quantifier is exponential.
24 +LOOKAROUND_CATASTROPHIC = "^(?=a)(a+)+$"
25 +# The Rust engine compiles this, so it must be enforced in linear time.
26 +RUST_CATASTROPHIC = "^(a+)+$"
27 +EVIL = "a" * 40 + "!"
28 +TIME_LIMIT = 1.0
29 +
30 +
31 +class _Timer:
32 + def __enter__(self) -> "_Timer":
33 + self.started = time.perf_counter()
34 + return self
35 +
36 + def __exit__(self, *_: t.Any) -> None:
37 + self.elapsed = time.perf_counter() - self.started
38 +
39 +
40 +def _adapter(schema: t.Dict[str, t.Any]) -> TypeAdapter:
41 + return TypeAdapter(json_schema_to_pydantic_type(schema))
42 +
43 +
44 +@pytest.fixture
45 +def warnings(monkeypatch: pytest.MonkeyPatch) -> t.List[str]:
46 + """Capture converter warnings; clear the warn-once matcher cache."""
47 + schema_converter._string_pattern_matcher.cache_clear()
48 + messages: t.List[str] = []
49 +
50 + def capture(message: str, *args: t.Any, **_: t.Any) -> None:
51 + messages.append(message % args)
52 +
53 + monkeypatch.setattr(schema_converter.logger, "warning", capture)
54 + return messages
55 +
56 +
57 +@pytest.mark.unit
58 +@pytest.mark.schema
59 +@pytest.mark.parametrize("entry_point", ["model", "type"])
60 +def test_rust_supported_pattern_property_routes_keys_in_linear_time(
61 + entry_point: str,
62 +) -> None:
63 + schema = {
64 + "type": "object",
65 + "title": "Routed",
66 + "patternProperties": {RUST_CATASTROPHIC: {"type": "integer"}},
67 + "additionalProperties": {"type": "string"},
68 + }
69 + validate: t.Callable[[t.Any], t.Any]
70 + if entry_point == "model":
71 + model = json_schema_to_model(schema)
72 + validate = model.model_validate
73 + else:
74 + validate = _adapter(schema).validate_python
75 +
76 + with _Timer() as timer:
77 + validate({"aaa": 1, EVIL: "routed to additionalProperties"})
78 + with pytest.raises(ValidationError):
79 + validate({"aaa": "matching key must be an integer"})
80 + with pytest.raises(ValidationError):
81 + validate({EVIL: 1})
82 + assert timer.elapsed < TIME_LIMIT
83 +
84 +
85 +@pytest.mark.unit
86 +@pytest.mark.schema
87 +def test_lookaround_catastrophic_pattern_is_not_enforced(
88 + warnings: t.List[str],
89 +) -> None:
90 + schema = {
91 + "type": "object",
92 + "properties": {"value": {"type": "string", "pattern": LOOKAROUND_CATASTROPHIC}},
93 + }
94 + model = json_schema_to_model(schema)
95 + assert any(LOOKAROUND_CATASTROPHIC in message for message in warnings)
96 +
97 + with _Timer() as timer:
98 + model.model_validate({"value": EVIL})
99 + assert timer.elapsed < TIME_LIMIT
100 +
101 +
102 +@pytest.mark.unit
103 +@pytest.mark.schema
104 +def test_forced_fallback_polynomial_pattern_is_not_enforced(
105 + warnings: t.List[str],
106 +) -> None:
107 + pattern = "(?=a).*.*.*.*.*!"
108 + assert not schema_converter._fallback_pattern_is_safe(pattern)
109 + adapter = _adapter({"type": "string", "pattern": pattern})
110 + assert any(pattern in message for message in warnings)
111 +
112 + with _Timer() as timer:
113 + assert adapter.validate_python("a" * 900) == "a" * 900
114 + assert timer.elapsed < TIME_LIMIT
115 +
116 +
117 +@pytest.mark.unit
118 +@pytest.mark.schema
119 +@pytest.mark.parametrize(
120 + "pattern",
121 + [
122 + "^(?=a){1000000000}a$",
123 + "^z|(?=a)" + "(?:a|aa)" * 5 + "a*a*b$",
124 + ],
125 +)
126 +def test_fixed_repeats_and_unanchored_alternatives_are_not_enforced(
127 + pattern: str,
128 + warnings: t.List[str],
129 +) -> None:
130 + # Assert rejection before matching, so a regression cannot execute the attack.
131 + assert not schema_converter._fallback_pattern_is_safe(pattern)
132 + adapter = _adapter({"type": "string", "pattern": pattern})
133 + assert any(pattern in message for message in warnings)
134 + value = "a" * schema_converter._BACKTRACKING_FALLBACK_MAX_INPUT
135 + assert adapter.validate_python(value) == value
136 +
137 +
138 +@pytest.mark.unit
139 +@pytest.mark.schema
140 +@pytest.mark.parametrize("entry_point", ["model", "type"])
141 +@pytest.mark.parametrize("pattern", [LOOKAROUND_CATASTROPHIC, "^(?=a)"])
142 +def test_lookaround_pattern_property_is_rejected_at_conversion(
143 + entry_point: str,
144 + pattern: str,
145 +) -> None:
146 + # Key patterns decide routing, so they are never dropped and never run on
147 + # Python `re`, even when the static check would accept them.
148 + schema = {
149 + "type": "object",
150 + "title": "Keys",
151 + "patternProperties": {pattern: {"type": "integer"}},
152 + }
153 + convert = json_schema_to_model if entry_point == "model" else _adapter
154 + with pytest.raises(ValueError, match="Unsupported patternProperties"):
155 + convert(schema)
156 +
157 +
158 +@pytest.mark.unit
159 +@pytest.mark.schema
160 +def test_nested_lookaround_pattern_property_is_rejected_at_conversion() -> None:
161 + schema = {
162 + "type": "object",
163 + "title": "Nested",
164 + "anyOf": [
165 + {"patternProperties": {LOOKAROUND_CATASTROPHIC: {"type": "integer"}}},
166 + {"required": ["x"]},
167 + ],
168 + }
169 + with pytest.raises(ValueError, match="Unsupported patternProperties"):
170 + json_schema_to_model(schema)
171 +
172 +
173 +@pytest.mark.unit
174 +@pytest.mark.schema
175 +def test_invalid_pattern_property_keeps_invalid_error() -> None:
176 + schema = {"type": "object", "patternProperties": {"(": {"type": "integer"}}}
177 + with pytest.raises(ValueError, match="Invalid patternProperties"):
178 + _adapter(schema)
179 +
180 +
181 +@pytest.mark.unit
182 +@pytest.mark.schema
183 +def test_safe_lookaround_pattern_is_enforced_on_bounded_input() -> None:
184 + adapter = _adapter({"type": "string", "pattern": "^(?=.*[a-y])a"})
185 + assert adapter.validate_python("ab") == "ab"
186 + with pytest.raises(ValidationError):
187 + adapter.validate_python("ba")
188 + with pytest.raises(ValidationError):
189 + adapter.validate_python("z" * 1000)
190 +
191 + # Past the fallback input cap the constraint widens to "satisfied".
192 + long_value = "z" * (schema_converter._BACKTRACKING_FALLBACK_MAX_INPUT + 1)
193 + assert adapter.validate_python(long_value) == long_value
194 +
195 +
196 +@pytest.mark.unit
197 +@pytest.mark.schema
198 +def test_typeless_catastrophic_patterns_complete_in_linear_time() -> None:
199 + enforced = _adapter({"pattern": RUST_CATASTROPHIC, "minLength": 1})
200 + ignored = _adapter({"pattern": LOOKAROUND_CATASTROPHIC, "minLength": 1})
201 +
202 + with _Timer() as timer:
203 + with pytest.raises(ValidationError):
204 + enforced.validate_python(EVIL)
205 + assert enforced.validate_python("aaa") == "aaa"
206 + assert enforced.validate_python(1) == 1
207 + assert ignored.validate_python(EVIL) == EVIL
208 + assert timer.elapsed < TIME_LIMIT
209 +
210 +
211 +@pytest.mark.unit
212 +@pytest.mark.schema
213 +@pytest.mark.parametrize(
214 + "dynamic_keyword", ["patternProperties", "additionalProperties"]
215 +)
216 +def test_catastrophic_pattern_inside_dynamic_key_schema(dynamic_keyword: str) -> None:
217 + def schema_for(pattern: str) -> t.Dict[str, t.Any]:
218 + value_schema = {"type": "string", "pattern": pattern}
219 + dynamic: t.Any = (
220 + {"^k": value_schema}
221 + if dynamic_keyword == "patternProperties"
222 + else value_schema
223 + )
224 + return {"type": "object", "title": "Dynamic", dynamic_keyword: dynamic}
225 +
226 + enforced = json_schema_to_model(schema_for(RUST_CATASTROPHIC))
227 + ignored = json_schema_to_model(schema_for(LOOKAROUND_CATASTROPHIC))
228 +
229 + with _Timer() as timer:
230 + with pytest.raises(ValidationError):
231 + enforced.model_validate({"k1": EVIL})
232 + enforced.model_validate({"k1": "aaa"})
233 + ignored.model_validate({"k1": EVIL})
234 + assert timer.elapsed < TIME_LIMIT
235 +
236 +
237 +@pytest.mark.unit
238 +@pytest.mark.schema
239 +def test_catastrophic_pattern_inside_any_of_branch() -> None:
240 + def schema_for(pattern: str) -> t.Dict[str, t.Any]:
241 + return {
242 + "type": "object",
243 + "properties": {
244 + "value": {
245 + "anyOf": [
246 + {"type": "string", "pattern": pattern},
247 + {"type": "integer"},
248 + ]
249 + }
250 + },
251 + }
252 +
253 + enforced = json_schema_to_model(schema_for(RUST_CATASTROPHIC))
254 + ignored = json_schema_to_model(schema_for(LOOKAROUND_CATASTROPHIC))
255 +
256 + with _Timer() as timer:
257 + with pytest.raises(ValidationError):
258 + enforced.model_validate({"value": EVIL})
259 + enforced.model_validate({"value": "aaa"})
260 + enforced.model_validate({"value": 1})
261 + ignored.model_validate({"value": EVIL})
262 + assert timer.elapsed < TIME_LIMIT
263 +
264 +
265 +@pytest.mark.unit
266 +@pytest.mark.schema
267 +@pytest.mark.parametrize(
268 + ("validator_class", "dialect"),
269 + [
270 + (Draft201909Validator, "https://json-schema.org/draft/2019-09/schema"),
271 + (Draft202012Validator, "https://json-schema.org/draft/2020-12/schema"),
272 + ],
273 +)
274 +def test_unevaluated_properties_match_keys_in_linear_time(
275 + validator_class: t.Any,
276 + dialect: str,
277 +) -> None:
278 + keyword = schema_converter._with_safe_regex_keywords(validator_class).VALIDATORS[
279 + "unevaluatedProperties"
280 + ]
281 + # Fails if a jsonschema upgrade stops the key matching from being rebound.
282 + assert keyword is not schema_converter._unsupported_unevaluated_properties
283 +
284 + schema = {
285 + "$schema": dialect,
286 + "type": "object",
287 + "additionalProperties": {
288 + "type": "object",
289 + "patternProperties": {RUST_CATASTROPHIC: {"type": "integer"}},
290 + "unevaluatedProperties": False,
291 + },
292 + }
293 + adapter = _adapter(schema)
294 + with _Timer() as timer:
295 + adapter.validate_python({"x": {"aaa": 1}})
296 + with pytest.raises(ValidationError):
297 + adapter.validate_python({"x": {EVIL: 1}})
298 + assert timer.elapsed < TIME_LIMIT
299 +
300 +
301 +@pytest.mark.unit
302 +@pytest.mark.schema
303 +def test_subschema_dialect_switch_keeps_linear_regex() -> None:
304 + # jsonschema switches to its stock validator class for any subschema that
305 + # declares `$schema`; the regex overrides must survive that switch.
306 + schema = {
307 + "type": "object",
308 + "title": "DialectSwitch",
309 + "properties": {
310 + "value": {
311 + "$schema": "http://json-schema.org/draft-07/schema#",
312 + "type": "object",
313 + "properties": {
314 + "text": {"type": "string", "pattern": RUST_CATASTROPHIC}
315 + },
316 + "patternProperties": {RUST_CATASTROPHIC: {"type": "integer"}},
317 + "additionalProperties": False,
318 + }
319 + },
320 + }
321 + model = json_schema_to_model(schema)
322 + with _Timer() as timer:
323 + model.model_validate({"value": {"aaa": 1, "text": "aaa"}})
324 + with pytest.raises(ValidationError):
325 + model.model_validate({"value": {EVIL: 1}})
326 + with pytest.raises(ValidationError):
327 + model.model_validate({"value": {"text": EVIL}})
328 + assert timer.elapsed < TIME_LIMIT
329 +
330 +
331 +@pytest.mark.unit
332 +@pytest.mark.schema
333 +@pytest.mark.parametrize("entry_point", ["model", "type"])
334 +@pytest.mark.parametrize(
335 + ("draft", "keyword"),
336 + [
337 + (draft, keyword)
338 + for draft in ["2019-09", "2020-12"]
339 + for keyword in [
340 + "dependentSchemas",
341 + "unevaluatedProperties",
342 + "unevaluatedItems",
343 + "prefixItems",
344 + ]
345 + if keyword != "prefixItems" or draft == "2020-12"
346 + ],
347 +)
348 +def test_switched_dialect_rejects_unsupported_key_patterns_at_conversion(
349 + entry_point: str,
350 + draft: str,
351 + keyword: str,
352 +) -> None:
353 + unsupported = {
354 + "type": "object",
355 + "patternProperties": {"^(?=a)": {"type": "integer"}},
356 + }
357 + value: t.Any = unsupported
358 + if keyword == "dependentSchemas":
359 + value = {"trigger": unsupported}
360 + elif keyword == "prefixItems":
361 + value = [unsupported]
362 + schema = {
363 + "type": "object",
364 + "properties": {
365 + "value": {
366 + "$schema": f"https://json-schema.org/draft/{draft}/schema",
367 + "type": "array"
368 + if keyword in {"prefixItems", "unevaluatedItems"}
369 + else "object",
370 + keyword: value,
371 + }
372 + },
373 + }
374 + convert = json_schema_to_model if entry_point == "model" else _adapter
375 + with pytest.raises(ValueError, match="Unsupported patternProperties"):
376 + convert(schema)
377 +
378 +
379 +@pytest.mark.unit
380 +@pytest.mark.schema
381 +def test_draft7_keeps_unsupported_dialect_keywords_as_annotations() -> None:
382 + schema = {
383 + "type": "object",
384 + "dependentSchemas": {
385 + "trigger": {"patternProperties": {"^(?=a)": {"type": "integer"}}}
386 + },
387 + }
388 + assert _adapter(schema).validate_python({"trigger": "annotation"}) == {
389 + "trigger": "annotation"
390 + }
391 +
392 +
393 +@pytest.mark.unit
394 +@pytest.mark.schema
395 +def test_shared_reference_is_checked_under_each_effective_dialect() -> None:
396 + schema = {
397 + "type": "object",
398 + "properties": {
399 + "old": {"$ref": "#/x"},
400 + "new": {
401 + "$schema": "https://json-schema.org/draft/2020-12/schema",
402 + "$ref": "#/x",
403 + },
404 + },
405 + "x": {
406 + "type": "object",
407 + "dependentSchemas": {
408 + "trigger": {"patternProperties": {"^(?=a)": {"type": "integer"}}}
409 + },
410 + },
411 + }
412 + with pytest.raises(ValueError, match="Unsupported patternProperties"):
413 + _adapter(schema)
414 +
415 +
416 +@pytest.mark.unit
417 +@pytest.mark.schema
418 +@pytest.mark.parametrize(
419 + "pattern",
420 + [
421 + "^(?=.*[a-y])a",
422 + "^(?=a){2}ab$",
423 + "(?<=a)b",
424 + r"^(?=.*[a-z])(?=.*\d)[A-Za-z\d]{8,}$",
425 + "^(?=x)(jpg|png|gif)$",
426 + ],
427 +)
428 +def test_static_check_accepts_bounded_patterns(pattern: str) -> None:
429 + assert schema_converter._fallback_pattern_is_safe(pattern)
430 +
431 +
432 +@pytest.mark.unit
433 +@pytest.mark.schema
434 +@pytest.mark.parametrize(
435 + "pattern",
436 + [
437 + "(a+)+",
438 + "(?=(a+)+)",
439 + # sre folds `(a|b)` into a character class, which is linear; a
440 + # multi-character alternation stays an alternation.
441 + "(a|bc)*",
442 + r"(a)\1",
443 + "(?P<x>a)(?P=x)",
444 + "(a)?(?(1)b|c)",
445 + "(?=(?=a))",
446 + ".*.*x",
447 + "(?=.*a.*b)x",
448 + "(?=a).*.*.*.*.*!",
449 + "(?=a)" + "a?" * 7,
450 + "(?<=)" + "(?:a|aa)" * 7 + "b",
451 + "^(?=a){1000000000}a$",
452 + "^(?=a)a{1000000000}$",
453 + "^(?=a)a{1,1000000000}$",
454 + "^(?=a)a{1000000000,}$",
455 + "^((?=a){64}){64}a$",
456 + "^(?=a)b|b+b+c",
457 + "^z|(?=a)" + "(?:a|aa)" * 5 + "a*a*b$",
458 + "(?m)^a*a*(?=b)b$",
459 + ],
460 +)
461 +def test_static_check_rejects_backtracking_patterns(pattern: str) -> None:
462 + assert not schema_converter._fallback_pattern_is_safe(pattern)
ts/packages/json-schema-to-zod/package.json
+2 -1
@@ -60,6 +60,7 @@
60 },
61 "dependencies": {
62 "@cfworker/json-schema": "^4.1.1",
63 - "dequal": "^2.0.3"
63 + "dequal": "^2.0.3",
64 + "re2js": "^2.8.6"
65 }
66 }
ts/packages/json-schema-to-zod/src/utils/compile-pattern.ts
+65 -8
@@ -1,4 +1,5 @@
1 import type { Refs } from '../types';
2 +import { createPatternMatcher, patternMatches, type PatternMatcher } from './linear-pattern';
3
4 /**
5 * Longest `pattern` / `patternProperties` key this package compiles. Tool
@@ -8,7 +9,7 @@ import type { Refs } from '../types';
9 */
10 export const MAX_PATTERN_LENGTH = 1024;
11
11 -export type InvalidPatternReason = 'syntax' | 'too-long';
12 +export type InvalidPatternReason = 'syntax' | 'too-long' | 'unsupported';
13
14 /**
15 * Raised at conversion time when a schema `pattern` cannot be turned into a
@@ -36,15 +37,59 @@ export class InvalidPatternError extends Error {
37 }
38 }
39
40 +/**
41 + * A schema pattern as a `RegExp` whose matching cannot be made to backtrack
42 + * catastrophically. The native `source` stays intact, so JSON Schema emitted
43 + * from the Zod schema still carries the original pattern for the model, while
44 + * `test` and `exec` go through a {@link PatternMatcher}: RE2 where it can run
45 + * the pattern, otherwise a statically vetted, length-bounded native match, or
46 + * no check at all.
47 + */
48 +export class SchemaPatternRegExp extends RegExp {
49 + readonly matcher: PatternMatcher;
50 +
51 + constructor(pattern: string, matcher: PatternMatcher = createPatternMatcher(pattern)) {
52 + super(pattern);
53 + this.matcher = matcher;
54 + }
55 +
56 + override test(input: string): boolean {
57 + return patternMatches(this.matcher, String(input));
58 + }
59 +
60 + override exec(input: string): RegExpExecArray | null {
61 + const string = String(input);
62 + switch (this.matcher.kind) {
63 + case 'linear':
64 + return this.matcher.regex.exec(string) as RegExpExecArray | null;
65 + case 'backtracking':
66 + return patternMatches(this.matcher, string) ? this.matcher.regex.exec(string) : null;
67 + case 'unenforced':
68 + return null;
69 + }
70 + }
71 +
72 + // `split` and `matchAll` build a derived regex through the species
73 + // constructor, whose signature this class does not share. Only `test` and
74 + // `exec`, which Zod and the guard call, are protected.
75 + static override get [Symbol.species](): RegExpConstructor {
76 + return RegExp;
77 + }
78 +}
79 +
80 /**
81 * Compile a schema pattern, rejecting anything that does not compile or that
82 * exceeds {@link MAX_PATTERN_LENGTH}.
83 *
43 - * Both guards have no false positives: a pattern that fails here would never
44 - * have worked. No backtracking (ReDoS) heuristic is applied on purpose — the
45 - * syntactic checks that exist flag linear patterns such as `^(\d+\.)*\d+$`,
46 - * and a wrong rejection fails the whole tool the schema belongs to. A hostile
47 - * `pattern` that backtracks catastrophically remains a known limitation.
84 + * Matching is protected against catastrophic backtracking (ReDoS): tool
85 + * arguments come from a model that may be steered by untrusted content, and
86 + * object keys are as attacker-influenced as values. Patterns run on RE2,
87 + * which is linear-time. A `pattern` RE2 cannot express (lookaround) falls back
88 + * to the native engine only when it passes a conservative static check, and
89 + * only on bounded input; otherwise it is left unenforced, which widens the
90 + * schema no further than omitting the pattern would. A `patternProperties`
91 + * key decides which schema applies to a property, so it is never silently
92 + * skipped: one RE2 cannot express is rejected at conversion.
93 *
94 * No `u` flag is passed: existing tool schemas rely on identity escapes that
95 * Unicode mode refuses, and the goal here is to fail on defects, not to
@@ -54,7 +99,7 @@ export const compilePattern = (
99 keyword: 'pattern' | 'patternProperties',
100 pattern: string,
101 refs: Pick<Refs, 'path'>
57 -): RegExp => {
102 +): SchemaPatternRegExp => {
103 const path = keyword === 'pattern' ? refs.path : [...refs.path, 'patternProperties', pattern];
104
105 if (pattern.length > MAX_PATTERN_LENGTH) {
@@ -68,9 +113,21 @@ export const compilePattern = (
113 }
114
115 try {
71 - return new RegExp(pattern);
116 + new RegExp(pattern);
117 } catch (cause) {
118 const detail = cause instanceof Error ? cause.message : String(cause);
119 throw new InvalidPatternError(keyword, pattern, path, 'syntax', detail, { cause });
120 }
121 +
122 + const matcher = createPatternMatcher(pattern);
123 + if (keyword === 'patternProperties' && matcher.kind !== 'linear') {
124 + throw new InvalidPatternError(
125 + keyword,
126 + pattern,
127 + path,
128 + 'unsupported',
129 + 'patternProperties keys must not use lookaround or backreferences'
130 + );
131 + }
132 + return new SchemaPatternRegExp(pattern, matcher);
133 };
ts/packages/json-schema-to-zod/src/utils/linear-pattern.ts new
+392
@@ -0,0 +1,392 @@
1 +import { RE2JS } from 're2js';
2 +import { toUnicodePattern } from './unicode-pattern';
3 +
4 +/**
5 + * ECMAScript `\s`: the characters it matches, spelled for an RE2 class body.
6 + * RE2's own `\s` is ASCII-only, so both `\s` and `\S` are spelled out.
7 + */
8 +const WHITESPACE =
9 + '\\t\\n\\x{B}\\f\\r\\x{20}\\x{A0}\\x{1680}\\x{2000}-\\x{200A}\\x{2028}\\x{2029}\\x{202F}\\x{205F}\\x{3000}\\x{FEFF}';
10 +/** The complement of {@link WHITESPACE}, usable inside another class. */
11 +const NON_WHITESPACE =
12 + '\\x{0}-\\x{8}\\x{E}-\\x{1F}\\x{21}-\\x{9F}\\x{A1}-\\x{167F}\\x{1681}-\\x{1FFF}\\x{200B}-\\x{2027}\\x{202A}-\\x{202E}\\x{2030}-\\x{205E}\\x{2060}-\\x{2FFF}\\x{3001}-\\x{FEFE}\\x{FF00}-\\x{10FFFF}';
13 +const ANY_CHARACTER = '\\x{0}-\\x{10FFFF}';
14 +/** ECMAScript `.` without the `s` flag excludes every line terminator, not just `\n`. */
15 +const DOT = '[^\\n\\r\\x{2028}\\x{2029}]';
16 +const SYNTAX_CHARACTERS = new Set('^$\\.*+?()[]{}|/-');
17 +const CONTROL_ESCAPES = new Set('fnrtv');
18 +const PASSTHROUGH_CLASS_ESCAPES = new Set('dDwW');
19 +
20 +const codePoint = (code: number): string => `\\x{${code.toString(16).toUpperCase()}}`;
21 +
22 +/** Thrown inside the translator for constructs RE2 cannot express. */
23 +class Unsupported extends Error {}
24 +
25 +/**
26 + * Reads one escape at `pattern[index] === '\\'` that is valid in both a class
27 + * and the top level, returning its RE2 spelling and the index after it.
28 + */
29 +const readEscape = (pattern: string, index: number, inClass: boolean): [string, number] => {
30 + const escaped = pattern[index + 1] ?? '';
31 +
32 + if (PASSTHROUGH_CLASS_ESCAPES.has(escaped) || CONTROL_ESCAPES.has(escaped)) {
33 + return [`\\${escaped}`, index + 2];
34 + }
35 + if (escaped === 's') {
36 + return [inClass ? WHITESPACE : `[${WHITESPACE}]`, index + 2];
37 + }
38 + if (escaped === 'S') {
39 + return [inClass ? NON_WHITESPACE : `[^${WHITESPACE}]`, index + 2];
40 + }
41 + if (escaped === 'b' || escaped === 'B') {
42 + // Inside a class `\b` is a backspace; `\B` is not allowed there.
43 + return [inClass ? codePoint(8) : `\\${escaped}`, index + 2];
44 + }
45 + if (escaped === '0') {
46 + return [codePoint(0), index + 2];
47 + }
48 + if (escaped === 'c') {
49 + return [codePoint(pattern.charCodeAt(index + 2) % 32), index + 3];
50 + }
51 + if (escaped === 'x') {
52 + return [codePoint(parseInt(pattern.slice(index + 2, index + 4), 16)), index + 4];
53 + }
54 + if (escaped === 'u') {
55 + if (pattern[index + 2] === '{') {
56 + const end = pattern.indexOf('}', index + 3);
57 + return [codePoint(parseInt(pattern.slice(index + 3, end), 16)), end + 1];
58 + }
59 + const unit = parseInt(pattern.slice(index + 2, index + 6), 16);
60 + const low = /^\\u([Dd][C-Fc-f][0-9A-Fa-f]{2})/.exec(pattern.slice(index + 6));
61 + if (unit >= 0xd800 && unit <= 0xdbff && low) {
62 + // The Unicode grammar reads an escaped surrogate pair as one code point.
63 + const combined = (unit - 0xd800) * 0x400 + (parseInt(low[1], 16) - 0xdc00) + 0x10000;
64 + return [codePoint(combined), index + 12];
65 + }
66 + return [codePoint(unit), index + 6];
67 + }
68 + if (SYNTAX_CHARACTERS.has(escaped)) {
69 + return [`\\${escaped}`, index + 2];
70 + }
71 + // Backreferences have no linear-time implementation, and RE2 names Unicode
72 + // properties differently.
73 + throw new Unsupported();
74 +};
75 +
76 +/** Translates a class starting at `pattern[index] === '['`. */
77 +const readClass = (pattern: string, index: number): [string, number] => {
78 + let cursor = index + 1;
79 + const negated = pattern[cursor] === '^';
80 + if (negated) {
81 + cursor++;
82 + }
83 + if (pattern[cursor] === ']') {
84 + // `[]` matches nothing and `[^]` matches everything; RE2 reads a leading
85 + // `]` as a literal instead.
86 + return [negated ? `[${ANY_CHARACTER}]` : `[^${ANY_CHARACTER}]`, cursor + 1];
87 + }
88 +
89 + let body = '';
90 + while (pattern[cursor] !== ']') {
91 + const char = pattern[cursor];
92 + if (char === '\\') {
93 + const [spelled, next] = readEscape(pattern, cursor, true);
94 + body += spelled;
95 + cursor = next;
96 + } else {
97 + // `[` and `^` are literals here but could start a POSIX class or a
98 + // negation in RE2.
99 + body += char === '[' || char === '^' ? `\\${char}` : char;
100 + cursor += char.length;
101 + }
102 + }
103 + return [`[${negated ? '^' : ''}${body}]`, cursor + 1];
104 +};
105 +
106 +/**
107 + * Translates a pattern written in the ECMAScript Unicode (`u` flag) grammar
108 + * into RE2 syntax with the same meaning. The input must already compile with
109 + * the `u` flag. Returns `undefined` for constructs that need a backtracking
110 + * engine (lookaround, backreferences) or that RE2 spells differently.
111 + */
112 +export const unicodePatternToRe2 = (pattern: string): string | undefined => {
113 + let result = '';
114 + let index = 0;
115 + try {
116 + while (index < pattern.length) {
117 + const char = pattern[index];
118 + if (char === '\\') {
119 + const [spelled, next] = readEscape(pattern, index, false);
120 + result += spelled;
121 + index = next;
122 + } else if (char === '[') {
123 + const [spelled, next] = readClass(pattern, index);
124 + result += spelled;
125 + index = next;
126 + } else if (char === '.') {
127 + result += DOT;
128 + index++;
129 + } else if (char === '(' && pattern[index + 1] === '?') {
130 + if (pattern[index + 2] === ':') {
131 + result += '(?:';
132 + index += 3;
133 + } else if (pattern[index + 2] === '<' && !'=!'.includes(pattern[index + 3] ?? '')) {
134 + // A named group keeps its number; the name is irrelevant to matching.
135 + result += '(';
136 + index = pattern.indexOf('>', index) + 1;
137 + } else {
138 + throw new Unsupported();
139 + }
140 + } else {
141 + result += char;
142 + index++;
143 + }
144 + }
145 + } catch (error) {
146 + if (error instanceof Unsupported) {
147 + return undefined;
148 + }
149 + throw error;
150 + }
151 + return result;
152 +};
153 +
154 +/**
155 + * Compiles a Unicode-grammar pattern for RE2JS, whose matching time is linear
156 + * in the input for every pattern. Returns `undefined` when RE2 cannot express
157 + * the pattern.
158 + */
159 +export const compileLinearPattern = (unicodePattern: string): RE2JS | undefined => {
160 + const re2 = unicodePatternToRe2(unicodePattern);
161 + if (re2 === undefined) {
162 + return undefined;
163 + }
164 + try {
165 + return RE2JS.compile(re2);
166 + } catch {
167 + return undefined;
168 + }
169 +};
170 +
171 +/** A quantifier whose range (maximum minus minimum) exceeds this counts as unbounded. */
172 +const LARGE_QUANTIFIER_RANGE = 16;
173 +/** Bound on the product of the ranges of all small variable quantifiers. */
174 +const MAX_SMALL_QUANTIFIER_PRODUCT = 64;
175 +/** Absolute explicit repeat bounds, including repetitions of zero-width assertions. */
176 +const MAX_FALLBACK_REPEAT_COUNT = 64;
177 +/** Bound on the polynomial degree of the backtracking search in the input length. */
178 +const MAX_BACKTRACKING_DEGREE = 2;
179 +const BRACE_QUANTIFIER = /^\{(\d+)(,(\d*))?\}/;
180 +
181 +type Group = {
182 + lookaround: boolean;
183 + quantified: boolean;
184 + alternation: boolean;
185 + branches: number;
186 +};
187 +
188 +/**
189 + * Whether a pattern that RE2 cannot run (it needs lookaround or a
190 + * backreference) is still safe for the native backtracking engine on short
191 + * input. Deliberately conservative, since only such patterns reach it:
192 + *
193 + * - no backreferences;
194 + * - no quantified group that contains a quantifier or an alternation, the
195 + * shapes behind exponential backtracking (`(a+)+`, `(a|a)*`);
196 + * - no lookaround nested in another lookaround;
197 + * - explicit repetition bounds are capped, including fixed assertions;
198 + * - the search is at most quadratic: one degree for an unanchored start, plus
199 + * the number of unbounded quantifiers at the top level, plus the largest
200 + * number inside a single lookaround (lookarounds are atomic, so they add
201 + * rather than multiply);
202 + * - small variable quantifiers (`?`, `{1,3}`) and alternations multiply the
203 + * work by a bounded constant (`(?:a|aa)(?:a|aa)...b` is exponential without
204 + * any quantifier).
205 + * A leading anchor removes search work only without a top-level alternative.
206 + *
207 + * Callers still bound the input length, which keeps the quadratic case cheap.
208 + */
209 +export const isSafeForBacktracking = (pattern: string): boolean => {
210 + const hasNamedGroup = /\(\?<(?![=!])/.test(pattern);
211 + const newGroup = (lookaround: boolean): Group => ({
212 + lookaround,
213 + quantified: false,
214 + alternation: false,
215 + branches: 1,
216 + });
217 + const stack: Group[] = [newGroup(false)];
218 + let topLevelLarge = 0;
219 + let lookaroundLarge = 0;
220 + let maxLookaroundLarge = 0;
221 + let smallProduct = 1;
222 + let lastAtomGroup: Group | undefined;
223 + let index = 0;
224 +
225 + const insideLookaround = () => stack.some(group => group.lookaround);
226 + const multiply = (factor: number): boolean => {
227 + smallProduct *= factor;
228 + return smallProduct <= MAX_SMALL_QUANTIFIER_PRODUCT;
229 + };
230 +
231 + while (index < pattern.length) {
232 + const char = pattern[index];
233 + const current = stack[stack.length - 1];
234 +
235 + let quantifierLength = 0;
236 + let range = 0;
237 + if (char === '*' || char === '+') {
238 + quantifierLength = 1;
239 + range = Infinity;
240 + } else if (char === '?') {
241 + quantifierLength = 1;
242 + range = 1;
243 + } else if (char === '{') {
244 + const brace = BRACE_QUANTIFIER.exec(pattern.slice(index));
245 + if (brace) {
246 + quantifierLength = brace[0].length;
247 + const minimum = Number(brace[1]);
248 + const maximum = brace[2] === undefined ? minimum : Number(brace[3]);
249 + if (
250 + minimum > MAX_FALLBACK_REPEAT_COUNT ||
251 + (brace[3] !== '' && maximum > MAX_FALLBACK_REPEAT_COUNT)
252 + ) {
253 + return false;
254 + }
255 + if (brace[2] !== undefined) {
256 + range = brace[3] === '' ? Infinity : Number(brace[3]) - Number(brace[1]);
257 + }
258 + }
259 + }
260 +
261 + if (quantifierLength > 0) {
262 + if (lastAtomGroup && (lastAtomGroup.quantified || lastAtomGroup.alternation)) {
263 + return false;
264 + }
265 + if (range > LARGE_QUANTIFIER_RANGE) {
266 + if (insideLookaround()) {
267 + maxLookaroundLarge = Math.max(maxLookaroundLarge, ++lookaroundLarge);
268 + } else {
269 + topLevelLarge++;
270 + }
271 + } else if (range > 0 && !multiply(range + 1)) {
272 + return false;
273 + }
274 + current.quantified = true;
275 + index += quantifierLength;
276 + if (pattern[index] === '?') {
277 + index++;
278 + }
279 + lastAtomGroup = undefined;
280 + continue;
281 + }
282 +
283 + lastAtomGroup = undefined;
284 + if (char === '\\') {
285 + const escaped = pattern[index + 1] ?? '';
286 + if (/^[1-9]$/.test(escaped) || (escaped === 'k' && hasNamedGroup)) {
287 + return false;
288 + }
289 + index += 2;
290 + } else if (char === '[') {
291 + index++;
292 + while (index < pattern.length && pattern[index] !== ']') {
293 + index += pattern[index] === '\\' ? 2 : 1;
294 + }
295 + index++;
296 + } else if (char === '(') {
297 + const lookaround =
298 + pattern[index + 1] === '?' &&
299 + ('=!'.includes(pattern[index + 2] ?? '') ||
300 + (pattern[index + 2] === '<' && '=!'.includes(pattern[index + 3] ?? '')));
301 + if (lookaround) {
302 + if (insideLookaround()) {
303 + return false;
304 + }
305 + lookaroundLarge = 0;
306 + }
307 + stack.push(newGroup(lookaround));
308 + index++;
309 + if (pattern[index] === '?') {
310 + // Skip the group prefix so its `?` is not read as a quantifier.
311 + index += lookaround && pattern[index + 1] === '<' ? 3 : 2;
312 + }
313 + } else if (char === ')') {
314 + const closed = stack.length > 1 ? stack.pop()! : current;
315 + if (!multiply(closed.branches)) {
316 + return false;
317 + }
318 + const parent = stack[stack.length - 1];
319 + parent.quantified ||= closed.quantified;
320 + parent.alternation ||= closed.alternation;
321 + lastAtomGroup = closed;
322 + index++;
323 + } else {
324 + if (char === '|') {
325 + current.alternation = true;
326 + current.branches++;
327 + }
328 + index++;
329 + }
330 + }
331 + if (!multiply(stack[0].branches)) {
332 + return false;
333 + }
334 +
335 + const unanchored = pattern.startsWith('^') && stack[0].branches === 1 ? 0 : 1;
336 + return unanchored + topLevelLarge + maxLookaroundLarge <= MAX_BACKTRACKING_DEGREE;
337 +};
338 +
339 +/**
340 + * Longest input the backtracking fallback runs on. Longer input is treated as
341 + * matching: skipping a constraint only widens what the schema accepts, which
342 + * a hostile schema could do anyway by omitting the pattern.
343 + */
344 +export const MAX_BACKTRACKING_INPUT_LENGTH = 1000;
345 +
346 +/**
347 + * How a schema pattern is evaluated:
348 + *
349 + * - `linear`: RE2JS runs it, in time linear in the input.
350 + * - `backtracking`: the pattern needs lookaround, but passes
351 + * {@link isSafeForBacktracking}; the native engine runs it on input up to
352 + * {@link MAX_BACKTRACKING_INPUT_LENGTH} characters.
353 + * - `unenforced`: neither applies, so the constraint is not checked rather
354 + * than exposing validation to catastrophic backtracking.
355 + */
356 +export type PatternMatcher =
357 + | { readonly kind: 'linear'; readonly regex: RE2JS }
358 + | { readonly kind: 'backtracking'; readonly regex: RegExp }
359 + | { readonly kind: 'unenforced' };
360 +
361 +/**
362 + * Builds the matcher for a schema pattern in the legacy (no `u` flag) grammar
363 + * the rest of this package accepts. `pattern` must already compile as a
364 + * `RegExp`.
365 + */
366 +export const createPatternMatcher = (pattern: string): PatternMatcher => {
367 + const unicodePattern = toUnicodePattern(pattern);
368 + const linear = unicodePattern === undefined ? undefined : compileLinearPattern(unicodePattern);
369 + if (linear !== undefined) {
370 + return { kind: 'linear', regex: linear };
371 + }
372 + if (isSafeForBacktracking(pattern)) {
373 + return { kind: 'backtracking', regex: new RegExp(pattern) };
374 + }
375 + return { kind: 'unenforced' };
376 +};
377 +
378 +/** Whether `input` satisfies the pattern behind `matcher`. */
379 +export const patternMatches = (matcher: PatternMatcher, input: string): boolean => {
380 + switch (matcher.kind) {
381 + case 'linear':
382 + return matcher.regex.test(input);
383 + case 'backtracking':
384 + if (input.length > MAX_BACKTRACKING_INPUT_LENGTH) {
385 + return true;
386 + }
387 + matcher.regex.lastIndex = 0;
388 + return matcher.regex.test(input);
389 + case 'unenforced':
390 + return true;
391 + }
392 +};
ts/packages/json-schema-to-zod/src/whole-schema-validation.ts
+337 -70
@@ -1,8 +1,17 @@
1 -import { encodePointer, format, Validator } from '@cfworker/json-schema';
1 +import {
2 + encodePointer,
3 + format,
4 + ignoredKeyword,
5 + schemaArrayKeyword,
6 + schemaMapKeyword,
7 + Validator,
8 +} from '@cfworker/json-schema';
9 import type { Schema as InterpreterSchema } from '@cfworker/json-schema';
10 import { z } from 'zod/v3';
11
12 import type { JsonSchema, JsonSchemaObject } from './types';
13 +import { InvalidPatternError } from './utils/compile-pattern';
14 +import { createPatternMatcher, patternMatches, type PatternMatcher } from './utils/linear-pattern';
15 import { toUnicodePattern } from './utils/unicode-pattern';
16
17 const REQUIRES_WHOLE_SCHEMA_VALIDATION = new Set([
@@ -65,6 +74,49 @@ const SCHEMA_VALUE_KEYWORDS = new Set([
74 const isObject = (value: unknown): value is Record<string, unknown> =>
75 typeof value === 'object' && value !== null && !Array.isArray(value);
76
77 +type SchemaNode = { value: Record<string, unknown>; path: ReadonlyArray<string | number> };
78 +
79 +/**
80 + * Every object the interpreter registers as a schema, found with its own walk
81 + * (`dereference`): each key it does not ignore, extension locations included.
82 + * A `$ref` resolves only through that registry, whether it is a JSON Pointer,
83 + * an `$id` or an `$anchor`, so these are all the schemas the interpreter can
84 + * ever validate against.
85 + */
86 +function* interpreterSchemaNodes(
87 + value: unknown,
88 + path: ReadonlyArray<string | number> = [],
89 + seen: WeakSet<object> = new WeakSet()
90 +): Generator<SchemaNode> {
91 + if (!isObject(value) || seen.has(value)) {
92 + return;
93 + }
94 + seen.add(value);
95 + yield { value, path };
96 +
97 + for (const [key, child] of Object.entries(value)) {
98 + // Indexed like `dereference` does, so inherited names are skipped alike.
99 + if ((ignoredKeyword as Record<string, boolean>)[key]) {
100 + continue;
101 + }
102 + if (Array.isArray(child)) {
103 + if ((schemaArrayKeyword as Record<string, boolean>)[key]) {
104 + for (const [index, nested] of child.entries()) {
105 + yield* interpreterSchemaNodes(nested, [...path, key, index], seen);
106 + }
107 + }
108 + } else if ((schemaMapKeyword as Record<string, boolean>)[key]) {
109 + if (isObject(child)) {
110 + for (const [name, nested] of Object.entries(child)) {
111 + yield* interpreterSchemaNodes(nested, [...path, key, name], seen);
112 + }
113 + }
114 + } else {
115 + yield* interpreterSchemaNodes(child, [...path, key], seen);
116 + }
117 + }
118 +}
119 +
120 const hasRequiredDefault = (schema: JsonSchemaObject): boolean => {
121 if (!Array.isArray(schema.required) || !isObject(schema.properties)) {
122 return false;
@@ -162,33 +214,180 @@ export const guardSchemaAt = (
214 { ...refs.root, $ref: `#/${refs.path.map(part => encodePointer(String(part))).join('/')}` }
215 : node;
216
165 -const patternKeyRenames = new WeakMap<object, ReadonlyMap<string, string>>();
217 +/** The interpreter spelling of one `patternProperties` key. */
218 +type PatternKeySpelling = (pattern: string) => string;
219 +/** The interpreter spelling of every key of one `patternProperties` object. */
220 +type PatternKeyRenamer = (patternProperties: object) => ReadonlyMap<string, string>;
221
222 /**
168 - * The Unicode spelling of each `patternProperties` key. Two keys that spell
169 - * the same way (`^a\_b$` and `^a_b$`) stay separate entries: the later one is
170 - * wrapped in a non-capturing group, which matches the same names, so neither
171 - * value schema is dropped. A key with no Unicode spelling is kept as is and
172 - * surfaces as a guard failure rather than an unenforced constraint.
223 + * Spells each key of a `patternProperties` object with `spell`. Two keys that
224 + * spell the same way (`^a\_b$` and `^a_b$`) stay separate entries: the later
225 + * one is wrapped in a non-capturing group, which matches the same names, so
226 + * neither value schema is dropped. `renamed` collects every spelling that
227 + * differs from its key.
228 */
174 -const renamePatternKeys = (patternProperties: object): ReadonlyMap<string, string> => {
175 - const cached = patternKeyRenames.get(patternProperties);
176 - if (cached) {
177 - return cached;
229 +const patternKeyRenamer = (
230 + spell: PatternKeySpelling,
231 + renamed: Map<string, string> = new Map()
232 +): PatternKeyRenamer => {
233 + const cache = new WeakMap<object, ReadonlyMap<string, string>>();
234 + return patternProperties => {
235 + const cached = cache.get(patternProperties);
236 + if (cached) {
237 + return cached;
238 + }
239 +
240 + const renames = new Map<string, string>();
241 + const used = new Set<string>();
242 + for (const key of Object.keys(patternProperties)) {
243 + let spelling = spell(key);
244 + while (used.has(spelling)) {
245 + spelling = `(?:${spelling})`;
246 + }
247 + used.add(spelling);
248 + renames.set(key, spelling);
249 + if (spelling !== key) {
250 + renamed.set(spelling, key);
251 + }
252 + }
253 + cache.set(patternProperties, renames);
254 + return renames;
255 + };
256 +};
257 +
258 +/**
259 + * The interpreter matches `patternProperties` keys itself, with a native
260 + * backtracking `RegExp`, against object keys that come from the caller. So a
261 + * key never reaches it as written. Instead, each validation spells it as the
262 + * literal set of instance keys it matches under RE2 (an anchored trie, which
263 + * cannot backtrack beyond the key length). Interpreter matching then agrees
264 + * with RE2 on every key it can see.
265 + *
266 + * Returns the RE2 matcher of each key, or `undefined` for a key that does not
267 + * compile in the Unicode grammar: the interpreter throws on it when reached,
268 + * which fails the guard closed, as it always has. A key that compiles but
269 + * needs a backtracking engine is rejected at conversion, as the native object
270 + * parser does.
271 + */
272 +const collectPatternKeyMatchers = (
273 + nodes: readonly SchemaNode[]
274 +): Map<string, PatternMatcher | undefined> => {
275 + const matchers = new Map<string, PatternMatcher | undefined>();
276 + for (const { value, path } of nodes) {
277 + if (!isObject(value.patternProperties)) {
278 + continue;
279 + }
280 + for (const key of Object.keys(value.patternProperties)) {
281 + if (matchers.has(key)) {
282 + continue;
283 + }
284 + let matcher: PatternMatcher | undefined;
285 + try {
286 + matcher = createPatternMatcher(key);
287 + } catch {
288 + matcher = undefined;
289 + }
290 + if (matcher !== undefined && matcher.kind !== 'linear') {
291 + if (unicodeSpellingCompiles(key)) {
292 + throw new InvalidPatternError(
293 + 'patternProperties',
294 + key,
295 + [...path, 'patternProperties', key],
296 + 'unsupported',
297 + 'patternProperties keys must not use lookaround or backreferences'
298 + );
299 + }
300 + matcher = undefined;
301 + }
302 + matchers.set(key, matcher);
303 + }
304 + }
305 +
306 + return matchers;
307 +};
308 +
309 +/** The Unicode spelling of a key that cannot be matched with RE2. */
310 +const fallbackKeySpelling = (pattern: string): string => toUnicodePattern(pattern) ?? pattern;
311 +
312 +const unicodeSpellingCompiles = (pattern: string): boolean => {
313 + try {
314 + new RegExp(fallbackKeySpelling(pattern), 'u');
315 + return true;
316 + } catch {
317 + return false;
318 }
319 +};
320
180 - const renames = new Map<string, string>();
181 - const used = new Set<string>();
182 - for (const key of Object.keys(patternProperties)) {
183 - let renamed = toUnicodePattern(key) ?? key;
184 - while (used.has(renamed)) {
185 - renamed = `(?:${renamed})`;
321 +/** Every object key anywhere in `value`. */
322 +const collectInstanceKeys = (
323 + value: unknown,
324 + keys: Set<string> = new Set(),
325 + seen: WeakSet<object> = new WeakSet()
326 +): Set<string> => {
327 + if (typeof value !== 'object' || value === null || seen.has(value)) {
328 + return keys;
329 + }
330 + seen.add(value);
331 + if (Array.isArray(value)) {
332 + value.forEach(item => collectInstanceKeys(item, keys, seen));
333 + } else {
334 + for (const [key, child] of Object.entries(value)) {
335 + keys.add(key);
336 + collectInstanceKeys(child, keys, seen);
337 }
187 - used.add(renamed);
188 - renames.set(key, renamed);
338 }
190 - patternKeyRenames.set(patternProperties, renames);
191 - return renames;
339 + return keys;
340 +};
341 +
342 +const UNICODE_SYNTAX_CHARACTER = /[$()*+./?[\\\]^{|}]/g;
343 +
344 +type KeyTrie = { end: boolean; children: Map<string, KeyTrie> };
345 +
346 +/** Spells a trie node as a Unicode-grammar pattern over its code points. */
347 +const spellKeyTrie = (node: KeyTrie): string => {
348 + const branches = [...node.children].map(([char, child]) => {
349 + let spelling = char.replace(UNICODE_SYNTAX_CHARACTER, '\\$&');
350 + let next = child;
351 + // Collapse single-path runs so nesting grows only at branch points.
352 + while (!next.end && next.children.size === 1) {
353 + const [[nextChar, grandchild]] = next.children;
354 + spelling += nextChar.replace(UNICODE_SYNTAX_CHARACTER, '\\$&');
355 + next = grandchild;
356 + }
357 + return spelling + spellKeyTrie(next);
358 + });
359 + if (branches.length === 0) {
360 + return '';
361 + }
362 + if (branches.length === 1 && !node.end) {
363 + return branches[0];
364 + }
365 + return `(?:${branches.join('|')}${node.end ? '|' : ''})`;
366 +};
367 +
368 +/**
369 + * An anchored Unicode-grammar pattern that matches exactly `keys`. Branches of
370 + * a trie start with distinct characters, so matching never backtracks further
371 + * than the key being tested. No keys gives `[]`, which matches nothing.
372 + */
373 +const literalKeysPattern = (keys: readonly string[]): string => {
374 + if (keys.length === 0) {
375 + return '[]';
376 + }
377 + const root: KeyTrie = { end: false, children: new Map() };
378 + for (const key of keys) {
379 + let node = root;
380 + for (const char of key) {
381 + let child = node.children.get(char);
382 + if (!child) {
383 + child = { end: false, children: new Map() };
384 + node.children.set(char, child);
385 + }
386 + node = child;
387 + }
388 + node.end = true;
389 + }
390 + return `^${spellKeyTrie(root)}$`;
391 };
392
393 const decodePointerSegment = (segment: string): string => {
@@ -228,7 +427,12 @@ const positionAfter = (position: PointerPosition, key: string, child: unknown):
427 * a segment in keyword position is a keyword, so a definition that happens to
428 * be named `patternProperties` is not mistaken for one.
429 */
231 -const renameRefThroughPatternKeys = (ref: string, root: unknown): string => {
430 +const renameRefThroughPatternKeys = (
431 + ref: string,
432 + root: unknown,
433 + renamePatternKeys: PatternKeyRenamer,
434 + schemaObjects: WeakSet<object>
435 +): string => {
436 if (!ref.startsWith('#/')) {
437 return ref;
438 }
@@ -240,6 +444,9 @@ const renameRefThroughPatternKeys = (ref: string, root: unknown): string => {
444 .slice(2)
445 .split('/')
446 .map(segment => {
447 + if (position === 'other' && isObject(node) && schemaObjects.has(node)) {
448 + position = 'keyword';
449 + }
450 const key = decodePointerSegment(segment);
451 let result = segment;
452 if (position === 'patternKey' && isObject(node)) {
@@ -265,21 +472,31 @@ const PATTERN_FORMAT_PREFIX = 'composio-pattern:';
472 type PatternFormats = Map<string, (value: string) => boolean>;
473
474 /**
268 - * Adds `pattern` to `formats` as an interpreter format that tests it exactly as
269 - * the native string parser does, without the `u` flag, and returns the format
270 - * name. Returns `undefined` for a pattern that does not compile at all, which
271 - * the interpreter then reports as it always has.
475 + * Adds `pattern` to `formats` as an interpreter format that tests it as the
476 + * native string parser does (legacy grammar, RE2 with a bounded fallback; see
477 + * `compilePattern`), and returns the format name. Returns `undefined` for a
478 + * pattern that does not compile at all, which the interpreter then reports as
479 + * it always has. `matchers` caches compiled patterns across validations.
480 */
273 -const patternFormat = (pattern: string, formats: PatternFormats): string | undefined => {
481 +const patternFormat = (
482 + pattern: string,
483 + formats: PatternFormats,
484 + matchers: Map<string, PatternMatcher>
485 +): string | undefined => {
486 const name = `${PATTERN_FORMAT_PREFIX}${pattern}`;
487 if (!formats.has(name)) {
276 - let regex: RegExp;
277 - try {
278 - regex = new RegExp(pattern);
279 - } catch {
280 - return undefined;
488 + let matcher = matchers.get(pattern);
489 + if (matcher === undefined) {
490 + try {
491 + new RegExp(pattern);
492 + } catch {
493 + return undefined;
494 + }
495 + matcher = createPatternMatcher(pattern);
496 + matchers.set(pattern, matcher);
497 }
282 - formats.set(name, value => regex.test(value));
498 + const compiled = matcher;
499 + formats.set(name, value => patternMatches(compiled, value));
500 }
501 return name;
502 };
@@ -291,25 +508,22 @@ const patternFormat = (pattern: string, formats: PatternFormats): string | undef
508 * - OpenAPI 3.0 / Draft 4 spell exclusive bounds as a boolean flag next to
509 * `minimum`/`maximum`. The interpreter ignores the flag, so it receives the
510 * numeric spelling the native number parser already honors.
294 - * - The interpreter compiles patterns with the `u` flag, which refuses legacy
295 - * syntax tool schemas use, such as `\_`. A `pattern` becomes a format that
296 - * tests it without the flag, exactly like the native string parser. A
297 - * `patternProperties` key cannot leave the interpreter, so it gets its
298 - * Unicode spelling (`toUnicodePattern`), and local `$ref`s through a renamed
299 - * key follow the rename. `root` is the unmodified document those refs
300 - * address. `formats` collects the pattern formats.
511 + * - The interpreter compiles patterns natively with the `u` flag, which
512 + * refuses legacy syntax tool schemas use, such as `\_`, and backtracks. A
513 + * `pattern` becomes a format that tests it exactly like the native string
514 + * parser. A `patternProperties` key cannot leave the interpreter, so it is
515 + * respelled by `renamePatternKeys`, and local `$ref`s through a renamed key
516 + * follow the rename. `root` is the unmodified document those refs address.
517 + * `formats` collects the pattern formats.
518 */
519 const prepareInterpreterSchema = (
303 - value: unknown,
304 - seen: WeakSet<object>,
520 + value: Record<string, unknown>,
521 root: unknown,
306 - formats: PatternFormats
522 + formats: PatternFormats,
523 + patternMatchers: Map<string, PatternMatcher>,
524 + renamePatternKeys: PatternKeyRenamer,
525 + schemaObjects: WeakSet<object>
526 ): void => {
308 - if (!isObject(value) || seen.has(value)) {
309 - return;
310 - }
311 - seen.add(value);
312 -
527 if (value.exclusiveMinimum === true && typeof value.minimum === 'number') {
528 value.exclusiveMinimum = value.minimum;
529 delete value.minimum;
@@ -324,10 +538,12 @@ const prepareInterpreterSchema = (
538 }
539
540 if (typeof value.$ref === 'string') {
327 - value.$ref = renameRefThroughPatternKeys(value.$ref, root);
541 + value.$ref = renameRefThroughPatternKeys(value.$ref, root, renamePatternKeys, schemaObjects);
542 }
543 const patternFormatName =
330 - typeof value.pattern === 'string' ? patternFormat(value.pattern, formats) : undefined;
544 + typeof value.pattern === 'string'
545 + ? patternFormat(value.pattern, formats, patternMatchers)
546 + : undefined;
547 if (patternFormatName !== undefined) {
548 delete value.pattern;
549 if (value.format === undefined) {
@@ -346,18 +562,6 @@ const prepareInterpreterSchema = (
562 ])
563 );
564 }
349 -
350 - for (const [key, child] of Object.entries(value)) {
351 - if (SCHEMA_MAP_KEYWORDS.has(key) && isObject(child)) {
352 - Object.values(child).forEach(nested => prepareInterpreterSchema(nested, seen, root, formats));
353 - } else if (SCHEMA_ARRAY_KEYWORDS.has(key) && Array.isArray(child)) {
354 - child.forEach(nested => prepareInterpreterSchema(nested, seen, root, formats));
355 - } else if (SCHEMA_VALUE_KEYWORDS.has(key)) {
356 - (Array.isArray(child) ? child : [child]).forEach(nested =>
357 - prepareInterpreterSchema(nested, seen, root, formats)
358 - );
359 - }
360 - }
565 };
566
567 type WholeSchemaGuard = (value: unknown) => string | undefined;
@@ -403,14 +607,70 @@ const guardedSchemaClass = (Base: ZodSchemaClass): ZodSchemaClass => {
607 return Guarded;
608 };
609
610 +type PreparedInterpreter = {
611 + validator: Validator;
612 + formats: PatternFormats;
613 + /** Interpreter spelling of each respelled `patternProperties` key, to its key. */
614 + renamed: ReadonlyMap<string, string>;
615 +};
616 +
617 export const withWholeSchemaValidation = (
618 jsonSchema: JsonSchema,
619 parsedSchema: z.ZodTypeAny
620 ): z.ZodTypeAny => {
410 - const interpreterSchema = structuredClone(jsonSchema);
411 - const formats: PatternFormats = new Map();
412 - prepareInterpreterSchema(interpreterSchema, new WeakSet(), jsonSchema, formats);
413 - const validator = new Validator(interpreterSchema as InterpreterSchema, '7', false);
621 + const patternMatchers = new Map<string, PatternMatcher>();
622 + const originalNodes = [...interpreterSchemaNodes(jsonSchema)];
623 + const schemaObjects = new WeakSet(originalNodes.map(node => node.value));
624 + const keyMatchers = collectPatternKeyMatchers(originalNodes);
625 +
626 + const prepare = (spell: PatternKeySpelling): PreparedInterpreter => {
627 + const interpreterSchema = structuredClone(jsonSchema);
628 + const formats: PatternFormats = new Map();
629 + const renamed = new Map<string, string>();
630 + const renamePatternKeys = patternKeyRenamer(spell, renamed);
631 + // Capture targets before key renaming changes the paths that address them.
632 + const nodes = [...interpreterSchemaNodes(interpreterSchema)];
633 + for (const { value } of nodes) {
634 + prepareInterpreterSchema(
635 + value,
636 + jsonSchema,
637 + formats,
638 + patternMatchers,
639 + renamePatternKeys,
640 + schemaObjects
641 + );
642 + }
643 + return {
644 + validator: new Validator(interpreterSchema as InterpreterSchema, '7', false),
645 + formats,
646 + renamed,
647 + };
648 + };
649 +
650 + // Without `patternProperties` the interpreter schema never changes. With
651 + // them, it is rebuilt for each value, spelling every key as the instance
652 + // keys it matches.
653 + const shared = keyMatchers.size === 0 ? prepare(fallbackKeySpelling) : undefined;
654 + const prepareFor = (value: unknown): PreparedInterpreter => {
655 + if (shared) {
656 + return shared;
657 + }
658 + const instanceKeys = [...collectInstanceKeys(value)];
659 + const spellings = new Map<string, string>();
660 + return prepare(pattern => {
661 + const cached = spellings.get(pattern);
662 + if (cached !== undefined) {
663 + return cached;
664 + }
665 + const matcher = keyMatchers.get(pattern);
666 + const spelling =
667 + matcher === undefined
668 + ? fallbackKeySpelling(pattern)
669 + : literalKeysPattern(instanceKeys.filter(key => patternMatches(matcher, key)));
670 + spellings.set(pattern, spelling);
671 + return spelling;
672 + });
673 + };
674
675 // Validate the source value before defaults and other Zod transforms run.
676 // Otherwise a missing required field can be synthesized and incorrectly
@@ -421,21 +681,28 @@ export const withWholeSchemaValidation = (
681 // guard's pattern formats live there only for this synchronous call, so
682 // the table does not grow with every distinct pattern ever converted.
683 let result: ReturnType<Validator['validate']>;
684 + let prepared: PreparedInterpreter | undefined;
685 try {
425 - formats.forEach((test, name) => {
686 + prepared = prepareFor(value);
687 + prepared.formats.forEach((test, name) => {
688 format[name] = test;
689 });
428 - result = validator.validate(value);
690 + result = prepared.validator.validate(value);
691 } catch (cause) {
692 return `JSON Schema validation failed: ${String(cause)}`;
693 } finally {
432 - formats.forEach((_, name) => {
694 + prepared?.formats.forEach((_, name) => {
695 delete format[name];
696 });
697 }
698
699 if (!result.valid) {
438 - return result.errors[0]?.error ?? 'Input does not satisfy the complete JSON Schema.';
700 + let error = result.errors[0]?.error ?? 'Input does not satisfy the complete JSON Schema.';
701 + // Report the schema's own pattern, not its per-value spelling.
702 + prepared.renamed.forEach((key, spelling) => {
703 + error = error.split(`"${spelling}"`).join(`"${key}"`);
704 + });
705 + return error;
706 }
707 return innerGuard?.(value);
708 };
ts/packages/json-schema-to-zod/test/linear-pattern.test.ts new
+368
@@ -0,0 +1,368 @@
1 +import fc from 'fast-check';
2 +import { describe, expect, it, vi } from 'vitest';
3 +
4 +import { InvalidPatternError, jsonSchemaToZod } from '../src/index';
5 +import type { JsonSchema } from '../src/types';
6 +import { compilePattern } from '../src/utils/compile-pattern';
7 +import {
8 + MAX_BACKTRACKING_INPUT_LENGTH,
9 + compileLinearPattern,
10 + createPatternMatcher,
11 + isSafeForBacktracking,
12 +} from '../src/utils/linear-pattern';
13 +import { toUnicodePattern } from '../src/utils/unicode-pattern';
14 +
15 +/** Patterns whose RE2 translation must agree with the native Unicode engine. */
16 +const TRANSLATED_PATTERNS = [
17 + '^\\s+$',
18 + '^\\S+$',
19 + '^[\\s\\S]$',
20 + '^[^\\s]+$',
21 + '^[^\\S]$',
22 + '^.$',
23 + '^..$',
24 + '^a.b$',
25 + '\\u00e9',
26 + '^\\ud83d\\ude00$',
27 + '^\\u{1F600}$',
28 + '^[]$',
29 + '^[^]$',
30 + '^[\\b]$',
31 + '^\\cJ$',
32 + '^\\0$',
33 + '^(?<word>a)\\d$',
34 + '\\bfoo\\b',
35 + '\\Bo',
36 + '^[\\[^]+$',
37 + '^a{2,3}$',
38 + '^\\x41$',
39 + '^[\\w-]+$',
40 + '^\\D\\W$',
41 + '^$',
42 + '^(?:ab|a)b?$',
43 + '^[a-z\\-\\]]$',
44 + '^\\/\\.\\*$',
45 +] as const;
46 +
47 +const UNITS = [
48 + ' ',
49 + '\t',
50 + '\n',
51 + '\r',
52 + '\v',
53 + '\u00a0',
54 + '\u1680',
55 + '\u2028',
56 + '\u2029',
57 + '\ufeff',
58 + '\u0085',
59 + 'a',
60 + 'b',
61 + 'A',
62 + '0',
63 + '_',
64 + '\b',
65 + '\u0000',
66 + '\u{1F600}',
67 + '\ud83d',
68 + '\u00e9',
69 + '-',
70 + '[',
71 + ']',
72 + '^',
73 + 'f',
74 + 'o',
75 + '/',
76 + '.',
77 + '*',
78 +];
79 +
80 +describe('unicodePatternToRe2', () => {
81 + it.each(TRANSLATED_PATTERNS)('matches %s exactly like the native Unicode engine', pattern => {
82 + const unicodePattern = toUnicodePattern(pattern) ?? pattern;
83 + const linear = compileLinearPattern(unicodePattern);
84 + expect(linear).toBeDefined();
85 + const native = new RegExp(unicodePattern, 'u');
86 +
87 + fc.assert(
88 + fc.property(fc.string({ unit: fc.constantFrom(...UNITS), maxLength: 4 }), input => {
89 + expect(linear!.test(input)).toBe(native.test(input));
90 + }),
91 + { numRuns: 300 }
92 + );
93 + });
94 +
95 + it.each(['^(?=a)a$', '(?<=a)b', '(?<!a)b', '^(a)\\1$', '^(?<n>a)\\k<n>$'])(
96 + 'leaves %s to the backtracking engine',
97 + pattern => {
98 + expect(compileLinearPattern(pattern)).toBeUndefined();
99 + }
100 + );
101 +});
102 +
103 +describe('isSafeForBacktracking', () => {
104 + it.each([
105 + '^(?=.*[a-y])a',
106 + '^(?=.*[a-z])(?=.*[A-Z])(?=.*\\d)[A-Za-z\\d]{8,}$',
107 + '^(?=a){2}ab$',
108 + '^(?!admin$)[a-z]+$',
109 + '^(?:jpg|png|gif)(?=\\.)',
110 + ])('accepts %s', pattern => {
111 + expect(isSafeForBacktracking(pattern)).toBe(true);
112 + });
113 +
114 + it.each([
115 + ['nested quantifiers', '^(?=(a+)+$)'],
116 + ['a quantified alternation', '^(?=x)(a|aa)*$'],
117 + ['a backreference', '^(a)\\1$'],
118 + ['a named backreference', '^(?<n>a)\\k<n>$'],
119 + ['polynomial quantifiers', '(?=x).*.*y'],
120 + ['two unbounded quantifiers in one lookaround', '(?=.*a.*b)x'],
121 + ['a nested lookaround', '^(?=(?=a)a)'],
122 + ['an unquantified alternation chain', `(?<=)${'(?:a|aa)'.repeat(28)}b`],
123 + ['many optional atoms', `^(?=x)${'a?'.repeat(30)}${'a'.repeat(30)}$`],
124 + ['huge fixed assertion repeats', '^(?=a){1000000000}a$'],
125 + ['huge fixed consuming repeats', '^(?=a)a{1000000000}$'],
126 + ['huge finite upper bounds', '^(?=a)a{1,1000000000}$'],
127 + ['huge unbounded minimums', '^(?=a)a{1000000000,}$'],
128 + ['nested fixed assertion repeats', '^((?=a){64}){64}a$'],
129 + ['an unanchored alternative', '^(?=a)b|b+b+c'],
130 + ['an unanchored alternative with bounded choices', `^z|(?=a)${'(?:a|aa)'.repeat(5)}a*a*b$`],
131 + ])('rejects %s', (_, pattern) => {
132 + expect(isSafeForBacktracking(pattern)).toBe(false);
133 + });
134 +});
135 +
136 +const finishesQuickly = (run: () => void): void => {
137 + const start = performance.now();
138 + run();
139 + expect(performance.now() - start).toBeLessThan(1000);
140 +};
141 +
142 +const ATTACK = `${'a'.repeat(40)}!`;
143 +
144 +describe('catastrophic backtracking (SEC-1178)', () => {
145 + it.each(['^(?=a){1000000000}a$', `^z|(?=a)${'(?:a|aa)'.repeat(5)}a*a*b$`])(
146 + 'leaves unsafe fallback pattern %s unenforced',
147 + pattern => {
148 + expect(createPatternMatcher(pattern).kind).toBe('unenforced');
149 + const parsed = jsonSchemaToZod({ type: 'string', pattern });
150 + expect(parsed.safeParse('a'.repeat(MAX_BACKTRACKING_INPUT_LENGTH)).success).toBe(true);
151 + }
152 + );
153 +
154 + it.each(['x', 'components', 'escaped', '$id', '$anchor'])(
155 + 'protects regexes referenced through %s',
156 + location => {
157 + const pattern = '^(a+)+$';
158 + let nativeExecutions = 0;
159 + const originalExec = RegExp.prototype.exec;
160 + const nativeExec = vi.spyOn(RegExp.prototype, 'exec').mockImplementation(function (
161 + this: RegExp,
162 + input: string
163 + ) {
164 + if (this.source === pattern) {
165 + nativeExecutions++;
166 + throw new Error('schema regex reached native matching');
167 + }
168 + return originalExec.call(this, input);
169 + });
170 + try {
171 + for (const keyword of ['pattern', 'patternProperties']) {
172 + const target =
173 + keyword === 'pattern'
174 + ? { type: 'string', pattern }
175 + : {
176 + type: 'object',
177 + patternProperties: { [pattern]: { type: 'integer' } },
178 + additionalProperties: false,
179 + };
180 + const { extension, ref } = {
181 + x: { extension: { x: target }, ref: '#/x' },
182 + components: {
183 + extension: { components: { schemas: { target } } },
184 + ref: '#/components/schemas/target',
185 + },
186 + escaped: { extension: { 'schema/~target': target }, ref: '#/schema~1~0target' },
187 + // The interpreter also resolves identifiers it registers anywhere.
188 + $id: {
189 + extension: { x: { $id: 'https://example.com/target', ...target } },
190 + ref: 'https://example.com/target',
191 + },
192 + $anchor: { extension: { x: { $anchor: 'target', ...target } }, ref: '#target' },
193 + }[location]!;
194 + const parsed = jsonSchemaToZod({
195 + type: 'object',
196 + properties: { value: { $ref: ref } },
197 + ...extension,
198 + } as JsonSchema);
199 + expect(
200 + parsed.safeParse({ value: keyword === 'pattern' ? 'aaa' : { aaa: 1 } }).success
201 + ).toBe(true);
202 + expect(
203 + parsed.safeParse({ value: keyword === 'pattern' ? 'a!' : { 'a!': 1 } }).success
204 + ).toBe(false);
205 + if (keyword === 'patternProperties') {
206 + expect(parsed.safeParse({ value: { aaa: 'bad' } }).success).toBe(false);
207 + }
208 + }
209 + expect(nativeExecutions).toBe(0);
210 + } finally {
211 + nativeExec.mockRestore();
212 + }
213 + }
214 + );
215 +
216 + it.each([
217 + ['#/x', {}],
218 + ['https://example.com/target', { $id: 'https://example.com/target' }],
219 + ['#target', { $anchor: 'target' }],
220 + ])(
221 + 'rejects unsupported key patterns in extension reference targets (%s) during conversion',
222 + (ref, identifier) => {
223 + expect(() =>
224 + jsonSchemaToZod({
225 + type: 'object',
226 + properties: { value: { $ref: ref } },
227 + x: {
228 + ...identifier,
229 + type: 'object',
230 + patternProperties: { '^(?=a)': { type: 'integer' } },
231 + },
232 + } as JsonSchema)
233 + ).toThrow(InvalidPatternError);
234 + }
235 + );
236 +
237 + it('keeps references through renamed keys in extension schemas valid', () => {
238 + const parsed = jsonSchemaToZod({
239 + type: 'object',
240 + properties: {
241 + keys: { $ref: '#/x' },
242 + value: { $ref: '#/x/patternProperties/^k' },
243 + },
244 + x: { type: 'object', patternProperties: { '^k': { type: 'integer' } } },
245 + } as JsonSchema);
246 + expect(parsed.safeParse({ keys: { k1: 1 }, value: 2 }).success).toBe(true);
247 + expect(parsed.safeParse({ keys: { k1: 1 }, value: 'bad' }).success).toBe(false);
248 + });
249 +
250 + it('follows reference cycles without rewriting instance data as schemas', () => {
251 + const parsed = jsonSchemaToZod({
252 + type: 'object',
253 + properties: { value: { $ref: '#/x' } },
254 + x: {
255 + type: 'object',
256 + properties: { next: { $ref: '#/x' } },
257 + default: { patternProperties: { '^(?=a)': {} } },
258 + },
259 + } as JsonSchema);
260 + expect(parsed.safeParse({ value: { next: {} } }).success).toBe(true);
261 + });
262 +
263 + it('runs a nested-quantifier pattern in linear time', () => {
264 + const parsed = jsonSchemaToZod({ type: 'string', pattern: '^(a+)+$' });
265 + finishesQuickly(() => expect(parsed.safeParse(ATTACK).success).toBe(false));
266 + expect(parsed.safeParse('aaa').success).toBe(true);
267 + });
268 +
269 + it('runs a backtracking patternProperties key in linear time', () => {
270 + const parsed = jsonSchemaToZod({
271 + type: 'object',
272 + patternProperties: { '^(a|a)*$': { type: 'integer' } },
273 + additionalProperties: false,
274 + });
275 + finishesQuickly(() => expect(parsed.safeParse({ [ATTACK]: 1 }).success).toBe(false));
276 + expect(parsed.safeParse({ aaa: 1 }).success).toBe(true);
277 + expect(parsed.safeParse({ aaa: 'x' }).success).toBe(false);
278 + });
279 +
280 + it('runs patterns that only the whole-schema guard sees in linear time', () => {
281 + const schema: JsonSchema = {
282 + type: 'object',
283 + properties: { handle: { $ref: '#/$defs/handle' }, tags: { $ref: '#/$defs/tags' } },
284 + $defs: {
285 + handle: { type: 'string', pattern: '^(a+)+$' },
286 + tags: {
287 + type: 'object',
288 + patternProperties: { '^(a|a)*$': { type: 'integer' } },
289 + additionalProperties: false,
290 + },
291 + },
292 + };
293 + const parsed = jsonSchemaToZod(schema);
294 +
295 + finishesQuickly(() => {
296 + expect(parsed.safeParse({ handle: ATTACK }).success).toBe(false);
297 + expect(parsed.safeParse({ tags: { [ATTACK]: 1 } }).success).toBe(false);
298 + });
299 + expect(parsed.safeParse({ handle: 'aa', tags: { aa: 1 } }).success).toBe(true);
300 + expect(parsed.safeParse({ tags: { aa: 'x' } }).success).toBe(false);
301 + });
302 +
303 + it('runs a backtracking propertyNames pattern in linear time', () => {
304 + const parsed = jsonSchemaToZod({
305 + type: 'object',
306 + propertyNames: { pattern: '^(a+)+$' },
307 + });
308 + finishesQuickly(() => expect(parsed.safeParse({ [ATTACK]: 1 }).success).toBe(false));
309 + expect(parsed.safeParse({ aa: 1 }).success).toBe(true);
310 + });
311 +
312 + it('reports the schema pattern, not its per-value spelling, in guard errors', () => {
313 + const parsed = jsonSchemaToZod({
314 + type: 'object',
315 + properties: { tags: { $ref: '#/$defs/tags' } },
316 + $defs: { tags: { type: 'object', patternProperties: { '^t_': { type: 'integer' } } } },
317 + });
318 + const result = parsed.safeParse({ tags: { t_1: 'x' } });
319 + expect(result.success).toBe(false);
320 + expect(JSON.stringify(result.error?.issues)).not.toContain('t_1$');
321 + });
322 +
323 + it('keeps enforcing a lookahead pattern that is safe to backtrack', () => {
324 + const parsed = jsonSchemaToZod({
325 + type: 'string',
326 + pattern: '^(?=.*[a-z])(?=.*\\d)[a-z\\d]{8,}$',
327 + });
328 + expect(parsed.safeParse('abcd1234').success).toBe(true);
329 + expect(parsed.safeParse('abcdefgh').success).toBe(false);
330 + // Past the input bound the fallback is skipped rather than risked.
331 + expect(parsed.safeParse('a'.repeat(MAX_BACKTRACKING_INPUT_LENGTH + 1)).success).toBe(true);
332 + });
333 +
334 + it('leaves a lookahead pattern that could backtrack catastrophically unenforced', () => {
335 + // Assembled at runtime so static analysis does not flag the test input
336 + // itself as an inefficient regular expression.
337 + const pattern = ['^(?=(a+)+', '$)a'].join('');
338 + const regex = compilePattern('pattern', pattern, { path: [] });
339 + expect(regex.matcher.kind).toBe('unenforced');
340 + finishesQuickly(() => expect(regex.test(ATTACK)).toBe(true));
341 + // The JSON Schema shown to the model still carries the pattern.
342 + expect(regex.source).toBe(pattern);
343 + });
344 +
345 + it('rejects a patternProperties key that needs a backtracking engine', () => {
346 + for (const schema of [
347 + { type: 'object', patternProperties: { '^(?!x)': { type: 'string' } } },
348 + {
349 + type: 'object',
350 + properties: { v: { $ref: '#/$defs/v' } },
351 + $defs: { v: { type: 'object', patternProperties: { '^(?!x)': { type: 'string' } } } },
352 + },
353 + ] as JsonSchema[]) {
354 + expect(() => jsonSchemaToZod(schema)).toThrow(InvalidPatternError);
355 + }
356 + try {
357 + compilePattern('patternProperties', '^(?!x)', { path: [] });
358 + } catch (error) {
359 + expect((error as InvalidPatternError).reason).toBe('unsupported');
360 + }
361 + });
362 +
363 + it('picks RE2 for every pattern it can express', () => {
364 + expect(createPatternMatcher('^(\\d+\\.)*\\d+$').kind).toBe('linear');
365 + expect(createPatternMatcher('^[a-z\\_]+$').kind).toBe('linear');
366 + expect(createPatternMatcher('^(?=.*[a-y])a').kind).toBe('backtracking');
367 + });
368 +});