feat(images): copyright-safe image validation, sourcing policy, and CI gate (#387)
Implements #329: copyright-safe image registry and sourcing policy - scripts/manage_image_registry.py: registry CRUD + validation - scripts/validate_content_images.py: CI gate for frontmatter image refs - docs/image-sourcing-policy.md: policy documentation - .github/workflows/ci.yml: registry + content image validation steps - Explicit registry file existence check (prevents silent pass on deletion) - Path normalization for Hugo asset-relative vs on-disk paths Closes #329 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
Jun 12, 2026 at 00:41 UTC
89f1ec00e2fcf08e9fd50ab9f5ba982d081416f6
7 files changed
+459
-5
.github/workflows/ci.yml
+7
-1
@@ -42,4 +42,10 @@ jobs:
42
run: python -m pip_audit -r requirements.txt
43
44
- name: Run Python tests
45
- run: python -m pytest
\ No newline at end of file
45
+ run: python -m pytest
46
+
47
+ - name: Validate image registry
48
+ run: python scripts/manage_image_registry.py validate
49
+
50
+ - name: Validate content images (no hotlinks/secrets)
51
+ run: python scripts/validate_content_images.py
\ No newline at end of file
data/image-registry.schema.json
+13
@@ -51,6 +51,19 @@
51
"width": { "type": "integer" },
52
"height": { "type": "integer" }
53
}
54
+ },
55
+ "checksum": {
56
+ "type": "string",
57
+ "description": "SHA-256 hash of the image file for integrity verification"
58
+ },
59
+ "review_status": {
60
+ "type": "string",
61
+ "enum": ["pending", "approved", "rejected"],
62
+ "description": "Policy review status (Hermes approval)"
63
+ },
64
+ "alt_text": {
65
+ "type": "string",
66
+ "description": "Accessible alt text describing the image content"
67
}
68
}
69
}
docs/image-sourcing-policy.md
new
+84
@@ -0,0 +1,84 @@
1
+# Image Sourcing Policy
2
+
3
+This document defines the copyright-safe image sourcing policy for Claracle (SquadScope).
4
+
5
+## Core Principle
6
+
7
+**Fair use is not an automated image policy.** All images used in generated content must be explicitly licensed, locally hosted, and registry-tracked.
8
+
9
+## Image Preference Order
10
+
11
+1. **Generated data visuals (preferred):** Mermaid diagrams, SVG charts, repo trend cards, signal/noise summaries, star/topic visualizations. These are original works created by the pipeline.
12
+2. **CC0 / Openverse images:** Only when downloaded, locally hosted, resized to required dimensions, attributed where the source requests it, and recorded in `data/image-registry.json`.
13
+3. **Local assets:** Original artwork or icons created for the project, registered as `local-asset` license type.
14
+
15
+## Prohibited Practices
16
+
17
+- **No hotlinking:** Never reference external image URLs in published content. All images must be served from the local repository/deploy.
18
+- **No `og:image` reuse:** Do not scrape or reuse `og:image` meta tags from external articles. An Open Graph tag is not a reuse license.
19
+- **No fair-use automation:** Do not rely on fair use as a justification for automated image sourcing. Fair use requires case-by-case human judgment.
20
+- **No unattributed use:** CC0 images do not legally require attribution, but the registry should still record the source for provenance.
21
+- **No secrets in URLs:** Image paths must never contain SAS tokens, API keys, credentials, or tracking parameters.
22
+
23
+## GitHub OG Previews
24
+
25
+GitHub auto-generated Open Graph preview images may be used **only when**:
26
+- They are generated/controlled by this repo or repo owner
27
+- They are downloaded and stored locally
28
+- Provenance is recorded in the image registry
29
+- They are NOT scraped from third-party repositories
30
+
31
+## Image Registry
32
+
33
+All non-generated cover images are tracked in `data/image-registry.json`:
34
+- **Schema:** `data/image-registry.schema.json`
35
+- **Management:** `scripts/manage_image_registry.py` (add, validate, list)
36
+- **Validation:** `scripts/validate_content_images.py` (CI gate)
37
+
38
+### Required Registry Fields
39
+
40
+| Field | Description |
41
+|-------|-------------|
42
+| `filename` | Relative path from repo root |
43
+| `license` | One of: `CC0`, `Openverse`, `local-asset` |
44
+| `added_by` | Person or automation identifier |
45
+
46
+### Recommended Fields
47
+
48
+| Field | Description |
49
+|-------|-------------|
50
+| `source_url` | Original source for provenance |
51
+| `attribution` | Attribution text |
52
+| `added_at` | ISO date added |
53
+| `used_in` | Content paths using this image |
54
+| `dimensions` | Width/height object |
55
+
56
+## Validation Gates
57
+
58
+CI enforces:
59
+1. `scripts/manage_image_registry.py validate` — registry schema integrity
60
+2. `scripts/validate_content_images.py` — no hotlinks, no secrets, no unregistered covers in published content
61
+
62
+## Hugo Configuration
63
+
64
+Hugo Goldmark must remain configured with `unsafe = false`. Image rendering relies on local paths and Hugo's resource pipeline, not raw HTML injection.
65
+
66
+## Workflow
67
+
68
+1. Pipeline generates data visuals (Mermaid/SVG) → used directly, no registry needed
69
+2. If a non-generated cover is needed:
70
+ a. Find CC0/Openverse source
71
+ b. Download to `assets/covers/`
72
+ c. Resize to required dimensions
73
+ d. Register via `scripts/manage_image_registry.py add`
74
+ e. Reference in frontmatter using the Hugo asset-relative path (e.g., `covers/foo.webp`)
75
+ — Hugo resolves this via `resources.Get`; the on-disk file lives at `assets/covers/foo.webp`
76
+3. CI validates on every PR (registry must exist; path normalization handles both forms)
77
+
78
+## Ownership
79
+
80
+- **Bender:** Pipeline metadata, registry schema, download/resize automation
81
+- **Hermes:** Copyright/privacy/security policy review
82
+- **Calculon:** Visual/cover design decisions
83
+- **Amy:** Hugo/frontmatter consumption of registered images
84
+- **Fry:** Validation tests and CI gates
scripts/manage_image_registry.py
+7
-2
@@ -33,8 +33,10 @@ class RegistryError(ValueError):
33
"""Raised when the image registry cannot be loaded safely."""
34
35
36
-def load_registry(path: Path = REGISTRY_PATH) -> dict:
36
+def load_registry(path: Path = REGISTRY_PATH, *, allow_missing: bool = True) -> dict:
37
if not path.exists():
38
+ if not allow_missing:
39
+ raise RegistryError(f"Image registry not found: {path}")
40
return {"images": []}
41
try:
42
registry = json.loads(path.read_text(encoding="utf-8"))
@@ -103,7 +105,10 @@ def add_image(args: argparse.Namespace) -> int:
105
106
107
def validate_registry(args: argparse.Namespace) -> int:
106
- registry = load_registry()
108
+ if not REGISTRY_PATH.exists():
109
+ print(f"FAIL: Image registry file not found: {REGISTRY_PATH}", file=sys.stderr)
110
+ return 1
111
+ registry = load_registry(allow_missing=False)
112
errors: list[str] = []
113
114
for i, img in enumerate(registry["images"]):
scripts/validate_content_images.py
new
+177
@@ -0,0 +1,177 @@
1
+#!/usr/bin/env python3
2
+"""Validate that published content does not hotlink external images.
3
+
4
+Scans Markdown content files and frontmatter for:
5
+- Remote image URLs (http/https) in Markdown image syntax or HTML img tags
6
+- External og:image or cover_image values in YAML frontmatter
7
+- SAS tokens, credentials, or tracking parameters in image URLs
8
+- References to images not present in the image registry
9
+
10
+Exit code 0 = clean, 1 = violations found.
11
+"""
12
+
13
+from __future__ import annotations
14
+
15
+import json
16
+import re
17
+import sys
18
+from pathlib import Path
19
+
20
+CONTENT_DIR = Path("content")
21
+REGISTRY_PATH = Path("data/image-registry.json")
22
+
23
+# Patterns that indicate remote/external image references
24
+REMOTE_URL_PATTERN = re.compile(r"https?://[^\s\"'>)\]]+", re.IGNORECASE)
25
+MD_IMAGE_PATTERN = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
26
+HTML_IMG_PATTERN = re.compile(r"<img[^>]+src=[\"']([^\"']+)[\"']", re.IGNORECASE)
27
+
28
+# Frontmatter fields that reference images
29
+IMAGE_FRONTMATTER_FIELDS = ("cover_image", "og_image", "image", "thumbnail")
30
+
31
+# Patterns indicating secrets/credentials in URLs
32
+SECRET_PATTERNS = [
33
+ re.compile(r"[?&](?:sig|sv|se|sp|spr|srt|ss)=", re.IGNORECASE), # Azure SAS
34
+ re.compile(r"[?&](?:token|api_key|apikey|secret|password)=", re.IGNORECASE),
35
+ re.compile(r"[?&](?:utm_source|utm_medium|utm_campaign|fbclid|gclid)=", re.IGNORECASE), # Tracking
36
+]
37
+
38
+
39
+def _is_remote_url(value: str) -> bool:
40
+ return value.startswith(("http://", "https://", "//"))
41
+
42
+
43
+def _extract_frontmatter(text: str) -> dict[str, str]:
44
+ """Extract YAML frontmatter image fields (simple key: value parsing)."""
45
+ fields: dict[str, str] = {}
46
+ if not text.startswith("---"):
47
+ return fields
48
+ end = text.find("\n---", 3)
49
+ if end == -1:
50
+ return fields
51
+ fm_block = text[3:end]
52
+ for line in fm_block.splitlines():
53
+ for field in IMAGE_FRONTMATTER_FIELDS:
54
+ if line.strip().startswith(f"{field}:"):
55
+ value = line.split(":", 1)[1].strip().strip("\"'")
56
+ if value:
57
+ fields[field] = value
58
+ return fields
59
+
60
+
61
+def _check_secrets_in_url(url: str) -> list[str]:
62
+ """Check for credentials or tracking params in a URL."""
63
+ issues = []
64
+ for pat in SECRET_PATTERNS:
65
+ if pat.search(url):
66
+ issues.append(f"URL contains suspicious parameters: {url[:120]}")
67
+ break
68
+ return issues
69
+
70
+
71
+def validate_file(filepath: Path) -> list[str]:
72
+ """Validate a single content file. Returns list of violations."""
73
+ violations: list[str] = []
74
+ try:
75
+ text = filepath.read_text(encoding="utf-8")
76
+ except (OSError, UnicodeDecodeError):
77
+ return violations
78
+
79
+ # Check frontmatter image fields
80
+ fm_fields = _extract_frontmatter(text)
81
+ for field, value in fm_fields.items():
82
+ if _is_remote_url(value):
83
+ violations.append(f"{filepath}: frontmatter '{field}' hotlinks external URL: {value[:100]}")
84
+ violations.extend(
85
+ f"{filepath}: frontmatter '{field}' {issue}"
86
+ for issue in _check_secrets_in_url(value)
87
+ )
88
+
89
+ # Check Markdown image syntax 
90
+ for match in MD_IMAGE_PATTERN.finditer(text):
91
+ url = match.group(2).strip()
92
+ if _is_remote_url(url):
93
+ violations.append(f"{filepath}: Markdown image hotlinks external URL: {url[:100]}")
94
+ violations.extend(f"{filepath}: {issue}" for issue in _check_secrets_in_url(url))
95
+
96
+ # Check HTML <img src="...">
97
+ for match in HTML_IMG_PATTERN.finditer(text):
98
+ url = match.group(1).strip()
99
+ if _is_remote_url(url):
100
+ violations.append(f"{filepath}: HTML img hotlinks external URL: {url[:100]}")
101
+ violations.extend(f"{filepath}: {issue}" for issue in _check_secrets_in_url(url))
102
+
103
+ return violations
104
+
105
+
106
+def validate_content(content_dir: Path = CONTENT_DIR) -> list[str]:
107
+ """Scan all .md files under content_dir for image policy violations."""
108
+ all_violations: list[str] = []
109
+ if not content_dir.exists():
110
+ return all_violations
111
+ for md_file in sorted(content_dir.rglob("*.md")):
112
+ all_violations.extend(validate_file(md_file))
113
+ return all_violations
114
+
115
+
116
+def validate_registry_references(
117
+ content_dir: Path = CONTENT_DIR,
118
+ registry_path: Path = REGISTRY_PATH,
119
+) -> list[str]:
120
+ """Verify local image references in frontmatter exist in the registry."""
121
+ violations: list[str] = []
122
+ if not registry_path.exists():
123
+ violations.append(f"Image registry file not found: {registry_path}")
124
+ return violations
125
+ try:
126
+ registry = json.loads(registry_path.read_text(encoding="utf-8"))
127
+ except (json.JSONDecodeError, ValueError):
128
+ violations.append(f"Cannot parse registry: {registry_path}")
129
+ return violations
130
+
131
+ # Build a normalized set for matching. Hugo resolves covers via resources.Get
132
+ # using asset-relative paths (e.g., "covers/foo.webp"), while on-disk files
133
+ # live under "assets/covers/". Register both forms for consistent matching.
134
+ registered_files: set[str] = set()
135
+ for img in registry.get("images", []):
136
+ filename = img.get("filename", "")
137
+ registered_files.add(filename)
138
+ if filename.startswith("assets/"):
139
+ registered_files.add(filename[len("assets/"):])
140
+ else:
141
+ registered_files.add(f"assets/{filename}")
142
+
143
+ if not content_dir.exists():
144
+ return violations
145
+
146
+ for md_file in sorted(content_dir.rglob("*.md")):
147
+ try:
148
+ text = md_file.read_text(encoding="utf-8")
149
+ except (OSError, UnicodeDecodeError):
150
+ continue
151
+ fm_fields = _extract_frontmatter(text)
152
+ for field, value in fm_fields.items():
153
+ if not _is_remote_url(value) and value and value not in registered_files:
154
+ # Only flag non-generated assets (covers/ prefix indicates registry-managed)
155
+ if value.startswith("covers/") or value.startswith("assets/covers/"):
156
+ violations.append(
157
+ f"{md_file}: frontmatter '{field}' references unregistered image: {value}"
158
+ )
159
+ return violations
160
+
161
+
162
+def main() -> int:
163
+ violations = validate_content()
164
+ violations.extend(validate_registry_references())
165
+
166
+ if violations:
167
+ print("Image policy violations found:", file=sys.stderr)
168
+ for v in violations:
169
+ print(f" FAIL: {v}", file=sys.stderr)
170
+ return 1
171
+
172
+ print("Content image validation passed: no hotlinks, secrets, or unregistered covers found.")
173
+ return 0
174
+
175
+
176
+if __name__ == "__main__":
177
+ raise SystemExit(main())
tests/test_manage_image_registry.py
+2
-2
@@ -24,8 +24,8 @@ def _run_with_registry(tmp_path: Path, images: list | None, argv: list[str]) ->
24
orig_load = registry_mod.load_registry
25
orig_save = registry_mod.save_registry
26
27
- def patched_load(path: Path = reg_path) -> dict:
28
- return orig_load(path)
27
+ def patched_load(path: Path = reg_path, **kwargs) -> dict:
28
+ return orig_load(path, **kwargs)
29
30
def patched_save(registry: dict, path: Path = reg_path) -> None:
31
return orig_save(registry, path)
tests/test_validate_content_images.py
new
+169
@@ -0,0 +1,169 @@
1
+"""Tests for scripts/validate_content_images.py."""
2
+
3
+from __future__ import annotations
4
+
5
+import json
6
+from pathlib import Path
7
+
8
+import scripts.validate_content_images as validator
9
+
10
+
11
+def _write_md(tmp_path: Path, filename: str, content: str) -> Path:
12
+ """Write a markdown file in a temp content dir."""
13
+ content_dir = tmp_path / "content"
14
+ content_dir.mkdir(exist_ok=True)
15
+ filepath = content_dir / filename
16
+ filepath.write_text(content, encoding="utf-8")
17
+ return content_dir
18
+
19
+
20
+class TestFrontmatterExtraction:
21
+ def test_extracts_cover_image(self) -> None:
22
+ text = '---\ntitle: "Test"\ncover_image: "covers/test.webp"\n---\nBody'
23
+ fields = validator._extract_frontmatter(text)
24
+ assert fields["cover_image"] == "covers/test.webp"
25
+
26
+ def test_extracts_og_image(self) -> None:
27
+ text = '---\nog_image: "covers/og.png"\n---\n'
28
+ fields = validator._extract_frontmatter(text)
29
+ assert fields["og_image"] == "covers/og.png"
30
+
31
+ def test_ignores_non_image_fields(self) -> None:
32
+ text = '---\ntitle: "Hello"\nauthor: "Test"\n---\n'
33
+ fields = validator._extract_frontmatter(text)
34
+ assert fields == {}
35
+
36
+ def test_handles_no_frontmatter(self) -> None:
37
+ text = "Just some content"
38
+ fields = validator._extract_frontmatter(text)
39
+ assert fields == {}
40
+
41
+
42
+class TestHotlinkDetection:
43
+ def test_detects_frontmatter_hotlink(self, tmp_path: Path) -> None:
44
+ content_dir = _write_md(
45
+ tmp_path, "test.md",
46
+ '---\ncover_image: "https://evil.com/img.png"\n---\nBody',
47
+ )
48
+ violations = validator.validate_content(content_dir)
49
+ assert any("hotlinks external URL" in v for v in violations)
50
+
51
+ def test_detects_markdown_image_hotlink(self, tmp_path: Path) -> None:
52
+ content_dir = _write_md(
53
+ tmp_path, "test.md",
54
+ "---\ntitle: test\n---\n\n",
55
+ )
56
+ violations = validator.validate_content(content_dir)
57
+ assert any("Markdown image hotlinks" in v for v in violations)
58
+
59
+ def test_detects_html_img_hotlink(self, tmp_path: Path) -> None:
60
+ content_dir = _write_md(
61
+ tmp_path, "test.md",
62
+ '---\ntitle: test\n---\n<img src="http://evil.com/x.png" alt="bad">\n',
63
+ )
64
+ violations = validator.validate_content(content_dir)
65
+ assert any("HTML img hotlinks" in v for v in violations)
66
+
67
+ def test_detects_protocol_relative_url(self, tmp_path: Path) -> None:
68
+ content_dir = _write_md(
69
+ tmp_path, "test.md",
70
+ '---\nog_image: "//cdn.example.com/image.png"\n---\n',
71
+ )
72
+ violations = validator.validate_content(content_dir)
73
+ assert any("hotlinks external URL" in v for v in violations)
74
+
75
+ def test_allows_local_paths(self, tmp_path: Path) -> None:
76
+ content_dir = _write_md(
77
+ tmp_path, "test.md",
78
+ '---\ncover_image: "covers/local.webp"\n---\n\n',
79
+ )
80
+ violations = validator.validate_content(content_dir)
81
+ assert len(violations) == 0
82
+
83
+
84
+class TestSecretDetection:
85
+ def test_detects_sas_token(self, tmp_path: Path) -> None:
86
+ content_dir = _write_md(
87
+ tmp_path, "test.md",
88
+ "---\ntitle: test\n---\n\n",
89
+ )
90
+ violations = validator.validate_content(content_dir)
91
+ assert any("suspicious parameters" in v for v in violations)
92
+
93
+ def test_detects_tracking_params(self, tmp_path: Path) -> None:
94
+ content_dir = _write_md(
95
+ tmp_path, "test.md",
96
+ "---\ntitle: test\n---\n\n",
97
+ )
98
+ violations = validator.validate_content(content_dir)
99
+ assert any("suspicious parameters" in v for v in violations)
100
+
101
+ def test_detects_api_key_param(self, tmp_path: Path) -> None:
102
+ content_dir = _write_md(
103
+ tmp_path, "test.md",
104
+ '---\ncover_image: "covers/x.webp?api_key=secret123"\n---\n',
105
+ )
106
+ violations = validator.validate_content(content_dir)
107
+ assert any("suspicious parameters" in v for v in violations)
108
+
109
+
110
+class TestRegistryValidation:
111
+ def test_flags_unregistered_cover(self, tmp_path: Path) -> None:
112
+ content_dir = _write_md(
113
+ tmp_path, "test.md",
114
+ '---\ncover_image: "covers/unregistered.webp"\n---\nBody',
115
+ )
116
+ reg_path = tmp_path / "registry.json"
117
+ reg_path.write_text(json.dumps({"images": []}), encoding="utf-8")
118
+ violations = validator.validate_registry_references(content_dir, reg_path)
119
+ assert any("unregistered image" in v for v in violations)
120
+
121
+ def test_passes_registered_cover(self, tmp_path: Path) -> None:
122
+ content_dir = _write_md(
123
+ tmp_path, "test.md",
124
+ '---\ncover_image: "covers/registered.webp"\n---\nBody',
125
+ )
126
+ reg_path = tmp_path / "registry.json"
127
+ reg_path.write_text(
128
+ json.dumps({"images": [{"filename": "covers/registered.webp", "license": "CC0", "added_by": "test"}]}),
129
+ encoding="utf-8",
130
+ )
131
+ violations = validator.validate_registry_references(content_dir, reg_path)
132
+ assert len(violations) == 0
133
+
134
+ def test_ignores_non_cover_local_paths(self, tmp_path: Path) -> None:
135
+ content_dir = _write_md(
136
+ tmp_path, "test.md",
137
+ '---\ncover_image: "images/generated-chart.svg"\n---\nBody',
138
+ )
139
+ reg_path = tmp_path / "registry.json"
140
+ reg_path.write_text(json.dumps({"images": []}), encoding="utf-8")
141
+ violations = validator.validate_registry_references(content_dir, reg_path)
142
+ # Non-cover paths (not starting with covers/ or assets/covers/) are not flagged
143
+ assert len(violations) == 0
144
+
145
+ def test_handles_missing_registry(self, tmp_path: Path) -> None:
146
+ content_dir = _write_md(
147
+ tmp_path, "test.md",
148
+ '---\ncover_image: "covers/x.webp"\n---\n',
149
+ )
150
+ violations = validator.validate_registry_references(content_dir, tmp_path / "nonexistent.json")
151
+ assert len(violations) == 1
152
+ assert "not found" in violations[0]
153
+
154
+
155
+class TestHelpers:
156
+ def test_is_remote_url_http(self) -> None:
157
+ assert validator._is_remote_url("http://example.com/img.png")
158
+
159
+ def test_is_remote_url_https(self) -> None:
160
+ assert validator._is_remote_url("https://example.com/img.png")
161
+
162
+ def test_is_remote_url_protocol_relative(self) -> None:
163
+ assert validator._is_remote_url("//cdn.example.com/img.png")
164
+
165
+ def test_is_not_remote_url_local(self) -> None:
166
+ assert not validator._is_remote_url("covers/local.webp")
167
+
168
+ def test_is_not_remote_url_relative(self) -> None:
169
+ assert not validator._is_remote_url("assets/images/chart.svg")