2026-05-25-second-sweep.md · cryptotaxi247/SquadScope-1

Code Commits Issues Pulls Sessions CI/CD

main

md 332 lines 19 KB

Rendered Raw

1	# Responsible AI — Second-Sweep Audit Report
2
3	Date: 2026-05-25
4	Auditor: Nibbler (Responsible AI / Safety Reviewer)
5	Scope: Current `main` branch after GA4 fork-safety, prompt-injection hardening, Cookie Consent v3, SEO/PWA/distribution work, Phase 5 dashboard, and design-system phases 1–6.
6	Out of scope: AI-content disclosure PR `squad/208-ai-content-disclosure`, home page restructure `squad/207`, and privacy-policy expansion `squad/206`.
7
8	---
9
10	## Summary
11
12	\| Severity \| Count \|
13	\|---\|---:\|
14	\| blocker \| 0 \|
15	\| high \| 2 \|
16	\| medium \| 5 \|
17	\| low \| 4 \|
18	\| info \| 5 \|
19
20	Overall: No blockers. The main new posture risk is not consent or prompt-injection anymore; those moved materially forward. The remaining high-risk gaps are editorial governance: SquadScope still lacks a reader-visible correction/report/retraction path, and its source methodology does not yet disclose or measure representation bias from star-ranked GitHub search plus a single English-language press source.
21
22	---
23
24	## Microsoft RAI Principle Sweep
25
26	### 1. Fairness
27
28	#### Finding F1 — Source selection can systematically over-represent high-star, English, and BigCo/press-visible projects
29
30	Severity: high
31	Principle: Fairness, Transparency
32	Artifacts: `scripts/crawl.py`, `scripts/techcrunch_crawler.py`, `examples/topics/ai-ml.yml`, `examples/topics/rust.yml`, `README.md`
33
34	Finding:
35	The crawler's default discovery path queries GitHub Search with `stars:>50`, sorts by `stars`, and caps results (`scripts/crawl.py`, search call and default queries). Topic configs repeat the same high-star threshold pattern, with AI/ML additionally boosting Python and Jupyter Notebook. The press context is currently TechCrunch-only (`scripts/techcrunch_crawler.py` uses `https://techcrunch.com/feed/`). README explains that the system crawls GitHub and applies heuristic filtering, but the public site does not disclose what those cutoffs mean for representation.
36
37	This creates predictable editorial skew:
38	- lower-star but regionally important projects are less likely to enter the candidate set;
39	- non-English descriptions and non-US ecosystems may be underweighted by English keyword/entity heuristics;
40	- vendor repos with established brands and press adjacency are more likely to look like durable signal;
41	- TechCrunch as the only press source overweights US startup/funding narratives.
42
43	Positive evidence:
44	The prompts explicitly require blind-spot analysis and warn not to mistake popularity for momentum. W22 also names coordinated star-farming as noise, showing the analyst is not blindly accepting star counts.
45
46	Recommendation:
47	Add a reader-facing methodology page and a machine-readable source-mix report that disclose: GitHub query filters, star thresholds, sorting, date windows, press sources, known blind spots, and weekly distribution by language/topic/org type. Add a fairness check to the analysis gate: each weekly report should either show a minimum diversity sanity check or explicitly state when the source pool is too skewed to generalize.
48
49	Issue: [#208](https://github.com/jmservera/SquadScope/issues/208) (`squad`, `rai`).
50
51	---
52
53	### 2. Reliability & Safety
54
55	#### Finding R1 — No public errata, report-this-article, or retraction process exists
56
57	Severity: high
58	Principle: Reliability & Safety, Accountability
59	Artifacts: `content/weekly/2026/W21.md`, `content/weekly/2026/W22.md`, `layouts/weekly/single.html`, `layouts/partials/article-footer.html`, `content/privacy/_index.md`
60
61	Finding:
62	When the AI analysis is wrong, a reader has no article-level recourse. Weekly pages render metadata, article content, taxonomy links, RSS, sharing, and last-updated text, but no "Report an issue with this article" link. The privacy page links GitHub issues for privacy rights, but that path is not present in the global footer or article footer as a content-correction route. The published W21/W22 articles contain no visible correction note pattern, errata section, or retraction convention.
63
64	This matters because SquadScope makes factual claims about named repositories, star counts, spam/star-farming patterns, and vendor behavior. Even with the quality gate, a hallucinated or outdated claim needs an explicit correction pathway.
65
66	Recommendation:
67	Add an article-footer link such as `Report an issue with this article` prefilled with the article URL, plus a short public corrections policy. Define statuses for correction, clarification, and retraction; require corrected articles to include a dated note rather than silent rewrites.
68
69	Issue: [#209](https://github.com/jmservera/SquadScope/issues/209) (`squad`, `rai`).
70
71	---
72
73	### 3. Privacy & Security
74
75	#### Finding P1 — No site-level Referrer-Policy is configured
76
77	Severity: medium
78	Principle: Privacy & Security
79	Artifacts: `layouts/partials/head.html`, `content/privacy/_index.md`
80
81	Finding:
82	The site links to GitHub, TechCrunch, Google Fonts, Google Analytics, Hugo, RSS, and share-mailto flows, but no `Referrer-Policy` meta tag or deploy header is present. Without a policy, external destinations can receive the full page URL as referrer depending on browser defaults and link context. Weekly article URLs can encode which report a reader was viewing.
83
84	Recommendation:
85	Set `Referrer-Policy: strict-origin-when-cross-origin` at the hosting layer if possible, and add `<meta name="referrer" content="strict-origin-when-cross-origin">` as a static-site fallback. If the project wants stronger privacy, use `same-origin`.
86
87	---
88
89	#### Finding P2 — Vendored CookieConsent assets are same-origin but not integrity-pinned in markup
90
91	Severity: low
92	Principle: Privacy & Security, Reliability & Safety
93	Artifacts: `layouts/partials/cookie-consent.html`, `static/vendor/cookieconsent/`
94
95	Finding:
96	CookieConsent v3 assets are vendored locally, which is better than loading a consent library from a third-party CDN. However, the script and stylesheet are loaded without an integrity attribute or documented checksum in the page markup. This is lower risk because the assets are same-origin, but the consent library is a privacy-control dependency; accidental drift should be detectable.
97
98	Recommendation:
99	Keep vendoring, but record version and SHA-256 hashes in the decision note and/or render Subresource Integrity attributes for the local script and stylesheet when practical.
100
101	---
102
103	#### Finding P3 — GA4 consent/fork-safety posture is materially improved
104
105	Severity: info
106	Principle: Privacy & Security
107	Artifacts: `hugo.toml`, `.github/workflows/deploy-site.yml`, `layouts/partials/analytics.html`, `data/cookieconsent.json`
108
109	Finding:
110	The GA measurement ID defaults to empty in Hugo config, deploy injects it from `secrets.GA_MEASUREMENT_ID`, analytics scripts are `type="text/plain"` with CookieConsent analytics categories, and CookieConsent is opt-in. The current `gtag('config', ...)` partial does not override GA4's default IP-anonymization behavior.
111
112	Recommendation:
113	No new gap beyond P1/P2. Keep the first-visit no-consent network test in the release checklist.
114
115	---
116
117	#### Finding P4 — No comment system or contact form found
118
119	Severity: info
120	Principle: Privacy & Security
121	Artifacts: `layouts/_default/single.html`, content tree, layout search
122
123	Finding:
124	No deployed comment provider or contact form was found. This is good: there is no extra visitor-submitted personal-data flow beyond analytics consent, Google Fonts requests, GitHub Pages hosting, outbound links, and RSS.
125
126	Recommendation:
127	If comments or forms are added later, require a fresh privacy/RAI review before merge.
128
129	---
130
131	### 4. Inclusiveness (WCAG 2.2 AA quick sweep)
132
133	#### Finding A1 — Small status badges fail AA contrast in light mode
134
135	Severity: medium
136	Principle: Inclusiveness
137	Artifacts: `assets/css/badges.css`, `layouts/partials/correlation-badge.html`, `assets/css/tokens.css`
138
139	Finding:
140	The `badge-organic` small-text color contrast is about 3.90:1 on its mixed light background, below the WCAG AA 4.5:1 threshold for normal text. `badge-hype` is approximately 4.48:1, effectively at/just below the threshold. These badges are `0.75rem` text, so they need normal-text contrast, not large-text contrast.
141
142	Recommendation:
143	Darken the success/danger badge foregrounds for light mode or use a higher-contrast background/foreground pairing. Re-run contrast checks for all badge variants in light and dark themes.
144
145	---
146
147	#### Finding A2 — Reduced-motion support exists for smooth scrolling, but CSS transitions are not globally disabled
148
149	Severity: low
150	Principle: Inclusiveness
151	Artifacts: `layouts/partials/footer.html`, `assets/css/common/footer.css`
152
153	Finding:
154	Anchor smooth scrolling respects `prefers-reduced-motion: reduce`, which is good. The floating top-link still has `transition: visibility .3s, opacity .3s ...` with no reduced-motion override. This is minor, but it shows reduced-motion coverage is component-by-component rather than systemic.
155
156	Recommendation:
157	Add a global reduced-motion rule that disables non-essential transitions/animations, or at least override `.top-link` transition under `@media (prefers-reduced-motion: reduce)`.
158
159	---
160
161	#### Finding A3 — Heading order, keyboard labels, and `lang` pass the quick source sweep
162
163	Severity: info
164	Principle: Inclusiveness
165	Artifacts: `layouts/index.html`, `layouts/weekly/single.html`, `layouts/partials/header.html`, `layouts/_default/baseof.html`, `layouts/partials/footer.html`
166
167	Finding:
168	Key templates use one H1 followed by H2/H3 structures; the GitHub icon button has an `aria-label`; the theme toggle and mobile menu summary have labels; the skip link targets `#main-content`; the footer Manage Cookies control is a native button; and `<html lang="{{ site.Language }}">` is set.
169
170	Recommendation:
171	No gap found in the source sweep. Once the Hugo build baseline is fixed, run rendered-page keyboard and screen-reader smoke tests on home, a weekly article, privacy, and dashboard.
172
173	---
174
175	### 5. Transparency
176
177	#### Finding T1 — Source-data methodology is not reader-facing
178
179	Severity: medium
180	Principle: Transparency, Fairness
181	Artifacts: `README.md`, `docs/analysis-spec.md`, `scripts/crawl.py`, `content/about/_index.md`
182
183	Finding:
184	Operator-facing docs explain the analyzer contract and README describes the pipeline. A normal reader on the public site sees an About page and cost dashboard, but not a methodology page that explains how repos are selected, what "trending" means, what cutoff dates apply, why TechCrunch is used for press context, or why star deltas may be missing.
185
186	Recommendation:
187	Publish a short `/methodology/` page linked from article footers and About. It should be plain-language, not an implementation spec: inputs, cutoffs, ranking, limitations, correction path, and source-bias caveats.
188
189	---
190
191	#### Finding T2 — Prompts are public and tracked
192
193	Severity: info
194	Principle: Transparency
195	Artifacts: `prompts/analyze-weekly.md`, `prompts/analyze-topic.md`, `prompts/reskill.md`, `.gitignore`
196
197	Finding:
198	The prompts directory is tracked and not ignored. This is a transparency strength: readers and contributors can inspect the instructions that shape the editorial output.
199
200	Recommendation:
201	Link to the prompts from the future methodology page so transparency is discoverable without browsing the repository tree.
202
203	---
204
205	#### Finding T3 — Model naming is mixed: disclosure can stay generic, cost dashboard is specific
206
207	Severity: low
208	Principle: Transparency
209	Artifacts: `content/about/_index.md`, `layouts/partials/cost-dashboard.html`, `data/metrics/cost-summary.json`
210
211	Finding:
212	The About page uses generic language ("automated AI analysis pipeline"), while the cost dashboard renders model names from the metrics ledger. Specific model names are useful for cost accountability, but they should not be presented as quality endorsements or marketing claims.
213
214	Recommendation:
215	Keep article-level AI disclosure generic, and label the dashboard's model field as operational metadata only.
216
217	---
218
219	### 6. Accountability
220
221	#### Finding C1 — No site-wide content contact path in footer
222
223	Severity: medium
224	Principle: Accountability, Reliability & Safety
225	Artifacts: `layouts/partials/footer.html`, `content/privacy/_index.md`
226
227	Finding:
228	The privacy page has a GitHub issues contact path, but the global footer only links Privacy, Manage cookies, GitHub, RSS, and Archive. A reader who finds an editorial error on an article page has to infer that GitHub is the right place to report it.
229
230	Recommendation:
231	Add a footer `Contact` or `Report an issue` link to GitHub Issues, and add article-specific links with page context. This should be implemented with Finding R1.
232
233	---
234
235	#### Finding C2 — No content license or reuse terms found
236
237	Severity: medium
238	Principle: Accountability, Transparency
239	Artifacts: repository root, `README.md`, `content/weekly/`, `content/about/_index.md`
240
241	Finding:
242	No `LICENSE`, `COPYING`, terms page, or content reuse statement was found. The site publishes AI-assisted editorial text and repository summaries, but contributors and readers do not know whether the code, generated content, raw data, and analysis are reusable under the same terms.
243
244	Recommendation:
245	Add an explicit repository license and content license/reuse statement. If code and content use different licenses, state that clearly in README and the public site footer.
246
247	---
248
249	#### Finding C3 — Maintainer ownership is partially named, but editorial ownership is still vague
250
251	Severity: low
252	Principle: Accountability
253	Artifacts: `content/privacy/_index.md`, `README.md`, `content/about/_index.md`
254
255	Finding:
256	The privacy page names `jmservera` as data controller and site operator. That is enough for privacy ownership, but editorial ownership is still framed mostly as an automated pipeline rather than a person or project maintainer accountable for corrections.
257
258	Recommendation:
259	On About or Methodology, state that jmservera is the maintainer/operator and that corrections should be filed through GitHub Issues.
260
261	---
262
263	## EU AI Act transparency checks
264
265	### Article 50 / Article 52 — AI-generated or manipulated content must be distinguishable at delivery time
266
267	Status: Covered by in-flight AI-content disclosure PR (`squad/208-ai-content-disclosure`) per scope instruction; not audited here.
268
269	Residual recommendation:
270	The disclosure must be per article and visible at delivery time, not only in a policy page. It should link to the methodology/corrections pages once those exist.
271
272	### Foundation model labeling
273
274	Status: In-flight disclosure is expected to cover this. Cost dashboard already records model metadata for operations, but article disclosure should avoid implying model endorsement.
275
276	---
277
278	## Cross-cutting issues
279
280	1. Governance is now the main RAI gap. Cookie consent, GA4 fork-safety, and prompt-injection hardening moved the technical baseline forward. The remaining risk is what happens after publication: who corrects, retracts, or explains errors.
281	2. Methodology needs to move from repo docs to reader docs. `README.md` and `docs/analysis-spec.md` are useful to operators, but public readers need a shorter explanation of source selection and limitations.
282	3. Representation risk is measurable. The pipeline already has enough structured data (`language`, `owner`, `topics`, `stars`, `forks`, `created_at`) to produce weekly source-mix telemetry. This should become a standard transparency artifact.
283	4. A11y regressions are small but concrete. The new design system mostly passes the quick source sweep, but badge contrast is below AA and reduced-motion coverage should be centralized.
284	5. Privacy posture is mostly improved. No comment/form surface was found; GA4 is consent-gated and fork-safe. The missing browser privacy control is Referrer-Policy.
285
286	---
287
288	## Comparison with First Sweep
289
290	\| First-sweep finding \| Status in second sweep \| Evidence / note \|
291	\|---\|---\|---\|
292	\| Old SS-like icon failure mode \| Closed / institutionalized \| Current robot/binoculars icon remains distinct; icon-safety skill exists. No new hate-symbol resemblance found in source sweep. \|
293	\| Cookie consent visual-weight parity \| Closed \| CookieConsent config sets `equalWeightButtons: true`; footer Manage Cookies exists. \|
294	\| GA4 fork-safety \| Closed \| Measurement ID defaults empty and deploy injects from secret; analytics partial emits nothing without ID. \|
295	\| GA4 consent gate \| Closed with watch item \| Scripts are `type="text/plain"` and category-gated by CookieConsent. Keep first-visit network test. \|
296	\| Distribution copy review \| Mostly closed \| Distribution strategy exists; no new manipulative copy found in current public site. Continue Nibbler review for launch/community posts. \|
297	\| Prompt-injection untrusted repo descriptions \| Mostly closed \| Prompts now wrap raw JSON in `<untrusted-content>` and sanitizer handles suspicious description patterns. Remaining watch item: post-generation external URL validation. \|
298	\| Article AI authorship disclosure \| In progress outside scope \| Amy-11 PR covers this; not audited here. \|
299	\| Header GitHub icon ARIA \| Closed \| Header GitHub link has `aria-label="GitHub"`; theme/menu controls also labeled. \|
300	\| `prefers-reduced-motion` absent \| Partially closed \| Smooth scrolling respects reduced motion, but top-link transition remains unguarded. \|
301	\| Reskill prompt human-review gap \| Open / not re-audited deeply \| No evidence of a new human checkpoint in this sweep; lower priority than public correction/methodology gaps. \|
302
303	### New findings in this sweep
304
305	- High: Source methodology/fairness disclosure gap.
306	- High: No article-level errata/report/retraction process.
307	- Medium: No Referrer-Policy.
308	- Medium: Badge contrast below WCAG AA.
309	- Medium: No site-wide content contact path.
310	- Medium: No content license/reuse terms.
311	- Low: CookieConsent vendored asset integrity not pinned in markup.
312	- Low: Reduced-motion coverage not global.
313	- Low: Model naming should stay operational, not promotional.
314	- Low: Maintainer/editorial accountability copy remains vague.
315
316	---
317
318	## Follow-Up Issues to Open
319
320	- Issue A: [#208](https://github.com/jmservera/SquadScope/issues/208) `rai: disclose and measure source-selection bias in methodology` — high / Fairness + Transparency.
321	- Issue B: [#209](https://github.com/jmservera/SquadScope/issues/209) `rai: add article errata/report/retraction path` — high / Reliability & Safety + Accountability.
322
323	---
324
325	## Validation notes
326
327	- Reviewed `scripts/analyze_fallback.py`, `scripts/sanitize_repo_content.py`, prompts, topic configs, workflows, privacy/about pages, key layouts, and design CSS.
328	- Ran a local contrast calculation for design tokens and badges.
329	- Attempted `hugo --minify`; build failed on the current baseline before any audit-doc change because `layouts/partials/head.html` expects missing PaperMod CSS resources (`css/core/zmedia.css`, `css/core/license.css`, `css/includes/*`). This audit is documentation-only; no runtime code was changed.
330
331	Nibbler — Responsible AI / Safety Reviewer
332	2026-05-25 \| Second sweep — posture: improved technical controls, open governance gaps