179d667 · cryptotaxi247/SquadScope-1

@@ -0,0 +1,332 @@

1 +# Responsible AI — Second-Sweep Audit Report

2 +

3 +**Date:** 2026-05-25

4 +**Auditor:** Nibbler (Responsible AI / Safety Reviewer)

5 +**Scope:** Current `main` branch after GA4 fork-safety, prompt-injection hardening, Cookie Consent v3, SEO/PWA/distribution work, Phase 5 dashboard, and design-system phases 1–6.

6 +**Out of scope:** AI-content disclosure PR `squad/208-ai-content-disclosure`, home page restructure `squad/207`, and privacy-policy expansion `squad/206`.

7 +

8 +---

9 +

10 +## Summary

11 +

12 +| Severity | Count |

13 +|---|---:|

14 +| blocker | 0 |

15 +| high | 2 |

16 +| medium | 5 |

17 +| low | 4 |

18 +| info | 5 |

19 +

20 +**Overall:** No blockers. The main new posture risk is not consent or prompt-injection anymore; those moved materially forward. The remaining high-risk gaps are editorial governance: SquadScope still lacks a reader-visible correction/report/retraction path, and its source methodology does not yet disclose or measure representation bias from star-ranked GitHub search plus a single English-language press source.

21 +

22 +---

23 +

24 +## Microsoft RAI Principle Sweep

25 +

26 +### 1. Fairness

27 +

28 +#### Finding F1 — Source selection can systematically over-represent high-star, English, and BigCo/press-visible projects

29 +

30 +**Severity:** high

31 +**Principle:** Fairness, Transparency

32 +**Artifacts:** `scripts/crawl.py`, `scripts/techcrunch_crawler.py`, `examples/topics/ai-ml.yml`, `examples/topics/rust.yml`, `README.md`

33 +

34 +**Finding:**

35 +The crawler's default discovery path queries GitHub Search with `stars:>50`, sorts by `stars`, and caps results (`scripts/crawl.py`, search call and default queries). Topic configs repeat the same high-star threshold pattern, with AI/ML additionally boosting Python and Jupyter Notebook. The press context is currently TechCrunch-only (`scripts/techcrunch_crawler.py` uses `https://techcrunch.com/feed/`). README explains that the system crawls GitHub and applies heuristic filtering, but the public site does not disclose what those cutoffs mean for representation.

36 +

37 +This creates predictable editorial skew:

38 +- lower-star but regionally important projects are less likely to enter the candidate set;

39 +- non-English descriptions and non-US ecosystems may be underweighted by English keyword/entity heuristics;

40 +- vendor repos with established brands and press adjacency are more likely to look like durable signal;

41 +- TechCrunch as the only press source overweights US startup/funding narratives.

42 +

43 +**Positive evidence:**

44 +The prompts explicitly require blind-spot analysis and warn not to mistake popularity for momentum. W22 also names coordinated star-farming as noise, showing the analyst is not blindly accepting star counts.

45 +

46 +**Recommendation:**

47 +Add a reader-facing methodology page and a machine-readable source-mix report that disclose: GitHub query filters, star thresholds, sorting, date windows, press sources, known blind spots, and weekly distribution by language/topic/org type. Add a fairness check to the analysis gate: each weekly report should either show a minimum diversity sanity check or explicitly state when the source pool is too skewed to generalize.

48 +

49 +**Issue:** [#208](https://github.com/jmservera/SquadScope/issues/208) (`squad`, `rai`).

50 +

51 +---

52 +

53 +### 2. Reliability & Safety

54 +

55 +#### Finding R1 — No public errata, report-this-article, or retraction process exists

56 +

57 +**Severity:** high

58 +**Principle:** Reliability & Safety, Accountability

59 +**Artifacts:** `content/weekly/2026/W21.md`, `content/weekly/2026/W22.md`, `layouts/weekly/single.html`, `layouts/partials/article-footer.html`, `content/privacy/_index.md`

60 +

61 +**Finding:**

62 +When the AI analysis is wrong, a reader has no article-level recourse. Weekly pages render metadata, article content, taxonomy links, RSS, sharing, and last-updated text, but no "Report an issue with this article" link. The privacy page links GitHub issues for privacy rights, but that path is not present in the global footer or article footer as a content-correction route. The published W21/W22 articles contain no visible correction note pattern, errata section, or retraction convention.

63 +

64 +This matters because SquadScope makes factual claims about named repositories, star counts, spam/star-farming patterns, and vendor behavior. Even with the quality gate, a hallucinated or outdated claim needs an explicit correction pathway.

65 +

66 +**Recommendation:**

67 +Add an article-footer link such as `Report an issue with this article` prefilled with the article URL, plus a short public corrections policy. Define statuses for correction, clarification, and retraction; require corrected articles to include a dated note rather than silent rewrites.

68 +

69 +**Issue:** [#209](https://github.com/jmservera/SquadScope/issues/209) (`squad`, `rai`).

70 +

71 +---

72 +

73 +### 3. Privacy & Security

74 +

75 +#### Finding P1 — No site-level Referrer-Policy is configured

76 +

77 +**Severity:** medium

78 +**Principle:** Privacy & Security

79 +**Artifacts:** `layouts/partials/head.html`, `content/privacy/_index.md`

80 +

81 +**Finding:**

82 +The site links to GitHub, TechCrunch, Google Fonts, Google Analytics, Hugo, RSS, and share-mailto flows, but no `Referrer-Policy` meta tag or deploy header is present. Without a policy, external destinations can receive the full page URL as referrer depending on browser defaults and link context. Weekly article URLs can encode which report a reader was viewing.

83 +

84 +**Recommendation:**

85 +Set `Referrer-Policy: strict-origin-when-cross-origin` at the hosting layer if possible, and add `<meta name="referrer" content="strict-origin-when-cross-origin">` as a static-site fallback. If the project wants stronger privacy, use `same-origin`.

86 +

87 +---

88 +

89 +#### Finding P2 — Vendored CookieConsent assets are same-origin but not integrity-pinned in markup

90 +

91 +**Severity:** low

92 +**Principle:** Privacy & Security, Reliability & Safety

93 +**Artifacts:** `layouts/partials/cookie-consent.html`, `static/vendor/cookieconsent/`

94 +

95 +**Finding:**

96 +CookieConsent v3 assets are vendored locally, which is better than loading a consent library from a third-party CDN. However, the script and stylesheet are loaded without an integrity attribute or documented checksum in the page markup. This is lower risk because the assets are same-origin, but the consent library is a privacy-control dependency; accidental drift should be detectable.

97 +

98 +**Recommendation:**

99 +Keep vendoring, but record version and SHA-256 hashes in the decision note and/or render Subresource Integrity attributes for the local script and stylesheet when practical.

100 +

101 +---

102 +

103 +#### Finding P3 — GA4 consent/fork-safety posture is materially improved

104 +

105 +**Severity:** info

106 +**Principle:** Privacy & Security

107 +**Artifacts:** `hugo.toml`, `.github/workflows/deploy-site.yml`, `layouts/partials/analytics.html`, `data/cookieconsent.json`

108 +

109 +**Finding:**

110 +The GA measurement ID defaults to empty in Hugo config, deploy injects it from `secrets.GA_MEASUREMENT_ID`, analytics scripts are `type="text/plain"` with CookieConsent analytics categories, and CookieConsent is opt-in. The current `gtag('config', ...)` partial does not override GA4's default IP-anonymization behavior.

111 +

112 +**Recommendation:**

113 +No new gap beyond P1/P2. Keep the first-visit no-consent network test in the release checklist.

114 +

115 +---

116 +

117 +#### Finding P4 — No comment system or contact form found

118 +

119 +**Severity:** info

120 +**Principle:** Privacy & Security

121 +**Artifacts:** `layouts/_default/single.html`, content tree, layout search

122 +

123 +**Finding:**

124 +No deployed comment provider or contact form was found. This is good: there is no extra visitor-submitted personal-data flow beyond analytics consent, Google Fonts requests, GitHub Pages hosting, outbound links, and RSS.

125 +

126 +**Recommendation:**

127 +If comments or forms are added later, require a fresh privacy/RAI review before merge.

128 +

129 +---

130 +

131 +### 4. Inclusiveness (WCAG 2.2 AA quick sweep)

132 +

133 +#### Finding A1 — Small status badges fail AA contrast in light mode

134 +

135 +**Severity:** medium

136 +**Principle:** Inclusiveness

137 +**Artifacts:** `assets/css/badges.css`, `layouts/partials/correlation-badge.html`, `assets/css/tokens.css`

138 +

139 +**Finding:**

140 +The `badge-organic` small-text color contrast is about **3.90:1** on its mixed light background, below the WCAG AA 4.5:1 threshold for normal text. `badge-hype` is approximately **4.48:1**, effectively at/just below the threshold. These badges are `0.75rem` text, so they need normal-text contrast, not large-text contrast.

141 +

142 +**Recommendation:**

143 +Darken the success/danger badge foregrounds for light mode or use a higher-contrast background/foreground pairing. Re-run contrast checks for all badge variants in light and dark themes.

144 +

145 +---

146 +

147 +#### Finding A2 — Reduced-motion support exists for smooth scrolling, but CSS transitions are not globally disabled

148 +

149 +**Severity:** low

150 +**Principle:** Inclusiveness

151 +**Artifacts:** `layouts/partials/footer.html`, `assets/css/common/footer.css`

152 +

153 +**Finding:**

154 +Anchor smooth scrolling respects `prefers-reduced-motion: reduce`, which is good. The floating top-link still has `transition: visibility .3s, opacity .3s ...` with no reduced-motion override. This is minor, but it shows reduced-motion coverage is component-by-component rather than systemic.

155 +

156 +**Recommendation:**

157 +Add a global reduced-motion rule that disables non-essential transitions/animations, or at least override `.top-link` transition under `@media (prefers-reduced-motion: reduce)`.

158 +

159 +---

160 +

161 +#### Finding A3 — Heading order, keyboard labels, and `lang` pass the quick source sweep

162 +

163 +**Severity:** info

164 +**Principle:** Inclusiveness

165 +**Artifacts:** `layouts/index.html`, `layouts/weekly/single.html`, `layouts/partials/header.html`, `layouts/_default/baseof.html`, `layouts/partials/footer.html`

166 +

167 +**Finding:**

168 +Key templates use one H1 followed by H2/H3 structures; the GitHub icon button has an `aria-label`; the theme toggle and mobile menu summary have labels; the skip link targets `#main-content`; the footer Manage Cookies control is a native button; and `<html lang="{{ site.Language }}">` is set.

169 +

170 +**Recommendation:**

171 +No gap found in the source sweep. Once the Hugo build baseline is fixed, run rendered-page keyboard and screen-reader smoke tests on home, a weekly article, privacy, and dashboard.

172 +

173 +---

174 +

175 +### 5. Transparency

176 +

177 +#### Finding T1 — Source-data methodology is not reader-facing

178 +

179 +**Severity:** medium

180 +**Principle:** Transparency, Fairness

181 +**Artifacts:** `README.md`, `docs/analysis-spec.md`, `scripts/crawl.py`, `content/about/_index.md`

182 +

183 +**Finding:**

184 +Operator-facing docs explain the analyzer contract and README describes the pipeline. A normal reader on the public site sees an About page and cost dashboard, but not a methodology page that explains how repos are selected, what "trending" means, what cutoff dates apply, why TechCrunch is used for press context, or why star deltas may be missing.

185 +

186 +**Recommendation:**

187 +Publish a short `/methodology/` page linked from article footers and About. It should be plain-language, not an implementation spec: inputs, cutoffs, ranking, limitations, correction path, and source-bias caveats.

188 +

189 +---

190 +

191 +#### Finding T2 — Prompts are public and tracked

192 +

193 +**Severity:** info

194 +**Principle:** Transparency

195 +**Artifacts:** `prompts/analyze-weekly.md`, `prompts/analyze-topic.md`, `prompts/reskill.md`, `.gitignore`

196 +

197 +**Finding:**

198 +The prompts directory is tracked and not ignored. This is a transparency strength: readers and contributors can inspect the instructions that shape the editorial output.

199 +

200 +**Recommendation:**

201 +Link to the prompts from the future methodology page so transparency is discoverable without browsing the repository tree.

202 +

203 +---

204 +

205 +#### Finding T3 — Model naming is mixed: disclosure can stay generic, cost dashboard is specific

206 +

207 +**Severity:** low

208 +**Principle:** Transparency

209 +**Artifacts:** `content/about/_index.md`, `layouts/partials/cost-dashboard.html`, `data/metrics/cost-summary.json`

210 +

211 +**Finding:**

212 +The About page uses generic language ("automated AI analysis pipeline"), while the cost dashboard renders model names from the metrics ledger. Specific model names are useful for cost accountability, but they should not be presented as quality endorsements or marketing claims.

213 +

214 +**Recommendation:**

215 +Keep article-level AI disclosure generic, and label the dashboard's model field as operational metadata only.

216 +

217 +---

218 +

219 +### 6. Accountability

220 +

221 +#### Finding C1 — No site-wide content contact path in footer

222 +

223 +**Severity:** medium

224 +**Principle:** Accountability, Reliability & Safety

225 +**Artifacts:** `layouts/partials/footer.html`, `content/privacy/_index.md`

226 +

227 +**Finding:**

228 +The privacy page has a GitHub issues contact path, but the global footer only links Privacy, Manage cookies, GitHub, RSS, and Archive. A reader who finds an editorial error on an article page has to infer that GitHub is the right place to report it.

229 +

230 +**Recommendation:**

231 +Add a footer `Contact` or `Report an issue` link to GitHub Issues, and add article-specific links with page context. This should be implemented with Finding R1.

232 +

233 +---

234 +

235 +#### Finding C2 — No content license or reuse terms found

236 +

237 +**Severity:** medium

238 +**Principle:** Accountability, Transparency

239 +**Artifacts:** repository root, `README.md`, `content/weekly/`, `content/about/_index.md`

240 +

241 +**Finding:**

242 +No `LICENSE`, `COPYING`, terms page, or content reuse statement was found. The site publishes AI-assisted editorial text and repository summaries, but contributors and readers do not know whether the code, generated content, raw data, and analysis are reusable under the same terms.

243 +

244 +**Recommendation:**

245 +Add an explicit repository license and content license/reuse statement. If code and content use different licenses, state that clearly in README and the public site footer.

246 +

247 +---

248 +

249 +#### Finding C3 — Maintainer ownership is partially named, but editorial ownership is still vague

250 +

251 +**Severity:** low

252 +**Principle:** Accountability

253 +**Artifacts:** `content/privacy/_index.md`, `README.md`, `content/about/_index.md`

254 +

255 +**Finding:**

256 +The privacy page names `jmservera` as data controller and site operator. That is enough for privacy ownership, but editorial ownership is still framed mostly as an automated pipeline rather than a person or project maintainer accountable for corrections.

257 +

258 +**Recommendation:**

259 +On About or Methodology, state that jmservera is the maintainer/operator and that corrections should be filed through GitHub Issues.

260 +

261 +---

262 +

263 +## EU AI Act transparency checks

264 +

265 +### Article 50 / Article 52 — AI-generated or manipulated content must be distinguishable at delivery time

266 +

267 +**Status:** Covered by in-flight AI-content disclosure PR (`squad/208-ai-content-disclosure`) per scope instruction; not audited here.

268 +

269 +**Residual recommendation:**

270 +The disclosure must be per article and visible at delivery time, not only in a policy page. It should link to the methodology/corrections pages once those exist.

271 +

272 +### Foundation model labeling

273 +

274 +**Status:** In-flight disclosure is expected to cover this. Cost dashboard already records model metadata for operations, but article disclosure should avoid implying model endorsement.

275 +

276 +---

277 +

278 +## Cross-cutting issues

279 +

280 +1. **Governance is now the main RAI gap.** Cookie consent, GA4 fork-safety, and prompt-injection hardening moved the technical baseline forward. The remaining risk is what happens after publication: who corrects, retracts, or explains errors.

281 +2. **Methodology needs to move from repo docs to reader docs.** `README.md` and `docs/analysis-spec.md` are useful to operators, but public readers need a shorter explanation of source selection and limitations.

282 +3. **Representation risk is measurable.** The pipeline already has enough structured data (`language`, `owner`, `topics`, `stars`, `forks`, `created_at`) to produce weekly source-mix telemetry. This should become a standard transparency artifact.

283 +4. **A11y regressions are small but concrete.** The new design system mostly passes the quick source sweep, but badge contrast is below AA and reduced-motion coverage should be centralized.

284 +5. **Privacy posture is mostly improved.** No comment/form surface was found; GA4 is consent-gated and fork-safe. The missing browser privacy control is Referrer-Policy.

285 +

286 +---

287 +

288 +## Comparison with First Sweep

289 +

290 +| First-sweep finding | Status in second sweep | Evidence / note |

291 +|---|---|---|

292 +| Old SS-like icon failure mode | Closed / institutionalized | Current robot/binoculars icon remains distinct; icon-safety skill exists. No new hate-symbol resemblance found in source sweep. |

293 +| Cookie consent visual-weight parity | Closed | CookieConsent config sets `equalWeightButtons: true`; footer Manage Cookies exists. |

294 +| GA4 fork-safety | Closed | Measurement ID defaults empty and deploy injects from secret; analytics partial emits nothing without ID. |

295 +| GA4 consent gate | Closed with watch item | Scripts are `type="text/plain"` and category-gated by CookieConsent. Keep first-visit network test. |

296 +| Distribution copy review | Mostly closed | Distribution strategy exists; no new manipulative copy found in current public site. Continue Nibbler review for launch/community posts. |

297 +| Prompt-injection untrusted repo descriptions | Mostly closed | Prompts now wrap raw JSON in `<untrusted-content>` and sanitizer handles suspicious description patterns. Remaining watch item: post-generation external URL validation. |

298 +| Article AI authorship disclosure | In progress outside scope | Amy-11 PR covers this; not audited here. |

299 +| Header GitHub icon ARIA | Closed | Header GitHub link has `aria-label="GitHub"`; theme/menu controls also labeled. |

300 +| `prefers-reduced-motion` absent | Partially closed | Smooth scrolling respects reduced motion, but top-link transition remains unguarded. |

301 +| Reskill prompt human-review gap | Open / not re-audited deeply | No evidence of a new human checkpoint in this sweep; lower priority than public correction/methodology gaps. |

302 +

303 +### New findings in this sweep

304 +

305 +- **High:** Source methodology/fairness disclosure gap.

306 +- **High:** No article-level errata/report/retraction process.

307 +- **Medium:** No Referrer-Policy.

308 +- **Medium:** Badge contrast below WCAG AA.

309 +- **Medium:** No site-wide content contact path.

310 +- **Medium:** No content license/reuse terms.

311 +- **Low:** CookieConsent vendored asset integrity not pinned in markup.

312 +- **Low:** Reduced-motion coverage not global.

313 +- **Low:** Model naming should stay operational, not promotional.

314 +- **Low:** Maintainer/editorial accountability copy remains vague.

315 +

316 +---

317 +

318 +## Follow-Up Issues to Open

319 +

320 +- **Issue A:** [#208](https://github.com/jmservera/SquadScope/issues/208) `rai: disclose and measure source-selection bias in methodology` — high / Fairness + Transparency.

321 +- **Issue B:** [#209](https://github.com/jmservera/SquadScope/issues/209) `rai: add article errata/report/retraction path` — high / Reliability & Safety + Accountability.

322 +

323 +---

324 +

325 +## Validation notes

326 +

327 +- Reviewed `scripts/analyze_fallback.py`, `scripts/sanitize_repo_content.py`, prompts, topic configs, workflows, privacy/about pages, key layouts, and design CSS.

328 +- Ran a local contrast calculation for design tokens and badges.

329 +- Attempted `hugo --minify`; build failed on the current baseline before any audit-doc change because `layouts/partials/head.html` expects missing PaperMod CSS resources (`css/core/zmedia.css`, `css/core/license.css`, `css/includes/*`). This audit is documentation-only; no runtime code was changed.

330 +

331 +*Nibbler — Responsible AI / Safety Reviewer*

332 +*2026-05-25 | Second sweep — posture: improved technical controls, open governance gaps*

chore(rai): second-sweep audit — gap analysis vs. shipped work (#210)