@cryptotaxi247 / SquadScope / commits / 92d4f9c

Document safe rerun and restore operations (#285)

* docs: document safe rerun modes, restore, no-AI policy, and map/reduce dry-run status (#259) Implement comprehensive operator documentation for: - Safe rerun modes (normal, dry-run, restore, force-replace, candidate-only) - Source refresh policies (reuse-same-day, refresh-missing-stale, force-refresh) - Source artifact tracking and provenance in publish manifests - Safe same-day artifact reuse with status recording - Backup creation and immutable restore procedures - No-AI fallback policy and why it doesn't replace AI-authored content - Rejected candidate diagnostics and diagnostic structure - Map/reduce analysis status and why it remains dry-run until QA gates pass Acceptance criteria: ✓ Operator can safely rerun current week without risking good content ✓ Operator can tell which sources were reused/refreshed/missing/stale/failed ✓ Restore steps are explicit with tested backup procedures ✓ Docs explain why no-AI fallback is not default replacement ✓ Docs explain map/reduce dry-run status and QA gate requirements Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(#259): Fix source artifacts and gate report structures in operator guide Revise docs/operator-guide.md to match actual implementation: 1. Source artifacts (lines 256-277): Replace incorrect simplified fields (name/status/timestamp/reason) with actual nested structure including role, path, exists, size_bytes, sha256, same_day_reuse, freshness, and provenance. 2. Gate report (lines 504-524): Replace inaccurate fields (validation_status/quality_score/validation_failures/repair_suggestions) with actual report structure: passed, gates (with nested gate results), errors_before_repair, repair_actions, errors_after_repair, and failure_class. 3. Remove non-existent workflow input 'map_reduce_enabled=true' from dry-run example. The feature is not yet implemented in the workflow. Update jq commands to query actual fields. All examples now match publish_manifest.py and analysis_gate.py implementations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed Jun 6, 2026 at 22:38 UTC 92d4f9c52be0426b816155c9201eb810358ea835

1 file changed +426 -7

docs/operator-guide.md

+426 -7

@@ -146,15 +146,37 @@ The workflow takes ~2-3 minutes depending on GitHub API response times.

146

147 Manual runs default to `run_mode=normal` and `source_refresh_policy=reuse-same-day`. Normal mode is fail-closed: it may publish only after the existing analysis and freshness gates pass, and same-day successful source artifacts are reused instead of scraping again. Missing, failed, stale, wrong-week, or wrong-window sources are refreshed.

148

149 -Use explicit modes for safer or destructive intent:

149 +##### Rerun mode reference

150

151 -- `dry-run`: build candidate artifacts only; never commit, deploy, notify, or publish a release.

152 -- `candidate-only`: run crawl/analysis and upload candidates; promotion is blocked by the manifest.

153 -- `restore`: requires `rebuild_week=YYYY-WNN`; hydrates artifacts from `publish` for audited restore/regeneration.

154 -- `force-replace`: explicit replacement intent, but gates still must pass before promotion.

155 -- `source_refresh_policy=force-refresh`: explicitly bypass same-day source reuse and refresh sources.

151 +All rerun modes are validated before any publishing side effects:

152

157 -Invalid combinations (for example `rebuild_week` without `run_mode=restore`, `publish_release` with `dry-run`, or `restore` with `force-refresh`) fail before publish content can be modified.

154 +|------|-------|-----------|--------|----------|

160 +

161 +##### Source refresh policies

162 +

163 +Control how same-day artifacts are handled during reruns:

164 +

165 +| Policy | Behavior | Use case |

166 +|--------|----------|----------|

167 +| `reuse-same-day` (default) | Reuse eligible same-day raw artifacts; refresh missing/stale/failed sources | Safe rerun without redundant API calls |

168 +| `refresh-missing-stale` | Like reuse-same-day but also refresh sources with missing or stale status | Partial refresh, correct specific source issues |

169 +| `force-refresh` | Refresh all sources regardless of prior status | Force all new data, ignore cache |

170 +

171 +Same-day artifact reuse is safe by design:

172 +- Only successfully crawled artifacts are eligible for reuse

173 +- Missing, failed, stale (>24 hours old), or wrong-week sources are always refreshed

174 +- Source status (reused/refreshed/missing/failed/stale) is recorded in the publish manifest for audit trail

175 +

176 +Invalid combinations fail immediately with clear error messages:

177 +- `rebuild_week` without `run_mode=restore`

178 +- `run_mode=restore` with `source_refresh_policy=force-refresh`

179 +- `publish_release=true` with `dry-run` or `candidate-only`

180

181 ### Option C: Run individual stages locally

182

@@ -206,6 +228,403 @@ hugo --minify

228

229 Output: `public/` directory ready for GitHub Pages.

230

231 +## Understanding Source Artifacts and Reuse

232 +

233 +### Source artifact tracking

234 +

235 +When SquadScope crawls, it records detailed information about each source artifact:

236 +

237 +- **Status:** One of `reused`, `refreshed`, `missing`, `failed`, or `stale`

238 +- **Artifact checksum:** SHA256 of successful crawls for integrity verification

239 +- **Timestamp:** When the artifact was produced or reused

240 +- **Code checksum:** Hash of the crawler code that produced it (detects version drift)

241 +

242 +This metadata is stored in the **publish manifest** (`data/candidates/YYYY-WNN/RUN_ID/publish-manifest.json`) for every run, creating an auditable trail of:

243 +- Which sources were fetched vs. reused

244 +- Why sources were refreshed (missing, failed, stale, code drift)

245 +- Provenance of every analysis artifact

246 +

247 +### Examining source status

248 +

249 +After a run completes, check the publish manifest to see which sources were reused or refreshed:

250 +

251 +```bash

252 +# Find the latest manifest for week 2026-W21

253 +find data/candidates/2026-W21 -name publish-manifest.json | sort -V | tail -1 | xargs cat | jq '.source_artifacts'

254 +```

255 +

256 +Output shows:

257 +```json

258 +{

259 + "source_artifacts": [

260 + {

261 + "role": "raw_github",

262 + "path": "data/candidates/2026-W21/github-crawl.json",

263 + "exists": true,

264 + "size_bytes": 45823,

265 + "sha256": "686085ace216e10d36837a91471e28a334b2fc3d93cc1085b8d5d0e7616891bf",

266 + "same_day_reuse": {

267 + "status": "reused",

268 + "source": "default"

269 + },

270 + "freshness": {

271 + "status": "fresh",

272 + "reasons": []

273 + },

274 + "provenance": {

275 + "generated_at": "2026-05-20T10:15:00Z",

276 + "sha256": "686085ace216e10d36837a91471e28a334b2fc3d93cc1085b8d5d0e7616891bf"

277 + }

278 + },

279 + {

280 + "role": "external_news",

281 + "path": "data/candidates/2026-W21/news-articles.json",

282 + "exists": true,

283 + "size_bytes": 28941,

284 + "sha256": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1",

285 + "same_day_reuse": {

286 + "status": "not_reused",

287 + "source": "refresh_policy"

288 + },

289 + "freshness": {

290 + "status": "stale",

291 + "reasons": ["source_refresh_policy=refresh-missing-stale"]

292 + },

293 + "provenance": {

294 + "generated_at": "2026-05-21T08:30:00Z",

295 + "sha256": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1"

296 + }

297 + }

298 + ]

299 +}

300 +```

301 +

302 +### Safe rerun scenario

303 +

304 +**Scenario:** You rerun Monday's analysis on Tuesday morning (same week) to fix a quality gate failure.

305 +

306 +**Expected behavior with `source_refresh_policy=reuse-same-day` (default):**

307 +1. Monday's successful GitHub crawl is reused (1 API call saved)

308 +2. Any failed or missing sources from Monday are refreshed

309 +3. Analysis gates run on fresh analysis only

310 +4. If gates pass, publish replaces Monday's article

311 +5. If gates fail, the publish manifest blocks the promotion and preserves Monday's good article

312 +

313 +**This is safe because:**

314 +- Only Monday's *successful* artifacts are reused

315 +- Any source that failed on Monday is fetched fresh

316 +- The manifest explicitly records what was reused

317 +- Quality gates prevent bad analysis from being published

318 +- Good prior analysis is preserved if the retry fails

319 +

320 +### When to use each policy

321 +

322 +**Use `reuse-same-day` (default):**

323 +- Standard reruns within the same day

324 +- Debugging analysis issues

325 +- Retrying quality gates after minor fixes

326 +

327 +**Use `refresh-missing-stale`:**

328 +- Some sources failed and you've fixed the crawler

329 +- You want to update stale sources without fully refreshing

330 +

331 +**Use `force-refresh`:**

332 +- You suspect source data is corrupted or needs manual validation

333 +- You're testing source updates

334 +- Policy: Always use explicit intent for full refresh

335 +

336 +## Safe Restore from Backup

337 +

338 +### Understanding backups

339 +

340 +Before any weekly article or analysis is replaced in the `publish` branch, an immutable backup is created at:

341 +

342 +```

343 +data/backups/YYYY-WNN/RUN_ID/[analysis|content]/manifest.json

344 +```

345 +

346 +Backups include:

347 +- The exact prior content being replaced (e.g., `content/weekly/2026/W21.md`)

348 +- The prior analysis file (e.g., `data/analyzed/2026-W21-summary.md`)

349 +- SHA256 checksums of all backed-up files

350 +- Timestamp and run context

351 +

352 +### When backups are created

353 +

354 +A backup is automatically created when:

355 +1. A new analysis is about to replace a prior week's analysis, OR

356 +2. A new content page is about to replace a prior week's HTML page

357 +

358 +Backups are immutable—they cannot be modified or deleted by subsequent runs.

359 +

360 +### Viewing available backups

361 +

362 +```bash

363 +# List all available backups for week 2026-W21

364 +find data/backups/2026-W21 -name manifest.json | sort -V

365 +

366 +# Inspect a backup manifest

367 +cat data/backups/2026-W21/RUN_ID/content/manifest.json | jq .

368 +```

369 +

370 +Backup manifest shows:

371 +```json

372 +{

373 + "schema_version": "publish_backup_v1",

374 + "week": "2026-W21",

375 + "run_id": 12345678,

376 + "timestamp": "2026-05-20T10:15:00Z",

377 + "backed_up_artifacts": [

378 + {

379 + "path": "data/analyzed/2026-W21-summary.md",

380 + "sha256": "abc123...",

381 + "exists_before_replacement": true

382 + },

383 + {

384 + "path": "content/weekly/2026/W21.md",

385 + "sha256": "def456...",

386 + "exists_before_replacement": true

387 + }

388 + ]

389 +}

390 +```

391 +

392 +### Restore a prior week from backup

393 +

394 +To restore a prior week (e.g., restore 2026-W21 to a known-good state):

395 +

396 +1. **Identify the backup manifest** you want to restore:

397 + ```bash

398 + # List backups for week 2026-W21, sorted by timestamp

399 + find data/backups/2026-W21 -name manifest.json | sort -V

400 + ```

401 +

402 +2. **Trigger the restore workflow:**

403 + ```bash

404 + gh workflow run restore-publish-backup.yml \

405 + -R YOUR_USERNAME/SquadScope \

406 + -f "backup_manifest=data/backups/2026-W21/RUN_ID/content/manifest.json"

407 + ```

408 +

409 + Or through the UI:

410 + - Go to **Actions → Restore publish backup**

411 + - Click **Run workflow**

412 + - Paste the backup manifest path (e.g., `data/backups/2026-W21/12345678/content/manifest.json`)

413 + - Click **Run workflow**

414 +

415 +3. **Restore will:**

416 + - Check out the `publish` branch

417 + - Validate the backup manifest integrity

418 + - Restore all backed-up files to their pre-replacement state

419 + - Commit with message: `restore: publish backup {manifest_path}`

420 + - Force-push to `publish` with lease safety guards

421 +

422 +4. **Monitor the restore:**

423 + ```bash

424 + gh run view --log -R YOUR_USERNAME/SquadScope

425 + ```

426 +

427 +### Restore operation guarantees

428 +

429 +- **Immutable:** Backup manifests are never modified after creation

430 +- **Atomic:** Restore applies all backed-up files or fails with no partial changes

431 +- **Lease-guarded:** Force-push uses `--force-with-lease` to detect concurrent modifications

432 +- **Audited:** Restore commit message includes the backup manifest path for traceability

433 +- **Non-destructive:** Restoring does not delete new backups created since the restore date

434 +

435 +**After a restore:**

436 +- The `publish` branch is reverted to the state before that run

437 +- Previous good analysis remains published

438 +- The restore itself appears in git history for audit trail

439 +

440 +## No-AI Fallback Policy

441 +

442 +### Why no-AI is not a replacement strategy

443 +

444 +SquadScope includes a no-AI fallback analysis script (`scripts/analyze_fallback.py`) that can generate a basic summary using heuristics when Copilot is unavailable. However, **no-AI output is explicitly not a replacement for Copilot analysis** and has specific constraints:

445 +

446 +**No-AI fallback is used only when:**

447 +1. Copilot CLI fails with a non-auth error (after retries), OR

448 +2. Copilot encounters a non-recoverable error (e.g., context too large), OR

449 +3. Copilot is completely inaccessible

450 +

451 +**No-AI output characteristics:**

452 +- Lower quality_score (typically 40–50 vs. 60+ for Copilot)

453 +- Simple heuristic categorization (no editorial synthesis)

454 +- May have incomplete signal/noise/gaps sections

455 +- Preserved as a "rejected candidate" artifact

456 +- **Does not publish without explicit operator override**

457 +

458 +### Publish manifest promotion policy

459 +

460 +When Copilot fails and no-AI fallback is generated:

461 +1. The no-AI candidate is created and stored in `data/candidates/YYYY-WNN/RUN_ID/`

462 +2. The publish manifest records `analysis_source: "no-ai"` and `quality_validation: "failed"`

463 +3. **The promotion guard blocks publication** regardless of whether gates pass

464 +4. The prior week's good analysis remains published (safe default)

465 +

466 +To inspect a rejected no-AI candidate:

467 +

468 +```bash

469 +# List rejected candidates for week 2026-W21

470 +find data/candidates/2026-W21 -name 'candidate-no-ai-attempt-*.md'

471 +

472 +# Review the no-AI candidate and its gate report

473 +cat data/candidates/2026-W21/RUN_ID/diagnostics/candidate-no-ai-attempt-0.md

474 +cat data/candidates/2026-W21/RUN_ID/diagnostics/gate-no-ai-attempt-0.json | jq '.validation_failures'

475 +```

476 +

477 +### Copilot access failures

478 +

479 +If Copilot CLI fails with an authentication or access error:

480 +- The workflow **fails immediately without attempting no-AI fallback**

481 +- An issue is created (or updated) to notify the operator to renew `COPILOT_GH_TOKEN`

482 +- The failure report is available at `data/candidates/YYYY-WNN/RUN_ID/diagnostics/copilot-cli-failure-*.json`

483 +

484 +This ensures that **transient Copilot issues do not silently degrade to no-AI analysis.**

485 +

486 +### Copilot retries

487 +

488 +If Copilot produces output that doesn't pass the quality gate, the workflow automatically retries up to 3 times:

489 +- Each retry includes focused diagnostics from the prior gate failure

490 +- If all retries fail, no-AI fallback is attempted as a last resort

491 +- Each retry and its diagnostics are recorded for audit trail

492 +

493 +## Rejected Candidate Diagnostics

494 +

495 +When analysis fails to pass quality gates, detailed diagnostics are recorded for investigation:

496 +

497 +### Candidate directory structure

498 +

499 +```

500 +data/candidates/YYYY-WNN/RUN_ID/

501 + ├── YYYY-WNN-summary.md # Candidate analysis (if produced)

502 + ├── YYYY-WNN-content.md # Generated HTML candidate (if produced)

503 + ├── publish-manifest.json # Eligibility and provenance

504 + └── diagnostics/

505 + ├── analysis-preflight.json # Pre-analysis context budget check

506 + ├── analysis-preflight.md # Preflight diagnostic report

507 + ├── copilot-cli-attempt-N.log # Raw Copilot CLI stderr/stdout

508 + ├── gate-copilot-cli-attempt-N.json # Quality gate failure details

509 + ├── candidate-copilot-cli-attempt-N.md # Candidate snapshot

510 + ├── candidate-no-ai-attempt-0.md # No-AI fallback (if used)

511 + └── gate-no-ai-attempt-0.json # No-AI gate report

512 +```

513 +

514 +### Examining a failed gate report

515 +

516 +```bash

517 +# View gate failure for attempt 0

518 +cat data/candidates/2026-W21/RUN_ID/diagnostics/gate-copilot-cli-attempt-0.json | jq '{

519 + passed: .passed,

520 + gates: .gates,

521 + errors_before_repair: .errors_before_repair,

522 + repair_actions: .repair_actions,

523 + failure_class: .failure_class

524 +}'

525 +```

526 +

527 +Gate report output:

528 +```json

529 +{

530 + "passed": false,

531 + "gates": {

532 + "structural_schema": {

533 + "passed": false,

534 + "errors": [

535 + "Signal section is empty or malformed",

536 + "Signal section must contain at least 3 significant claims"

537 + ]

538 + },

539 + "editorial_quality": {

540 + "passed": false,

541 + "errors": [

542 + "Noise section has fewer than 3 spurious claims"

543 + ]

544 + },

545 + "ai_provenance": {

546 + "passed": true,

547 + "errors": []

548 + },

549 + "evidence_citation": {

550 + "passed": true,

551 + "errors": []

552 + }

553 + },

554 + "errors_before_repair": [

555 + "Signal section is empty or malformed",

556 + "Signal section must contain at least 3 significant claims",

557 + "Noise section has fewer than 3 spurious claims"

558 + ],

559 + "repair_actions": [

560 + "Expanded Signal section with 3 significant claims from trending repositories",

561 + "Added 3 spurious/false claims to Noise section"

562 + ],

563 + "errors_after_repair": [],

564 + "failure_class": "passed"

565 +}

566 +```

567 +

568 +### Quality gate specifics

569 +

570 +The publish manifest records:

571 +- `quality_score`: 0–100, where 60+ is publishable

572 +- `quality_source`: "copilot-cli", "no-ai", etc.

573 +- `validation_status`: "passed" or "failed"

574 +- `validation_failures`: Array of specific failures with repair suggestions

575 +

576 +Failed gates block promotion but don't prevent the candidate from being stored for audit trail.

577 +

578 +## Map/Reduce Analysis Status (Dry-Run Only)

579 +

580 +### Why map/reduce remains experimental

581 +

582 +SquadScope's analysis pipeline is currently single-pass for production use. Map/reduce analysis—dividing evidence into smaller chunks, analyzing each independently, then combining results—is **only available as a dry-run experimental feature** and cannot publish. This decision was made after evidence from live runs and is documented in `docs/PRD-matrix-crawl-map-reduce-analysis.md`.

583 +

584 +### Current limitations preventing map/reduce publication

585 +

586 +1. **Analysis specification mismatch:** Map/reduce mapper outputs would create intermediate artifacts not conforming to `docs/analysis-spec.md`

587 +2. **Citation preservation:** Combining mapper outputs risks losing original citations and creating false attribution

588 +3. **Claim deduplication:** Reducer must reliably dedupe claims across mappers; no production-grade deduplication exists yet

589 +4. **Token accounting:** Final combined analysis may exceed token budgets; mechanism for bounded reduction is unproven

590 +5. **Quality gate compliance:** Existing gates expect single-pass analysis structure; map/reduce will need new gates

591 +

592 +### QA gates required before map/reduce can publish (#258)

593 +

594 +Before map/reduce analysis can be enabled for production promotion, all of these QA gates must pass:

595 +

596 +- [ ] **Mapper-reducer contract testing:** Mappers and reducers in sandboxed runs must produce deterministic, validated outputs

597 +- [ ] **Citation preservation testing:** Full analysis -> map/reduce roundtrip must preserve or improve citation count/accuracy

598 +- [ ] **Claim deduplication testing:** Reducer must reliably identify and merge duplicate claims across mappers

599 +- [ ] **Token budget compliance:** End-to-end analysis must stay within token limits; no truncation or quality regression

600 +- [ ] **Spec compliance testing:** Generated analysis must pass all existing `analysis_gate.py` checks without modification

601 +- [ ] **Human editorial review:** Blind A/B comparison of single-pass vs. map/reduce outputs from 4+ weeks of real data

602 +- [ ] **Fallback behavior:** Ensure Copilot retries and no-AI fallback work correctly with map/reduce logic

603 +

604 +### Testing map/reduce in dry-run mode

605 +

606 +To test map/reduce without risk of publication:

607 +

608 +```bash

609 +gh workflow run crawl-and-publish.yml \

610 + -R YOUR_USERNAME/SquadScope \

611 + -f "run_mode=dry-run"

612 +```

613 +

614 +Dry-run mode ensures:

615 +- Analysis is generated but never promoted

616 +- No HTML is published

617 +- Candidates are stored for inspection

618 +- You can review results before any live traffic sees them

619 +

620 +### Expected timeline

621 +

622 +Map/reduce publication will be enabled in a future phase after:

623 +- All QA gates (#258) are implemented and passing

624 +- Human review confirms output quality meets or exceeds single-pass

625 +- Operator documentation is completed

626 +- Rollback procedures are tested

627 +

628 ## Monitoring the Cron Schedule

629

630 ### View recent runs