@cryptotaxi247 / SquadScope / commits / 8598889

Create weekly Actions crawl job with artifact handoff (#27)

* feat: add weekly crawl workflow for #8 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address crawl workflow review Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore cache artifact to data/cache/ instead of repo root The actions/download-artifact step was extracting the crawl-cache artifact to path: . (repo root), but scripts/crawl.py reads the cache from data/cache/. This prevented cache reuse between runs. Now restore to path: data/cache/ so the extracted cache files land in the correct directory where the crawler expects them. Fixes: blocking comment on PR #27 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed May 18, 2026 at 12:58 UTC 85988897cc00c4011c3ce85f0196add190c08810
3 files changed +152
.github/workflows/crawl-and-publish.yml new
+132
@@ -0,0 +1,132 @@
1 +name: Crawl and publish raw data
2 +
3 +on:
4 + schedule:
5 + - cron: '0 8 * * 1'
6 + workflow_dispatch:
7 +
8 +permissions:
9 + actions: read
10 + contents: write
11 +
12 +concurrency:
13 + group: weekly-crawl
14 + cancel-in-progress: false
15 +
16 +jobs:
17 + crawl:
18 + runs-on: ubuntu-latest
19 +
20 + steps:
21 + - name: Check out repository
22 + uses: actions/checkout@v4
23 + with:
24 + fetch-depth: 0
25 + ref: ${{ github.event.repository.default_branch }}
26 +
27 + - name: Find latest successful crawl cache
28 + id: previous-cache-run
29 + uses: actions/github-script@v7
30 + with:
31 + script: |
32 + const workflowId = 'crawl-and-publish.yml';
33 + const branch = context.payload.repository.default_branch;
34 + const currentRunId = String(context.runId);
35 + const perPage = 100;
36 + let page = 1;
37 + let previous = null;
38 +
39 + try {
40 + while (!previous) {
41 + const { data } = await github.rest.actions.listWorkflowRuns({
42 + owner: context.repo.owner,
43 + repo: context.repo.repo,
44 + workflow_id: workflowId,
45 + branch,
46 + status: 'completed',
47 + per_page: perPage,
48 + page,
49 + });
50 +
51 + previous = data.workflow_runs.find((run) => run.conclusion === 'success' && String(run.id) !== currentRunId) ?? null;
52 + if (previous || data.workflow_runs.length < perPage) {
53 + break;
54 + }
55 +
56 + page += 1;
57 + }
58 +
59 + core.setOutput('run_id', previous ? String(previous.id) : '');
60 + core.info(previous ? `Restoring crawl cache from run ${previous.id}.` : 'No previous successful crawl cache found.');
61 + } catch (error) {
62 + core.warning(`Skipping cache restore lookup: ${error.message}`);
63 + core.setOutput('run_id', '');
64 + }
65 +
66 + - name: Restore previous crawl cache
67 + if: steps.previous-cache-run.outputs.run_id != ''
68 + continue-on-error: true
69 + uses: actions/download-artifact@v4
70 + with:
71 + github-token: ${{ secrets.GITHUB_TOKEN }}
72 + run-id: ${{ steps.previous-cache-run.outputs.run_id }}
73 + name: crawl-cache
74 + path: data/cache/
75 +
76 + - name: Set up Python
77 + uses: actions/setup-python@v5
78 + with:
79 + python-version: '3.12'
80 +
81 + - name: Run crawler
82 + env:
83 + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
84 + run: python scripts/crawl.py
85 +
86 + - name: Upload raw crawl artifact
87 + if: always()
88 + uses: actions/upload-artifact@v4
89 + with:
90 + name: crawl-raw
91 + path: data/raw/
92 + if-no-files-found: warn
93 +
94 + - name: Upload snapshot artifact
95 + if: always()
96 + uses: actions/upload-artifact@v4
97 + with:
98 + name: crawl-snapshots
99 + path: data/snapshots/
100 + if-no-files-found: warn
101 +
102 + - name: Upload cache artifact
103 + if: always()
104 + uses: actions/upload-artifact@v4
105 + with:
106 + name: crawl-cache
107 + path: data/cache/
108 + if-no-files-found: warn
109 +
110 + - name: Commit crawl data
111 + env:
112 + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
113 + run: |
114 + git config user.name "github-actions[bot]"
115 + git config user.email "github-actions[bot]@users.noreply.github.com"
116 + if ! git status --short -- data/raw data/snapshots | grep -q .; then
117 + echo "No crawl data changes to commit."
118 + exit 0
119 + fi
120 + git stash push --include-untracked --message crawl-data -- data/raw data/snapshots
121 + git fetch origin "$DEFAULT_BRANCH"
122 + git checkout -B "$DEFAULT_BRANCH" "origin/$DEFAULT_BRANCH"
123 + git stash pop || {
124 + echo "Failed to reapply crawl data after syncing $DEFAULT_BRANCH."
125 + exit 1
126 + }
127 + git add data/raw/ data/snapshots/
128 + git commit -m "data: weekly crawl $(date +%Y-W%V)"
129 + git push origin "HEAD:$DEFAULT_BRANCH" || {
130 + echo "Push failed after syncing with $DEFAULT_BRANCH."
131 + exit 1
132 + }
.squad/agents/bender/history.md
+1
@@ -15,6 +15,7 @@
15 ## Learnings
16
17 - **2026-05-18T12:07:20.778+02:00:** Copilot review follow-up on crawler hardening: keep star snapshots broad for `stars_gained`, but document that they intentionally cover pre-filter candidates; restore `get_json()` payload compatibility via an internal `get_json_entry()` helper; treat malformed JSON as non-retryable; and search both `RAW_ROOT` and custom `--output` parents when loading prior star snapshots so reruns keep working.
18 +- **2026-05-18T12:07:20.778+02:00:** Issue #8 adds a dedicated `crawl-and-publish.yml` workflow for the crawl stage only: weekly Monday 08:00 UTC plus manual dispatch, serialized with `concurrency`, committing refreshed `data/raw/`, `data/snapshots/`, and `data/cache/`, and restoring the latest successful `crawl-cache` artifact via Actions API lookup so weekly runs can reuse the crawler cache.
19 - **2026-05-18T10:06:38.734+02:00:** GitHub Actions can run the standalone `copilot` CLI (`@github/copilot`) in programmatic mode with `copilot -p ...`. The safest documented CI auth flow is a fine-grained PAT with the **Copilot Requests** account permission passed as `COPILOT_GITHUB_TOKEN`; `gh auth token` only exposes an existing `gh` token and `gh-copilot` is deprecated in favor of the standalone CLI. GitHub Models (`models: read`) is the clean fallback if direct Copilot CLI automation proves brittle.
20 - **2026-05-18T10:11:20Z:** Team decided Phase 0 PRD decomposition is final; 24 GitHub issues created (4 investigation + 20 implementation). PRD decomposition captures all decisions. MCP tools can crawl beyond GitHub with remote call allowlist. Ready for issue creation.
21 - **2026-05-18T10:27:35.339+02:00:** The crawler now uses `GET /search/repositories` for both `created:>{last_week_date} stars:>50` and `pushed:>{last_week_date} stars:>50`, comparing current stars against the most recent prior `data/raw/*.json` snapshot when available to estimate weekly star gains. It authenticates with `GITHUB_TOKEN`, paginates up to the GitHub Search API's 1,000-result ceiling, caches README checks in-process, applies exponential backoff with jitter for rate limits, and skips repos whose README lookup is blocked by org SAML enforcement.
.squad/decisions/inbox/bender-actions-crawl.md new
+19
@@ -0,0 +1,19 @@
1 +# Bender decision inbox: Actions crawl workflow
2 +
3 +- **Date:** 2026-05-18T12:07:20.778+02:00
4 +- **Author:** Bender
5 +- **Issue:** #8
6 +
7 +## Proposed team decision
8 +
9 +Restore `data/cache/` from the latest successful `crawl-and-publish.yml` run before executing `scripts/crawl.py`.
10 +
11 +## Why
12 +
13 +- Reuses the crawler's on-disk GitHub API cache across weekly runs
14 +- Lowers repeated README/search calls on warm runs
15 +- Keeps the crawl stage self-contained until downstream jobs arrive in later issues
16 +
17 +## Workflow implication
18 +
19 +The workflow needs `actions: read` in addition to `contents: write` so it can discover the prior successful run and download the `crawl-cache` artifact.