| 1 | # Shard 435 Crawl Experiment Plan |
| 2 | |
| 3 | ## Goal |
| 4 | |
| 5 | Run a local-only, no-publish experiment that compares the monolithic GitHub crawl with a sharded variant while enforcing: |
| 6 | |
| 7 | - at least 25% wall-clock improvement |
| 8 | - no more than 10% GitHub API growth |
| 9 | - no secondary-rate-limit regression |
| 10 | - byte-stable downstream payloads after deterministic fan-in |
| 11 | |
| 12 | ## Shard Boundaries |
| 13 | |
| 14 | 1. **Shard 1 — new-search** |
| 15 | - runs the new-repository search query group |
| 16 | - in config mode, runs primary queries first and only falls back to secondary queries when needed |
| 17 | |
| 18 | 2. **Shard 2 — trending-search** |
| 19 | - runs the trending-repository search query group |
| 20 | - skipped only when config mode intentionally has no trending bucket and must preserve existing semantics |
| 21 | |
| 22 | 3. **Shard 3+ — validate-N** |
| 23 | - consume deduplicated candidate chunks from a shared queue |
| 24 | - perform README/detail validation and final filtering |
| 25 | - requeue unfinished work when a shard hits its wall-clock limit so other shards can absorb the remainder |
| 26 | |
| 27 | ## Deterministic Fan-in |
| 28 | |
| 29 | - keep new/trending buckets separate to preserve the canonical raw-artifact shape |
| 30 | - dedupe within each bucket by `full_name` |
| 31 | - sort final outputs like `crawl.py` |
| 32 | - new repos by stars descending |
| 33 | - trending repos by `stars_gained`, then stars descending |
| 34 | - rebuild signals and star snapshots only after fan-in |
| 35 | - compare canonical payload bytes after stripping volatile timestamp metadata |
| 36 | |
| 37 | ## Guardrails |
| 38 | |
| 39 | - **Wall-clock budget:** each shard gets a configurable deadline (default 120s) |
| 40 | - **API budget:** shard mode shares a global counter capped at `baseline_calls × 1.1` |
| 41 | - **Shared backoff:** retry backoff windows are propagated across shard clients |
| 42 | - **Rollback condition:** any secondary rate limit aborts remaining shard work and records a guardrail event |
| 43 | |
| 44 | ## Assumptions |
| 45 | |
| 46 | - threading is sufficient because the workload is API I/O bound |
| 47 | - cache reads are safe to share across shard clients |
| 48 | - cross-bucket duplicates may still appear in both `new_repos` and `trending_repos` because the baseline crawl already allows that |
| 49 | - at least three representative real runs are still required after implementation |
| 50 | |
| 51 | ## Output Artifacts |
| 52 | |
| 53 | All experiment artifacts stay under `data/experiments/shard-435/EXPERIMENT_ID/`: |
| 54 | |
| 55 | - `baseline-raw.json` |
| 56 | - `baseline-stars.json` |
| 57 | - `shard-raw.json` |
| 58 | - `shard-stars.json` |
| 59 | - `report.json` |
| 60 | |
| 61 | ## Suggested Command |
| 62 | |
| 63 | ```bash |
| 64 | python -m scripts.crawl_shard_experiment \ |
| 65 | --since 2026-06-06 \ |
| 66 | --as-of 2026-06-13 \ |
| 67 | --shards 3 \ |
| 68 | --wall-clock-budget 300 \ |
| 69 | --api-budget-multiplier 1.1 \ |
| 70 | --output-dir data/experiments/shard-435 |
| 71 | ``` |