| 1 | # Bounded NSE daily OHLCV backfill |
| 2 | |
| 3 | ## Entry point and universe |
| 4 | |
| 5 | `IndiaMarketDataPopulationJobs.backfill_daily_bars(identity_headers=..., |
| 6 | correlation_id=None, offset=0, instrument_ids=None, start=None, end=None, |
| 7 | force=False)` is an explicit async worker operation. It shares the existing |
| 8 | daily-bar lock with single-instrument acquisition. No GET, scanner, submit, |
| 9 | ensure, or scheduler automatically invokes it. Yahoo close-only behavior stays |
| 10 | unchanged. This extends the population boundary, not a second job framework. |
| 11 | |
| 12 | The source is `active_global_equities()` (paginated canonical ACTIVE EQUITY |
| 13 | metadata). India/NSE candidates are deduplicated and sorted by UUID string; |
| 14 | at most `market_data_population_batch_size` (default 50) are selected. |
| 15 | Each selected candidate then resolves current detail metadata and passes the |
| 16 | committed `verified_identity` gate before NSE acquisition. Invalid mappings |
| 17 | consume a bounded batch slot and appear as failures; they are never fetched. |
| 18 | Listing metadata uses `exchange`; detail metadata uses `primaryExchange`, as |
| 19 | specified by the existing canonical API. No holdings/watchlists/sector filter. |
| 20 | |
| 21 | Optional IDs restrict canonical membership; IDs absent from the universe are |
| 22 | not acquired. `next_offset` is null at completion, otherwise the next offset in |
| 23 | the sorted candidate universe or explicit-ID intersection. Selection is stable |
| 24 | for the same snapshot. Universe membership changes can shift offsets; no durable |
| 25 | snapshot cursor is claimed. Only identity metadata is enumerated globally; |
| 26 | history is read one instrument at a time and persisted per window. |
| 27 | |
| 28 | ## Date and coverage policy |
| 29 | |
| 30 | Default interval: Asia/Kolkata current DATE minus existing initial lookback |
| 31 | (default 400 days), through current DATE inclusive. Optional bounds may narrow |
| 32 | but not extend that horizon or request future dates. Retrieval timestamps remain |
| 33 | UTC. `plan_windows` produces consecutive non-overlapping oldest-first windows, |
| 34 | using the configured provider limit (default 30 inclusive calendar days). A |
| 35 | 401-calendar-date interval uses 14 windows. Thirty days is an operational bound, |
| 36 | not a claimed official NSE maximum. |
| 37 | |
| 38 | Reads reuse `daily_market_bars_for_instruments({id}, end_date=end, provider='NSE')`. |
| 39 | Only the canonical ID, current symbol/currency, REAL NSE rows with non-null OHLC |
| 40 | satisfy coverage. Optional volume/turnover are not required. |
| 41 | |
| 42 | - NO_HISTORY: fetch the target interval. |
| 43 | - PARTIAL_HISTORY: fetch the missing prefix before the earliest usable date; |
| 44 | also fetch a stale tail when needed. |
| 45 | - STALE_HISTORY: retrieval age or latest-date lag exceeds configured historical |
| 46 | freshness (72 hours). Fetch latest date + 1 through target end. If latest date |
| 47 | already equals target end, refresh that final date for corrections/freshness. |
| 48 | - CURRENT_HISTORY: observed prefix is covered and tail sufficiently fresh; skip. |
| 49 | This does not assert that every internal exchange session is present. |
| 50 | - GAP_DETECTED is deliberately not inferred without authoritative sessions. |
| 51 | |
| 52 | INTERNAL_GAP_REPAIR = UNSUPPORTED_IN_THIS_PHASE. Weekly schedules and optional |
| 53 | calendar exceptions do not establish a complete historical NSE holiday calendar. |
| 54 | No weekday/holiday is synthesized or labelled missing market data. |
| 55 | |
| 56 | Successful request ranges are remembered for the existing freshness interval |
| 57 | within the worker, keyed by canonical ID/symbol/currency. This suppresses repeated |
| 58 | boundary probes when a completed range has no bar on its exact first date. |
| 59 | Memory is used only alongside existing usable persisted rows. Missing rows still |
| 60 | cause NO_HISTORY acquisition. Empty/failed requests are never marked complete. |
| 61 | State is process-local, like existing jobs; after restart boundary probes can |
| 62 | recur. Listing-date-aware prefix suppression and durable empty-range evidence |
| 63 | are not implemented. |
| 64 | |
| 65 | Force defaults false. Explicit force reacquires the requested bounded interval |
| 66 | for idempotent correction, but does not bypass cooldown or throttling. |
| 67 | |
| 68 | ## Execution and results |
| 69 | |
| 70 | One lazily created NseHistoricalDailyProvider/session per job, closed in finally |
| 71 | including cancellation. Its existing cookie/header, spacing and bounded retry |
| 72 | behavior is reused. The population lock and inter-job spacing serialize this |
| 73 | operation with single-instrument acquisition. No global HTTP session, proxy |
| 74 | logic or concurrent NSE requests. Coordination is process-local, not distributed. |
| 75 | |
| 76 | Each successful window uses `persist_daily_result` and existing |
| 77 | `ResearchRepository.upsert_daily_market_bars_async`. No SQL/schema change and no |
| 78 | close-only dual-write. Earlier good rows survive later failures. |
| 79 | |
| 80 | Failures distinguish identity lookup, mapping, provider, throttling, HTTP, parser, |
| 81 | empty response, persistence and planning. Most failures stop the current |
| 82 | instrument's remaining windows, then continue with the next instrument. Empty |
| 83 | windows remain failures but allow later windows (for example, after listing). |
| 84 | Partial parser rejections persist accepted rows but count as failed windows and |
| 85 | never mark request coverage complete. A 429 stops further NSE calls for the job |
| 86 | and sets the existing 12-hour cooldown for subsequent backfill calls. Other |
| 87 | failures use per-instrument cooldown. Force cannot override either cooldown. |
| 88 | |
| 89 | Requested-window counts are actual attempts, excluding unattempted windows |
| 90 | after failures. Processed instruments include failures and skips, and equal |
| 91 | succeeded + failed + skipped_current + skipped_cooldown. Per-instrument results |
| 92 | include requested DATE windows, coverage state, status/failure class/reason, |
| 93 | HTTP status, row counts and observed persisted bounds. Persisted row counts are |
| 94 | upserts, not necessarily newly inserted keys. No cookies, authentication headers |
| 95 | or raw exception text appear in summaries. |
| 96 | |
| 97 | ## Runtime smoke, 2026-09-13 |
| 98 | |
| 99 | A read-only canonical PostgreSQL snapshot contained 2,568 ACTIVE equities. The |
| 100 | existing trusted gate accepted all six examples: NILKAMAL, RAYMOND, GODREJAGRO, |
| 101 | ADANIPORTS, TDPOWERSYS, GEEKAYWIRE, plus POLYCAB. Names are validation examples, |
| 102 | not implementation selection logic. |
| 103 | |
| 104 | A runtime adapter exposed this canonical snapshot through existing universe and |
| 105 | metadata method shapes. The generic worker was restricted to NILKAMAL |
| 106 | `4b085a61-0864-4ef1-ae23-0ba7e3ec6afb` and POLYCAB |
| 107 | `f8cb0fc7-082c-4d95-a77d-b1a9ca21d5a4`, September 1–4, 2026. |
| 108 | |
| 109 | First run: two NO_HISTORY instruments, two successful windows, eight rows |
| 110 | received/accepted/persisted. Three NSE HTTP calls total: one bootstrap and two |
| 111 | history requests, all HTTP 200. Retries were disabled for this smoke. Current |
| 112 | ResearchRepository/SQLite persistence read back eight unique provider/day rows. |
| 113 | Immediate repeat: two CURRENT_HISTORY skips, zero windows, zero NSE calls, |
| 114 | identical rows. Runtime artifacts are under ignored `.tmp/`. |
| 115 | |
| 116 | This validates live NSE and current local orchestration/persistence, not |
| 117 | authenticated canonical HTTP enumeration or deployed PostgreSQL application |
| 118 | end-to-end behavior. Mocked provider/HTTP and real SQLite repository tests cover |
| 119 | multi-window failure isolation, continuation, corrections, and cooldowns. |
| 120 | |
| 121 | Out of scope: deployed PostgreSQL validation, ATR/ADX and volume technical |
| 122 | wiring, internal-gap repair without an authoritative calendar, sector benchmark |
| 123 | mapping/history, and broad-market benchmark history. |