| 1 | # NSE historical daily acquisition |
| 2 | |
| 3 | Integration: `IndiaMarketDataPopulationJobs.populate_daily_bars(global_instrument_id, |
| 4 | start=date, end=date, identity_headers=...)` is an explicit, single-instrument |
| 5 | worker entry point. It resolves current portfolio-service metadata, applies the |
| 6 | existing trusted provider-mapping gate plus active equity/ambiguity/currency |
| 7 | checks, and calls `ResearchRepository.upsert_daily_market_bars_async`. |
| 8 | It is deliberately not called by submit, ensure, scanners, or GET handlers. |
| 9 | The existing `HistoricalPriceProvider.closes` protocol and Yahoo priority remain |
| 10 | close-only. No dual-write occurs. |
| 11 | |
| 12 | ## Observed contract |
| 13 | |
| 14 | `tests/fixtures/nse_historical_daily.csv` is the unmodified successful official |
| 15 | NSE response captured on 2026-09-13, after a normal cookie-bearing bootstrap at |
| 16 | https://www.nseindia.com/report-detail/eq_security (HTTP 200). |
| 17 | |
| 18 | The canonical store was checked at runtime: globalInstrumentId |
| 19 | `f8cb0fc7-082c-4d95-a77d-b1a9ca21d5a4`, POLYCAB, ACTIVE EQUITY, NSE, IN, INR; |
| 20 | NSE mapping VERIFIED, resolution source OFFICIAL_NSE_NIFTY500. |
| 21 | |
| 22 | One historical request, one window, no retries: |
| 23 | https://www.nseindia.com/api/historicalOR/generateSecurityWiseHistoricalData?from=01-09-2026&to=04-09-2026&symbol=POLYCAB&type=priceVolumeDeliverable&series=EQ&csv=true |
| 24 | |
| 25 | HTTP 200, text/csv, 910 bytes, four rows; both requested boundaries were present. |
| 26 | Observed headers (surrounding whitespace omitted): Symbol, Series, Date, |
| 27 | Prev Close, Open Price, High Price, Low Price, Last Price, Close Price, |
| 28 | Average Price, Total Traded Quantity, Turnover ₹, No. of Trades, |
| 29 | Deliverable Qty, % Dly Qt to Traded Qty. |
| 30 | |
| 31 | Required: Date, Symbol, Series, Open/High/Low/Close (the explicit ` Price` |
| 32 | aliases are supported). Optional: Prev Close, Total Traded Quantity, Turnover ₹. |
| 33 | Headers are trimmed, case-folded, and whitespace-collapsed; duplicate normalized |
| 34 | headers fail the response. Symbols and series are trimmed and uppercased only. |
| 35 | Dates accept DD-MM-YYYY and observed DD-Mon-YYYY with an explicit English month |
| 36 | map, without locale or UTC conversion. Numeric grouping accepts Western and |
| 37 | Indian comma grouping; prices never pass through float. Missing optional cells |
| 38 | (blank or `-`) remain null. Duplicate valid dates reject all contenders. |
| 39 | |
| 40 | TURNOVER_UNIT: VERIFIED for the observed `Turnover ₹` header (INR). |
| 41 | TURNOVER_CONVERSION: remove validated grouping commas, parse Decimal directly; |
| 42 | multiplier 1, and only with trusted INR currency metadata. For example, |
| 43 | `13,02,54,81,530.00` becomes Decimal(`13025481530.00`). Unobserved turnover |
| 44 | headers, including `Turnover (in Lacs)`, remain unavailable/null; no lakh |
| 45 | conversion is implemented or claimed verified. |
| 46 | |
| 47 | ## Operational policy |
| 48 | |
| 49 | `nse_historical_request_window_days` defaults to 30 inclusive calendar days. |
| 50 | This is an operational bound, not an NSE guaranteed maximum. Larger requests |
| 51 | fail before metadata/network calls; this slice does not split or backfill them. |
| 52 | `nse_historical_max_retries` defaults to 2 (validated range 0–3). |
| 53 | The existing `market_data_population_request_interval_seconds` governs all |
| 54 | bootstrap/history attempts; ordinary httpx cookie storage and supported content |
| 55 | decoders are used. One provider serializes its requests; the population entry |
| 56 | point serializes calls and applies spacing between sessions. No distributed |
| 57 | rate limiter is claimed; use the existing single worker deployment convention. |
| 58 | |
| 59 | 403/404 and other non-429 4xx fail without retry. 429 and 5xx, timeouts and network |
| 60 | errors have bounded exponential backoff (1s, 2s, capped at 8s). Retry-After is |
| 61 | honored; when it exceeds 30 seconds this call fails without retrying early. |
| 62 | Cookies and request headers are never logged by this module. Results expose |
| 63 | HTTP status, identity, bounds, provenance, retrieval time, counts, date coverage, |
| 64 | row rejection reasons, and explicit acquisition/persistence failure reasons. |
| 65 | Failures never delete previously persisted evidence. |
| 66 | |
| 67 | ## Runtime persistence validation |
| 68 | |
| 69 | The same captured live response (no second history request) was parsed and four |
| 70 | bars were persisted/read back through the current ResearchRepository and |
| 71 | SqliteResearchPersistence at `.tmp/nse-runtime.sqlite`. DATE, OHLC range, integer |
| 72 | volume, provider, symbol, source URL and exact model roundtrip were checked. |
| 73 | The deployed research-engine does not yet contain DailyMarketBar, so deployed |
| 74 | PostgreSQL end-to-end validation was not possible without a separate deployment. |
| 75 | No deployment or migration was performed. The local runtime database is ignored |
| 76 | and is not part of the change. |