| 1 | # NSE canonical sector benchmarks and price history |
| 2 | |
| 3 | ## Identity and classification |
| 4 | |
| 5 | Baseline: `076215b5ee757061357aa6a71e5a6e85988d8baa` (OHLCV technical features). |
| 6 | The audited master contained 2,568 equities and no indices. Stock classification |
| 7 | comes from `portfolio.nifty500_universe.canonical_sector`, populated by |
| 8 | `Nifty500ReferenceService` and `NiftyIndustrySectorClassifier` from the official |
| 9 | Nifty 500 constituent file. The persisted source is `NSE_INDICES_NIFTY500`. |
| 10 | Portfolio/watchlist membership is irrelevant. |
| 11 | |
| 12 | `SECTOR_BENCHMARK_MAPPING_V1` resolves exact canonical classifications to platform |
| 13 | benchmark keys. `NSE_BENCHMARK_CATALOG_V1` records the authoritative registration |
| 14 | evidence. These are **price indices**, not total-return series. |
| 15 | |
| 16 | | Canonical classification | Platform key | Canonical UUID | Verified NSE request symbol | |
| 17 | |---|---|---|---| |
| 18 | | India broad market | INDIA_BROAD_PRICE | 56623ea0-224c-3481-b6d9-66c448b282a0 | NIFTY 500 | |
| 19 | | Technology | INDIA_TECHNOLOGY_PRICE | 4af7634a-1894-333d-ba68-a1c41a32c71c | NIFTY IT | |
| 20 | | Financials | INDIA_FINANCIALS_PRICE | 13ba7849-43e2-3e1e-a3f3-d93ee32d3d2f | NIFTY FINANCIAL SERVICES | |
| 21 | | Healthcare | INDIA_HEALTHCARE_PRICE | 055c4e9f-5592-3be6-8f87-3d05bb25b95a | NIFTY HEALTHCARE INDEX | |
| 22 | |
| 23 | UUIDs use Java `UUID.nameUUIDFromBytes(UTF8("aip:benchmark:" + platformKey))`; |
| 24 | the Python adapter implements the identical UUIDv3 operation. A calculated UUID |
| 25 | alone is **not** registration or identity evidence. Both acquisition and context |
| 26 | resolution require the actual master row and exactly one verified NSE mapping. |
| 27 | Provider symbols never identify stocks or benchmark instruments. |
| 28 | |
| 29 | The existing master service now registers `AssetType.INDEX` through explicit |
| 30 | `POST /api/v1/instruments/benchmarks/{benchmarkKey}/register`. It uses the existing |
| 31 | transaction/advisory-lock, master, and mapping repositories. Unknown keys and |
| 32 | conflicting identities fail. Repeating a registration is idempotent. The existing |
| 33 | VARCHAR asset-type column accepts INDEX; no migration is required or supplied. |
| 34 | `GET /api/v1/instruments/benchmarks` only reads registered identities and mappings. |
| 35 | It does not register instruments or acquire data. Authentication follows the |
| 36 | existing global-instrument controller convention. |
| 37 | |
| 38 | Acquisition requires ACTIVE / INDEX / IN / NSE / INR master metadata and exactly |
| 39 | one NSE mapping with VERIFIED status, matching exchange/currency/request symbol, |
| 40 | and catalog resolution source. Inactive, missing, conflicting, or untrusted |
| 41 | identity fails closed. Context construction also requires trusted India universe |
| 42 | membership before assigning the India broad benchmark. |
| 43 | |
| 44 | ## Mapping evidence and exclusions |
| 45 | |
| 46 | - [Nifty 500](https://www.niftyindices.com/indices/equity/broad-based-indices/nifty-500) |
| 47 | is the broad benchmark: its large/mid/small-cap coverage matches the platform's |
| 48 | existing Nifty 500 classification universe. There is no Nifty 50 fallback. |
| 49 | - [Nifty IT](https://www.niftyindices.com/indices/equity/sectoral-indices/nifty-it) |
| 50 | matches the classifier's Information Technology → Technology mapping. |
| 51 | - [Nifty Financial Services](https://www.niftyindices.com/indices/equity/sectoral-indices/nifty-financial-services) |
| 52 | covers banks and other financial services, matching Financial Services → |
| 53 | Financials. A bank-only index would exclude part of this classification. |
| 54 | - [Nifty Healthcare](https://www.niftyindices.com/indices/equity/sectoral-indices/nifty-healthcare-index) |
| 55 | matches Healthcare; a pharma-only substitution is not made. |
| 56 | |
| 57 | The official NSE `/api/allIndices` catalog and the captured history responses |
| 58 | confirmed these four index contracts on 2026-09-13. Fixtures retain those responses. |
| 59 | |
| 60 | Observed unmapped classifications: Communication Services (14), Consumer |
| 61 | Discretionary (88), Consumer Staples (28), Energy (17), Industrials (75), Materials |
| 62 | (55), Real Estate (11), Utilities (17); 17 records lacked classification. Mapped |
| 63 | counts were Financials 101, Healthcare 48, Technology 27. These counts describe |
| 64 | the audit snapshot, not an ongoing invariant. Broad compound classifications |
| 65 | are not assigned to narrower indices. Raw `FINANCIAL SERVICES`, `Bank`, and other |
| 66 | aliases are not accepted by the mapping adapter; classification must first come |
| 67 | from the canonical classifier. Duplicate classification evidence fails closed. |
| 68 | |
| 69 | ## Official history contract |
| 70 | |
| 71 | History version: `NSE_INDEX_HISTORY_V1`. |
| 72 | |
| 73 | `GET https://www.nseindia.com/api/historicalOR/indicesHistory` |
| 74 | with `indexType=<verified mapping>`, `from=DD-MM-YYYY`, `to=DD-MM-YYYY`. |
| 75 | Bootstrap: `https://www.nseindia.com/report-detail/eq_security`. |
| 76 | The older `/api/historical/indicesHistory` returned HTTP 200 HTML during discovery |
| 77 | and is deliberately not used. The equity security CSV endpoint is not reused. |
| 78 | |
| 79 | The index provider composes the existing NSE provider's transport/session, |
| 80 | browser headers, cookies, lock, request spacing and bounded retry behavior; it |
| 81 | does not call its equity identity gate or CSV parser. One transport/session is |
| 82 | reused sequentially per explicit benchmark population invocation and closed in |
| 83 | `finally`. There is no global cookie pool, parallel fetch, proxy, or bypass. |
| 84 | This small composition uses the existing private transport methods; a future |
| 85 | transport refactor must preserve the tested shared-session contract. |
| 86 | |
| 87 | JSON is `{ "data": [ ... ] }`. Required fields: |
| 88 | `EOD_INDEX_NAME`, `EOD_TIMESTAMP`, `EOD_OPEN_INDEX_VAL`, `EOD_HIGH_INDEX_VAL`, |
| 89 | `EOD_LOW_INDEX_VAL`, `EOD_CLOSE_INDEX_VAL`. |
| 90 | |
| 91 | Verified response-name aliases, confined to this parser: |
| 92 | |
| 93 | | Requested symbol | EOD_INDEX_NAME | |
| 94 | |---|---| |
| 95 | | NIFTY 500 | NIFTY 500 | |
| 96 | | NIFTY IT | NIFTY IT | |
| 97 | | NIFTY FINANCIAL SERVICES | NIFTY FIN SERVICE | |
| 98 | | NIFTY HEALTHCARE INDEX | NIFTY HEALTHCARE | |
| 99 | |
| 100 | Case/whitespace normalization is explicit; no other symbol equivalence is inferred. |
| 101 | `EOD_TIMESTAMP` (for example `11-SEP-2026`) is the exchange-local DATE. |
| 102 | `HI_TIMESTAMP` can be the previous UTC day and is never used for trading dates. |
| 103 | JSON decimals are decoded directly to Decimal; domain OHLC validation applies. |
| 104 | Missing fields, malformed values, out-of-window dates, duplicate dates, or wrong |
| 105 | index names reject the complete response before persistence. Empty responses and |
| 106 | HTTP 200 HTML are explicit failures, not zero-valued observations. |
| 107 | |
| 108 | `HIT_TRADED_QTY` and `HIT_TURN_OVER` were observed, but their index aggregation and |
| 109 | units are not normalized in this phase. Volume, turnover, and previous close |
| 110 | remain null. Actual OHLC values are index points; INR is the trusted index market |
| 111 | denomination, not an assertion that each point is a tradable currency amount. |
| 112 | |
| 113 | ## Population and persistence |
| 114 | |
| 115 | An explicit worker calls `IndiaMarketDataPopulationJobs.populate_benchmark_history` |
| 116 | with registered canonical IDs and one bounded inclusive start/end window. It |
| 117 | uses `ResearchRepository.upsert_daily_market_bars_async` via `persist_daily_result`. |
| 118 | Primary identity remains `(global_instrument_id, trading_date, provider='NSE')`. |
| 119 | Corrections update that key; another provider remains separate. No deletes, |
| 120 | fabricated OHLC, or close-history dual-writes occur. |
| 121 | |
| 122 | The existing `nse_historical_request_window_days` defaults to 30 inclusive |
| 123 | calendar days. This is an operational bound, **not an NSE guaranteed maximum**. |
| 124 | Invocations are capped by `market_data_population_batch_size`, ordered by UUID, |
| 125 | and persist per successful window. This explicit bounded operation does not |
| 126 | schedule a broad backfill or automatically extend lookbacks. |
| 127 | |
| 128 | Spacing uses `market_data_population_request_interval_seconds` (default 0.20). |
| 129 | The existing default two retries apply to transient network/5xx/429 failures; |
| 130 | ordinary 4xx, including 403, are not retried. Exponential backoff and Retry-After |
| 131 | handling are inherited. Failure cooldown uses the existing population cooldown; |
| 132 | 429 also inhibits subsequent instruments in the worker. Cooldowns are process |
| 133 | local, like the existing population jobs. Results reuse `NseHistoricalResult`, |
| 134 | including HTTP status, failure reason, accepted/rejected counts and persisted rows. |
| 135 | |
| 136 | ## Computation and batching |
| 137 | |
| 138 | Prepare contexts with |
| 139 | `await orchestrator.sector_benchmark_contexts(candidate_ids, identity_headers=...)`, |
| 140 | then pass them to `GlobalScanner.enrich_candidates(..., sector_contexts=contexts)`. |
| 141 | This keeps the existing explicit Stage-B dependency boundary. No dashboard, |
| 142 | watchlist, portfolio, scanner GET, or feature computation invokes acquisition. |
| 143 | |
| 144 | The context adapter reads the existing paged canonical universe and one benchmark |
| 145 | catalog response. Portfolio-service batches master and mapping reads. Stage B |
| 146 | collects distinct canonical stock/benchmark IDs and performs the existing bounded |
| 147 | close-history read plus one NSE daily-bar read, grouping in memory. With the |
| 148 | default batch bound, history SELECT counts are 0 for empty, 2 for one stock, |
| 149 | 2 for 18 same-sector stocks, and 2 for 18 stocks across the three mapped sectors. |
| 150 | Canonical metadata preparation is separate from these history query counts. |
| 151 | |
| 152 | The existing SectorRelativeStrengthEngine consumes persisted NSE daily closes |
| 153 | with DATE semantics, selecting a coherent provider series. Existing close-only |
| 154 | fallback remains supported. Stock short/stale OHLC fallback follows the existing |
| 155 | technical input policy; individual dates/providers are never spliced together. |
| 156 | |
| 157 | Lookbacks preserve the existing 5 / 21 / 63 / 126 **stock observations** for |
| 158 | 1W / 1M / 3M / 6M. Both exact stock endpoints must exist in the benchmark history. |
| 159 | Returns are `(end / start - 1) * 100`. Weekends, holidays, and missing intermediate |
| 160 | dates are not synthesized. No forward/back fill, nearest-date matching, or |
| 161 | calendar-session guessing is performed. A missing endpoint makes that horizon |
| 162 | unavailable; at least two horizons are needed for the existing score/state rules. |
| 163 | Thresholds, score weights, Technical Features V2 and Rule Engine V1 are unchanged. |
| 164 | |
| 165 | Snapshots expose mapping version, canonical benchmark IDs, history source and |
| 166 | explicit benchmark states: NO_SECTOR_CLASSIFICATION, UNMAPPED_SECTOR_BENCHMARK, |
| 167 | BENCHMARK_IDENTITY_UNAVAILABLE, BENCHMARK_HISTORY_UNAVAILABLE, |
| 168 | INSUFFICIENT_OVERLAP, STALE_BENCHMARK_HISTORY, AVAILABLE. Per-horizon diagnostics |
| 169 | remain available even when another horizon has overlap. Missing evidence is never |
| 170 | zero or a weak-sector conclusion. Existing seven-day freshness policy remains. |
| 171 | |
| 172 | ## Controlled runtime validation and limits |
| 173 | |
| 174 | One four-day Nifty 500 probe established the JSON contract. Subsequently just |
| 175 | four 30-day requests (2026-08-13 through 2026-09-11, one per registered index) |
| 176 | returned HTTP 200 and 22 rows each. No all-sector or all-equity backfill ran. |
| 177 | The captured responses were replayed locally without additional provider calls. |
| 178 | |
| 179 | The real master service registered/read back all four identities in the existing |
| 180 | H2 test schema; its public metadata fed the Python runtime. The existing repository |
| 181 | persisted 88 captured rows to local SQLite. Repeat upserts left 88 rows. Real stock |
| 182 | classification and close history were copied read-only from local PostgreSQL: |
| 183 | |
| 184 | | Stock | Verified canonical sector | Actual overlapping dates | Result | |
| 185 | |---|---|---:|---| |
| 186 | | HDFCBANK | Financials | 22 | Sector and market evidence available | |
| 187 | | SUNPHARMA | Healthcare | 22 | Sector and market evidence available | |
| 188 | | TCS | Technology | 22 | Sector and market evidence available | |
| 189 | | LT | Industrials | 22 with market | Sector explicitly unmapped | |
| 190 | |
| 191 | Repeated/reversed-input results matched, with networking blocked during compute. |
| 192 | The small benchmark capture supports short horizons only; 3M/6M formulas are |
| 193 | validated independently in tests, not claimed as available from this smoke. |
| 194 | These outputs are engineering diagnostics, not production investment conclusions. |
| 195 | |
| 196 | The deployed PostgreSQL instance lacks the pre-existing daily-bar table. No |
| 197 | migration was created/applied, and no benchmark was directly inserted into that |
| 198 | master by SQL. Deployment of the prior daily-bar schema plus this canonical |
| 199 | registration path remains necessary before deployed acquisition can work. |
| 200 | |
| 201 | Unresolved: unmapped sectors, deployed PostgreSQL validation, Global Opportunity |
| 202 | Ranker, short/long action model, recommendation history/lifecycle, news/macro, |
| 203 | prediction/backtesting. None is implemented by this slice. |
| 204 | |
| 205 | ## Validation results |
| 206 | |
| 207 | Final research-engine suite: **992 passed**, one existing dependency deprecation |
| 208 | warning. Included: 64 new benchmark tests, 21 sector-relative tests, 25 technical |
| 209 | tests, 42 OHLCV tests, 35 daily-bar persistence tests, 45 backfill tests, 55 NSE |
| 210 | equity provider tests, 22 population tests, 31 Phase-1/batch scanner tests, and |
| 211 | 29 Rule Engine V1 tests. |
| 212 | |
| 213 | Java benchmark registration/instrument-master tests: 19 passed, including the |
| 214 | H2 registration/read-back integration test. The broader portfolio reactor run |
| 215 | reported 194 tests with two failures: `CanonicalIdentityBootstrapTest` assumes |
| 216 | an empty shared H2 database (passes when rerun alone), and |
| 217 | `AppUserProvisionerTest` expects an obsolete conflict target/SQL shape. Their |
| 218 | implementation/test files are unchanged by this phase; no unrelated fixes were |
| 219 | included. The benchmark H2 test rolls its writes back after exporting public |
| 220 | metadata for the runtime smoke. |
| 221 | |
| 222 | `git diff --check` passed. No changes to Technical Features V2, Rule Engine V1, |
| 223 | NSE equity provider/backfill, frontend, or migrations. `platform.ps1` is unchanged |
| 224 | by this phase (phase-start SHA256 |
| 225 | `4577DA738957A6C35A8035812586344E76A557DFE41A7355F5A4D5F3E9D2E3AF`). |
| 226 | Nothing staged, committed, or pushed. `smtp.password` was not opened. |