| 1 | # Phase 5E scalable multi-broker architecture |
| 2 | |
| 3 | ## 1. Final architecture |
| 4 | |
| 5 | Phase 5E separates authorization, connection state, ownership synchronization, persisted portfolios, and market valuation. |
| 6 | |
| 7 | ```mermaid |
| 8 | flowchart LR |
| 9 | UI[Portfolio UI] --> D[Broker discovery and auth actions] |
| 10 | UI --> P[Persisted portfolio dashboard] |
| 11 | D --> B[BrokerConnectionService] |
| 12 | B --> BP[BrokerProvider / BrokerConnector] |
| 13 | UI --> S[Portfolio broker-sync orchestrator] |
| 14 | S --> B |
| 15 | S --> DB[(Portfolio PostgreSQL)] |
| 16 | P --> DB |
| 17 | P --> M[MarketDataProvider and quote cache] |
| 18 | ``` |
| 19 | |
| 20 | Portfolio data loads from PostgreSQL without a live broker session. A broker session is needed only to synchronize ownership and cash. Quote retrieval is a separate read through the existing market-data provider/cache; broker APIs are not the default quote source. |
| 21 | |
| 22 | ## 2. Authentication models |
| 23 | |
| 24 | `BrokerAuthenticationModel` is independent from capabilities: |
| 25 | |
| 26 | | Model | Meaning | |
| 27 | |---|---| |
| 28 | | `LOCAL_GATEWAY` | A user-scoped local connector or gateway, currently IBKR | |
| 29 | | `INDIVIDUAL_API_CREDENTIALS` | User-owned API application credentials, currently retail Breeze | |
| 30 | | `PLATFORM_OAUTH` | Future platform application with per-user authorization | |
| 31 | | `PARTNER_OAUTH` | Future negotiated broker partnership | |
| 32 | | `CONSENT_AGGREGATOR` | Future regulated consent/data aggregation | |
| 33 | | `UNSUPPORTED` | No verified authentication contract | |
| 34 | |
| 35 | Assignments: IBKR is `LOCAL_GATEWAY`; ICICI Direct retail Breeze and HDFC InvestRight Individual API are `INDIVIDUAL_API_CREDENTIALS`. HDFC remains publicly `UNSUPPORTED` for the desired consumer flow because no public multi-customer Partner API contract is available. |
| 36 | |
| 37 | ### Verified authentication contracts (reviewed 2026-08-29) |
| 38 | |
| 39 | | Broker | Official product | Model | Credential ownership / multi-user support | Login and return | Token/session behavior | Read APIs | Quota | Consumer UX readiness | |
| 40 | |---|---|---|---|---|---|---|---|---| |
| 41 | | Interactive Brokers | Client Portal Gateway | `LOCAL_GATEWAY` | User-scoped Gateway runtime; the platform persists only its own connector identity | Gateway supplies its login URL; authentication occurs in the official IBKR UI | Runtime reports authenticated/expired state; re-authentication reuses the same connector ID | Accounts, positions and cash implemented read-only | No new quota assumption | **Supported in DEV** | |
| 42 | | ICICI Direct | Breeze retail API | `INDIVIDUAL_API_CREDENTIALS` | The customer registers an app and owns its AppKey/secret. No public contract permits one platform app for unrelated customers | ICICI-hosted login accepts AppKey and returns to a registered redirect with `API_Session`; no documented `state` contract was found | Customer Details exchanges `API_Session` for a signed-request session token; public refresh/lifetime contract is incomplete, so re-login is required when invalid | Customer Details, Demat Holdings and Funds are documented and already available through inspection APIs | 100 calls/minute and 5,000/day; official documentation does not define the scope key | **Blocked pending Partner API contract** | |
| 43 | | HDFC Securities | InvestRight Individual API | `INDIVIDUAL_API_CREDENTIALS` | Each account holder creates an app/API key and secret. HDFC’s official support states these are exclusive to that account; Partner APIs require commercial contact | Individual API documents frontend/API access-token acquisition and a registered redirect URL, but this is not a public platform-wide authorization grant; public `state`/CSRF terms were not verified | Access token is used with API key; a scalable refresh/revocation contract for unrelated customers is not public | Profile, holdings/portfolio, positions, and funds/margins are documented | No authoritative production read quota was found | **Blocked pending Partner API contract** | |
| 44 | |
| 45 | Official evidence: |
| 46 | |
| 47 | - ICICI Direct: [Breeze API reference](https://api.icicidirect.com/breezeapi/documents/index.html), including registration, redirect login, Customer Details exchange, signing headers, holdings/funds endpoints, and published rate limits. |
| 48 | - HDFC Securities: [InvestRight Open API portal](https://developer.hdfcsec.com/), [Individual API announcement and credential setup](https://www.hdfcsec.com/blog/details/introducing-individual-api-for-investright-clients), and [InvestRight API documentation](https://developer.hdfcsec.com/ir-docs/docs/intro). HDFC’s official InvestRight community response states individual keys are exclusive to their account holder and directs platform integrations to Partner APIs. |
| 49 | |
| 50 | The words OAuth or redirect in an individual product do not establish delegated multi-customer authorization. Platform-owned credentials, callback parameters, CSRF/state guarantees, scopes, token lifetimes, refresh behavior, quotas, and partner onboarding must come from the applicable written provider contract before either provider becomes connectable. |
| 51 | |
| 52 | ## 3. Provider capability model |
| 53 | |
| 54 | Capabilities continue to describe implemented operations, not authentication style. `ORDER_EXECUTION` is filtered and remains prohibited by the shared capability value object. ICICI retains verified account, Demat Holdings, and funds inspection behavior, while `PORTFOLIO_READ` remains absent because its normalized holdings are not valuation-complete. HDFC advertises no capabilities. |
| 55 | |
| 56 | ## 4. Connection lifecycle |
| 57 | |
| 58 | The existing shared lifecycle covers `AUTHENTICATION_REQUIRED`, `CONNECTING`, `CONNECTED`, `SYNCING`, `ERROR`, and `DISCONNECTED`. Generic discovery exposes a safe descriptor. A connection-scoped authentication-action endpoint normalizes `NONE`, `REDIRECT_REQUIRED`, `POPUP_REQUIRED`, `USER_CREDENTIALS_REQUIRED`, `CONSENT_REQUIRED`, `AUTHENTICATION_REQUIRED`, `UNAVAILABLE`, and `UNSUPPORTED` without returning credentials, cookies, or raw tokens. |
| 59 | |
| 60 | ```mermaid |
| 61 | stateDiagram-v2 |
| 62 | [*] --> AUTHENTICATION_REQUIRED: start connection |
| 63 | AUTHENTICATION_REQUIRED --> CONNECTING: user follows official action |
| 64 | CONNECTING --> CONNECTED: provider confirms session |
| 65 | CONNECTED --> SYNCING: explicit Sync |
| 66 | SYNCING --> CONNECTED: complete snapshot persisted |
| 67 | SYNCING --> AUTHENTICATION_REQUIRED: session expired |
| 68 | SYNCING --> ERROR: provider failure |
| 69 | CONNECTED --> DISCONNECTED: explicit disconnect |
| 70 | ``` |
| 71 | |
| 72 | ## 5. Portfolio auto-create and reuse |
| 73 | |
| 74 | The stable broker-backed portfolio key is: |
| 75 | |
| 76 | `authenticated user + broker connection + provider broker account` |
| 77 | |
| 78 | The display name is presentation only. A unique database index on `(user_id, broker_connection_id, broker_account_id)` protects concurrent creation. A synchronization creates one portfolio for each broker account, or reuses the matching portfolio. Reconnection/synchronization never selects a portfolio by name. |
| 79 | |
| 80 | ## 6. Legacy adoption |
| 81 | |
| 82 | Existing portfolios are preserved. A legacy portfolio is adopted only when all existing positions unambiguously identify the same connection, broker account, and provider. Zero or multiple candidates are not inferred. Existing Phase 5C position-level adoption remains ownership-scoped. |
| 83 | |
| 84 | ## 7. Synchronization lifecycle |
| 85 | |
| 86 | `POST /api/v1/portfolios/broker-connections/{connectionId}/sync`: |
| 87 | |
| 88 | 1. derives the application user from trusted authentication headers; |
| 89 | 2. asks broker-service to synchronize/check the owned connection; |
| 90 | 3. requires `CONNECTED` and `PORTFOLIO_READ`; |
| 91 | 4. obtains one complete authoritative snapshot before mutation; |
| 92 | 5. partitions accounts, holdings, and cash by broker account; |
| 93 | 6. creates or reuses the source portfolio; |
| 94 | 7. applies Phase 5C provider-native upsert and sync-generation semantics; |
| 95 | 8. marks missing positions inactive only inside the completed source snapshot; |
| 96 | 9. records the successful broker-sync timestamp; |
| 97 | 10. returns the affected portfolios. |
| 98 | |
| 99 | The operation is idempotent. An in-process connection lock prevents duplicate clicks in one instance, a broker-service pessimistic row lock serializes a connection across requests, and database uniqueness is the final race guard. |
| 100 | |
| 101 | ## 8. Failure semantics |
| 102 | |
| 103 | Provider/authentication failure occurs before portfolio mutation. A missing, malformed, or incomplete snapshot is an error, never an empty successful portfolio. Transaction rollback protects against persistence failures. Existing holdings and last successful sync remain visible. Only a successfully received authoritative empty account snapshot can inactivate positions within that exact source. |
| 104 | |
| 105 | ## 9. Market data versus broker sync |
| 106 | |
| 107 | Broker sync answers ownership and cash. Market data answers valuation. Dashboard and position reads use the existing `MarketDataProvider`, which can use the existing quote cache, independently of broker sessions. Broker freshness (`lastBrokerSyncAt`) and quote freshness (`quote.sourceTimestamp`, freshness/source) are separately rendered. No automatic broker sync occurs at application login. |
| 108 | |
| 109 | ## 10. Currency aggregation |
| 110 | |
| 111 | `GET /api/v1/portfolios/dashboard` returns `currencyTotals` keyed by ISO currency. Aggregation uses Java `BigDecimal` and only adds values within the same currency bucket. No implicit FX conversion or single global total is produced. Portfolios with unknown/incomplete valuation are explicitly listed in `incompleteValuationPortfolioIds`. |
| 112 | |
| 113 | ## 11. Frontend flow |
| 114 | |
| 115 | - On application login, load `/api/v1/portfolios/dashboard` immediately. |
| 116 | - Render an `All` tab plus one tab for every persisted portfolio. |
| 117 | - Render same-currency totals separately and all source-scoped holdings. |
| 118 | - Populate the broker selector from `/api/v1/brokers`. |
| 119 | - Start a connection through the generic connect operation, then obtain its normalized authentication action. |
| 120 | - Synchronously open a blank popup in the user click handler, before awaiting the connection/authentication API, so browser popup policy preserves the user gesture. |
| 121 | - Navigate that same popup only for `REDIRECT_REQUIRED` or `POPUP_REQUIRED`; otherwise close it and render the normalized provider message. |
| 122 | - A blocked popup, missing authentication URL, or API failure is reported safely and never hides the persisted portfolio. |
| 123 | - On Sync, call the generic portfolio synchronization endpoint; if authentication is required, obtain the provider action and resume after authentication. |
| 124 | - Disconnecting a connection does not delete its persisted portfolio. |
| 125 | |
| 126 | ## 12. IBKR migration |
| 127 | |
| 128 | IBKR remains behind `BrokerConnector` and its existing Client Portal Gateway implementation. The generic action delegates to its scoped connector login URL; no connector ID is needed by the UI. The generic portfolio sync consumes the established account/position/cash snapshot and creates or reuses account portfolios. No IBKR authentication algorithm was changed. |
| 129 | |
| 130 | ## 13. ICICI limitations |
| 131 | |
| 132 | Retail Breeze is accurately described as individual API credentials. Its provider and inspection endpoints remain. The generic portfolio synchronizer refuses it while `PORTFOLIO_READ` is absent, so incomplete Demat records cannot weaken portfolio constraints or fabricate price, cost, exchange, or currency. Commercial partner authentication remains a future phase. |
| 133 | |
| 134 | ## 14. HDFC integration point |
| 135 | |
| 136 | `HDFC_SECURITIES` exists only as discoverable, non-connectable metadata with no advertised capabilities. The verified InvestRight Individual API requires account-holder-owned credentials and therefore is not wired into the normal consumer flow. A future Partner API adapter must be based on HDFC's written multi-customer authorization, callback/state, token, quota, and read-scope contract. There are no guessed URLs or API fields. |
| 137 | |
| 138 | ## 15. Security and tenancy |
| 139 | |
| 140 | All controllers derive the user from trusted authenticated context. Broker-service validates connection ownership before status, authentication action, sync, snapshot, and disconnect. Portfolio repositories require user ownership; source portfolio adoption additionally checks provider/connection/account identity. Secrets and sessions remain broker-service concerns and are never stored in portfolio-service or returned in dashboard responses. Generic connection responses also omit user IDs, connector IDs, external account references, and internal session references. |
| 141 | |
| 142 | ## 16. Observability |
| 143 | |
| 144 | Synchronization logs provider, connection ID, result, portfolio count, position count, duration, and safe error category. It does not log credentials, authentication headers, cookies, sessions, or broker payloads. |
| 145 | |
| 146 | ## 17. Database change |
| 147 | |
| 148 | Flyway `V10__broker_backed_portfolio_identity.sql` adds nullable source identity and broker-sync freshness columns to `portfolios`, a unique source index, and an ownership/connection lookup index. V1-V9 are unchanged. Existing data remains valid because all new columns are nullable. |
| 149 | |
| 150 | ## 18. API contracts |
| 151 | |
| 152 | - `GET /api/v1/brokers`: consumer-safe descriptors including display name, connectability, reason, and status. Authentication models and raw capabilities remain internal domain/provider concerns. |
| 153 | - `POST /api/v1/broker-connections/{provider}/connect`: creates an owned connection using the provider implementation. |
| 154 | - `GET /api/v1/broker-connections/{id}/authentication-action`: safe normalized next action. |
| 155 | - `GET /api/v1/broker-connections/{id}/status`: owned connection status. |
| 156 | - `POST /api/v1/portfolios/broker-connections/{id}/sync`: connection sync plus portfolio create/reuse/import. |
| 157 | - `DELETE /api/v1/broker-connections/{id}`: disconnect without portfolio deletion. |
| 158 | - `GET /api/v1/portfolios/dashboard`: persisted portfolios, currency totals, all holdings, and incomplete-valuation markers. |
| 159 | |
| 160 | Provider-specific ICICI inspection/session endpoints and connector-specific compatibility endpoints remain available but are no longer required by the generic UI. |
| 161 | |
| 162 | ## 19. Test coverage |
| 163 | |
| 164 | Coverage includes broker descriptors/authentication models, unavailable HDFC, generic IBKR authentication actions and cross-user denial, connection locking, portfolio create/reuse, multiple accounts, repeated snapshot idempotency, source-scoped missing-position handling, failure preservation, cash, deterministic currency buckets, Flyway V10, and all earlier IBKR/ICICI/Phase 5C tests. |
| 165 | |
| 166 | ## 20. Known gaps and future phases |
| 167 | |
| 168 | - Live IBKR validation requires an already authenticated DEV session; automated tests remain mock-bound. |
| 169 | - Durable distributed sync coordination beyond database row/index protection may be added if sync becomes asynchronous. |
| 170 | - A production ICICI route awaits a verified partner/commercial contract. |
| 171 | - ICICI and HDFC require commercial Partner API onboarding and written multi-customer authentication contracts. Their individual developer credentials are deliberately not requested in the consumer UI. |
| 172 | - Incomplete ICICI/HDFC ownership data is not persisted as a fully valued `PortfolioPosition`. A future holdings-only model should use a separate deliberate representation with nullable valuation and explicit provenance rather than relaxing Phase 5C constraints. |
| 173 | - Market-data coverage remains limited to existing providers/cache; unsupported instruments display last-known broker valuation with explicit freshness. |
| 174 | - A regulated Account Aggregator/FIU route and optional explicit FX view are separate future work. |
| 175 | |
| 176 | ## 21. Phase 5E.1 UX and legacy-adoption correction |
| 177 | |
| 178 | The consumer broker page now exposes only broker name, friendly connection state, a short explanation, linked portfolio name, and Connect/Re-authenticate/Manage/Disconnect actions. Demo Broker is excluded from public discovery unless `broker.public-demo-enabled=true`; its compatibility endpoint remains available for tests. ICICI Direct and HDFC Securities remain cleanly unavailable. Authentication-model and capability enum lists are no longer part of public discovery/authentication-action DTOs. |
| 179 | |
| 180 | An expired IBKR session is returned as normalized `AUTHENTICATION_REQUIRED` rather than thrown as a generic provider error. Portfolio Sync uses the connection-level authentication action and can be retried after sign-in. Broker-page Sync was removed; only a linked broker-backed portfolio renders Sync. |
| 181 | |
| 182 | On persisted portfolio listing, portfolio-service performs an idempotent, authentication-independent legacy adoption when every position is real broker data and resolves to exactly one Phase 5C `(connection, provider, broker account)` source. It attaches the existing portfolio row to that identity without changing position IDs, quantities, source identity, activity, or sync generations. Multiple sources are logged as ambiguous and left untouched. V10 already supplies the necessary columns and unique index, so no V11 migration is required. |
| 183 | |
| 184 | The gateway narrowly forwards `/`, `/_next/**`, `/favicon.ico`, and `/robots.txt` to the frontend service. Existing `/api/**` routes retain their current backend routing and authentication behavior. |