@cryptotaxi247 / CoPilot / commits / 57e599ad

docs: add AI-agent architecture + extension docs (#693)

* docs: add AI-agent architecture and extension docs * docs: add database schema map for AI agents * docs: add critical relationship graph and SIEM query notes * docs: note tag access enforcement locations * precommit fixes --------- Co-authored-by: Clawdbot <clawdbot@Clawdbots-Mac-mini.local> Co-authored-by: taylorwalton <taylor.walton@socfortress.co>

taylorcopilot committed Feb 14, 2026 at 10:19 UTC 57e599add7a6e566f44768963567d36d2264d6f6
7 files changed +997
docs/README.md new
+48
@@ -0,0 +1,48 @@
1 +# AI Agent Docs Entrypoint
2 +
3 +This folder is the **starting point for AI agents** making code changes in this repo.
4 +
5 +## Read This First
6 +
7 +1. `docs/architecture/ARCHITECTURE.md` (existing system architecture)
8 +2. `docs/architecture/MAP.md` (existing system map)
9 +3. `docs/architecture/DEPLOYMENT.md` (runtime stack, ports, env, persistence)
10 +4. `docs/architecture/DATA_FLOWS.md` (critical request + job + data flows)
11 +5. `docs/integrations/ADDING_A_CONNECTOR.md` (exact connector extension checklist)
12 +
13 +## Fast Repo Orientation (Code Pointers)
14 +
15 +- Backend entrypoint: `backend/copilot.py`
16 +- Backend router registration: `backend/copilot.py`, `backend/app/routers/*.py`
17 +- DB bootstrap + seed logic: `backend/app/db/db_setup.py`, `backend/app/db/db_populate.py`
18 +- Connector registry + verify dispatch: `backend/app/connectors/services.py`
19 +- Connector DB lookup helper: `backend/app/connectors/utils.py`
20 +- Auth routes + JWT handling: `backend/app/auth/routes/auth.py`, `backend/app/auth/utils.py`
21 +- Scheduler init + job mapping: `backend/app/schedulers/scheduler.py`
22 +- Scheduler APIs: `backend/app/schedulers/routes/scheduler.py`
23 +- Incident alert/case paths: `backend/app/incidents/routes/incident_alert.py`, `backend/app/incidents/routes/db_operations.py`, `backend/app/incidents/services/db_operations.py`
24 +- MinIO data store operations: `backend/app/data_store/data_store_operations.py`, `backend/app/data_store/data_store_setup.py`, `backend/app/data_store/data_store_session.py`
25 +- Frontend API client base: `frontend/src/api/httpClient.ts`
26 +- Frontend connector endpoints + UI: `frontend/src/api/endpoints/connectors.ts`, `frontend/src/views/Connectors.vue`, `frontend/src/components/connectors/*`
27 +
28 +## Agent Guardrails
29 +
30 +- Prefer editing the smallest set of files that completes the task.
31 +- When adding backend functionality, wire all layers explicitly:
32 + 1. service/util
33 + 2. route
34 + 3. router include
35 + 4. top-level include in `backend/copilot.py`
36 +- When adding user-facing connector features, update both:
37 + - backend endpoint(s)
38 + - frontend endpoint wrapper + UI wiring
39 +- Verify any change affecting auth, scheduler, or storage paths with focused tests/manual API calls.
40 +
41 +## Common Starting Commands
42 +
43 +```bash
44 +rg --files backend/app
45 +rg --files frontend/src
46 +rg -n "include_router|APIRouter" backend/app
47 +rg -n "connector|verify" backend/app/connectors frontend/src
48 +```
docs/architecture/ARCHITECTURE.md new
+226
@@ -0,0 +1,226 @@
1 +# SOCFortress CoPilot Architecture
2 +
3 +## Overview
4 +CoPilot is a “single pane of glass” security operations platform. It centralizes data from Wazuh, Graylog, Velociraptor, Grafana, InfluxDB, and other tools, provides alert/case management, and adds automation such as scheduled collectors, active response, and report generation. The backend is a FastAPI service with a MySQL database and MinIO object storage. The UI is a Vue 3 SPA, with an optional customer-facing portal.
5 +
6 +Key entry points:
7 +- Backend runtime: `backend/copilot.py`
8 +- Frontend app: `frontend/src`
9 +- Customer portal app: `customer_portal/src`
10 +
11 +## Tech Stack
12 +
13 +### Backend
14 +- FastAPI, Starlette, Uvicorn: API server and routing. `backend/copilot.py`
15 +- SQLModel + SQLAlchemy: ORM and DB access. `backend/app/db`
16 +- Alembic: DB migrations. `backend/alembic`
17 +- APScheduler: background job scheduling. `backend/app/schedulers/scheduler.py`
18 +- MySQL: primary database. `docker-compose.yml`, `backend/app/db/db_session.py`
19 +- MinIO: object storage for case/report artifacts, sysmon configs, Velociraptor artifacts. `backend/app/data_store`
20 +- Loguru: logging. `backend` modules
21 +- Requests/HTTPX: integrations with external systems. `backend/app/connectors/*/utils`, `backend/app/integrations/*`
22 +
23 +### Frontend
24 +- Vue 3 + TypeScript + Vite: SPA. `frontend/package.json`, `frontend/src`
25 +- Pinia: state management. `frontend/src/stores`
26 +- Naive UI + Tailwind CSS: UI components and styling. `frontend/package.json`, `frontend/tailwind.config.js`
27 +- Axios: API client. `frontend/src/api`
28 +- Cypress + Vitest: tests. `frontend/cypress`, `frontend/vitest.config.ts`
29 +
30 +### Customer Portal (Optional)
31 +- Vue 3 + Vite + Pinia + Naive UI. `customer_portal/package.json`, `customer_portal/src`
32 +
33 +## Directory Layout & Responsibilities
34 +
35 +### Root
36 +- `docker-compose.yml`: default deployment stack (backend, frontend, MySQL, MinIO, MCP, Nuclei).
37 +- `.env.example`: required runtime configuration for connectors and services.
38 +- `build-dockers.sh`: build utility script.
39 +
40 +### Backend
41 +- `backend/copilot.py`: application entrypoint, API router mount, startup/shutdown hooks.
42 +- `backend/settings.py`: local env loading (not the primary runtime env in Docker).
43 +- `backend/app/routers/*`: API route modules for each feature/integration.
44 +- `backend/app/auth/*`: authentication, user/role management.
45 +- `backend/app/db/*`: sessions, migrations, bootstrapping, and data seeding.
46 +- `backend/app/schedulers/*`: APScheduler instance, scheduled jobs, job metadata models.
47 +- `backend/app/connectors/*`: connectors for platform services (Wazuh, Graylog, Grafana, etc.).
48 +- `backend/app/integrations/*`: third‑party integrations and per‑customer integration config.
49 +- `backend/app/network_connectors/*`: “network connectors” with customer‑scoped auth keys and configs.
50 +- `backend/app/customer_provisioning/*`: provisioning workflows for external services (Graylog, Grafana, Portainer, Wazuh manager).
51 +- `backend/app/stack_provisioning/graylog/*`: content packs, pipelines, streams, and input templates for Graylog.
52 +- `backend/app/agents/*`: Wazuh and Velociraptor agent management, SCA, vulnerabilities.
53 +- `backend/app/incidents/*`: incident management (cases, alerts, tags, reports).
54 +- `backend/app/active_response/*`: automation scripts and invoke endpoints.
55 +- `backend/app/data_store/*`: MinIO storage (cases, templates, artifacts).
56 +- `backend/app/threat_intel/*`: EPSS, VirusTotal, SOCFortress threat intel.
57 +- `backend/app/integrations/copilot_mcp/*`: local and cloud MCP queries.
58 +
59 +### Frontend
60 +- `frontend/src/router/index.ts`: primary navigation and feature routes.
61 +- `frontend/src/api/endpoints/*`: typed API clients for each backend domain.
62 +- `frontend/src/components/*`: feature UI modules (alerts, cases, agents, connectors, integrations, etc.).
63 +- `frontend/src/views/*`: route views; corresponds closely to backend feature areas.
64 +- `frontend/.env.example`: API base URL and UI behavior.
65 +
66 +### Customer Portal
67 +- `customer_portal/src/router/index.ts`: simple login + alerts/cases/agents views.
68 +- `customer_portal/src/views/*`: customer‑limited UI.
69 +
70 +## Core Runtime Flows
71 +
72 +### Startup
73 +1. `backend/copilot.py` loads env vars and initializes FastAPI.
74 +2. On startup event:
75 + - Creates MySQL database and user if needed. `backend/app/db/db_setup.py`
76 + - Applies Alembic migrations. `backend/app/db/db_setup.py`
77 + - Creates MinIO buckets. `backend/app/data_store/data_store_setup.py`
78 + - Seeds connectors, roles, available integrations, available network connectors. `backend/app/db/db_setup.py`, `backend/app/db/db_populate.py`
79 + - Ensures admin and scheduler users. `backend/app/db/db_setup.py`
80 + - Initializes APScheduler and schedules enabled jobs. `backend/app/schedulers/scheduler.py`
81 +3. Static mount: `scoutsuite-report` directory for cloud security assessment outputs. `backend/copilot.py`
82 +
83 +### Auth & Authorization
84 +- JWT auth with OAuth2 password flow. `backend/app/auth/utils.py`
85 +- Roles/scopes: `admin`, `analyst`, `scheduler`, `customer_user`.
86 +- Routes use `Security(AuthHandler().get_current_user, scopes=[...])` or `require_any_scope(...)`.
87 +
88 +### API Routing
89 +- `FastAPI` app mounts a single APIRouter at `/api`.
90 +- Each domain lives in `backend/app/routers/*.py` and delegates to feature modules in `backend/app/<domain>/*`.
91 +- Frontend API clients match route structure. `frontend/src/api/endpoints/*`
92 +
93 +### Background Jobs
94 +- APScheduler runs inside the FastAPI app.
95 +- Job metadata is stored in MySQL and loaded on startup. `backend/app/schedulers/scheduler.py`
96 +- Jobs invoke integration collectors and internal maintenance:
97 + - Agent sync (Wazuh + Velociraptor)
98 + - Wazuh index resize
99 + - Alert creation collection
100 + - Snapshot schedule execution
101 + - Integration collectors (Duo, Darktrace, Cato, Huntress, Carbon Black, etc.)
102 +- Job definitions live in `backend/app/schedulers/services/*`.
103 +
104 +### Integrations & Connectors
105 +- Connectors are core service connections (Wazuh, Graylog, Grafana, etc.) stored in `Connectors` DB table.
106 +- Connectors are seeded from env vars on startup. `backend/app/db/db_populate.py`
107 +- Each connector implements a verify function in `backend/app/connectors/<service>/utils/universal.py` and is mapped in `backend/app/connectors/services.py`.
108 +- Integration settings are customer‑scoped and stored in `CustomerIntegrations` / `CustomerIntegrationsMeta` tables.
109 +
110 +### Data Storage
111 +- MySQL for all operational data (users, integrations, connectors, alerts/cases, scheduler metadata).
112 +- MinIO for case artifacts, report templates, sysmon configs, Velociraptor artifacts. `backend/app/data_store`
113 +- Local filesystem: `scoutsuite-report` directory for report outputs.
114 +
115 +## Major User‑Facing Features (Admin UI)
116 +- Overview dashboard and system health.
117 +- Connectors management (test/verify/update).
118 +- Wazuh management: rules, groups, Sysmon config, MITRE browsing.
119 +- Graylog management, metrics, pipelines, streams, inputs.
120 +- Alerts and SIEM views, MITRE/Atomic Red Team views.
121 +- Incident management: sources, alerts, cases, tags, comments, reports, data store.
122 +- Agents (Wazuh + Velociraptor), SCA, vulnerabilities, data store.
123 +- Artifacts and file collection.
124 +- Active response actions.
125 +- External services: third‑party integrations and network connectors.
126 +- Reporting (Grafana dashboards, case reports, vuln/SCA reports).
127 +- Scheduler management.
128 +- Cloud security assessment (ScoutSuite).
129 +- Web vulnerability assessment (Nuclei).
130 +- GitHub Audit.
131 +- Customer portal branding/settings.
132 +- License management.
133 +
134 +## Customer Portal Features
135 +- Login and role‑restricted access for `customer_user`.
136 +- Views for alerts, cases, case details, and agents.
137 +- Separate SPA served by `copilot-customer-portal` container.
138 +
139 +## Integrations & Open Source Services
140 +
141 +### Wazuh
142 +- Connector: `backend/app/connectors/wazuh_manager/*` and `backend/app/connectors/wazuh_indexer/*`
143 +- Auth: API token cached in memory with TTL; requests via `requests`.
144 +- Wazuh manager routes: `backend/app/connectors/wazuh_manager/routes/*` and `backend/app/routers/wazuh_manager.py`.
145 +- Wazuh indexer routes: `backend/app/connectors/wazuh_indexer/routes/*`.
146 +
147 +### Graylog
148 +- Connector: `backend/app/connectors/graylog/*`
149 +- Event shipper: GELF TCP to Graylog input. `backend/app/connectors/event_shipper/*`, `backend/app/integrations/utils/event_shipper.py`
150 +- Graylog provisioning: content packs, pipelines, streams, inputs. `backend/app/stack_provisioning/graylog/*`
151 +- Graylog API management used in routes and service modules.
152 +
153 +### Grafana
154 +- Connector: `backend/app/connectors/grafana/*`
155 +- Reporting endpoints for orgs/dashboards/panels and iframe generation. `backend/app/routers/grafana.py`, `frontend/src/components/reportCreation`
156 +
157 +### Velociraptor
158 +- Connector: `backend/app/connectors/velociraptor/*`
159 +- Agent management + artifacts; artifacts also stored in MinIO. `backend/app/agents/velociraptor/*`, `backend/app/data_store`
160 +
161 +### Shuffle
162 +- Connector: `backend/app/connectors/shuffle/*`
163 +- Endpoints for Shuffle metadata. `backend/app/routers/shuffle.py`
164 +
165 +### InfluxDB
166 +- Connector: `backend/app/connectors/influxdb/*`
167 +- Healthcheck and monitoring endpoints. `backend/app/routers/influxdb.py`, `frontend/src/components/healthcheck`
168 +
169 +### Portainer
170 +- Connector: `backend/app/connectors/portainer/*`
171 +- Customer provisioning workflows can call Portainer. `backend/app/customer_provisioning/services/portainer.py`
172 +
173 +### Nuclei
174 +- Integration: `backend/app/integrations/nuclei/*` and `backend/app/routers/nuclei.py`
175 +- Container present in `docker-compose.yml` as `copilot-nuclei-module`.
176 +
177 +### ScoutSuite
178 +- Integration: `backend/app/integrations/scoutsuite/*` and `backend/app/routers/scoutsuite.py`
179 +- Outputs served from `scoutsuite-report` static mount.
180 +
181 +### Threat Intel
182 +- VirusTotal, EPSS, SOCFortress. `backend/app/threat_intel/*`
183 +
184 +### MCP (CoPilot AI)
185 +- Local MCP service container configured in Docker.
186 +- Backend routes call `backend/app/integrations/copilot_mcp/services/copilot_mcp.py` to query local OpenSearch/MySQL/Wazuh/Velociraptor or cloud threat intel endpoints.
187 +
188 +## Configuration Management & Secrets
189 +- `.env` provides connector URLs, API keys, DB creds, MinIO creds, MCP settings. `.env.example` documents expected values.
190 +- Frontend uses `VITE_API_URL` to target the backend.
191 +- TLS handled by `copilot-frontend` container; TLS cert/key paths are configurable in `docker-compose.yml` and documented in README.
192 +
193 +## Deployment (Docker Compose)
194 +- `copilot-backend`: FastAPI service on port 5000.
195 +- `copilot-frontend`: Vue app with TLS, ports 80/443.
196 +- `copilot-mysql`: MySQL 8.
197 +- `copilot-minio`: object storage.
198 +- `copilot-nuclei-module`: web vulnerability scanner module.
199 +- `copilot-mcp`: MCP service for AI queries.
200 +- Optional `copilot-customer-portal` for customer‑facing UI.
201 +
202 +## Extension Points
203 +
204 +### Add a New Connector
205 +1. Add connector metadata in `backend/app/db/db_populate.py`.
206 +2. Add verification logic in `backend/app/connectors/<new_service>/utils/universal.py`.
207 +3. Map the connector name in `backend/app/connectors/services.py`.
208 +4. Provide routes in `backend/app/connectors/<new_service>/routes` and `backend/app/routers/<new_service>.py` as needed.
209 +5. Add frontend UI and API client in `frontend/src/components` and `frontend/src/api/endpoints`.
210 +
211 +### Add a New Integration (Per‑Customer)
212 +1. Add integration metadata in `backend/app/db/db_populate.py`.
213 +2. Add models/schema in `backend/app/integrations/models` and `backend/app/integrations/schema`.
214 +3. Add integration service/routes in `backend/app/integrations/<integration_name>`.
215 +4. Expose route in `backend/app/routers/<integration_name>.py`.
216 +5. Add scheduled collection jobs if needed: `backend/app/schedulers/services/*` and `backend/app/schedulers/scheduler.py`.
217 +
218 +### Add a New Scheduler Job
219 +1. Implement job function in `backend/app/schedulers/services`.
220 +2. Add to `known_jobs` in `backend/app/schedulers/scheduler.py`.
221 +3. Add function mapping in `get_function_by_name`.
222 +4. Expose job control in `backend/app/schedulers/routes/scheduler.py`.
223 +
224 +### Add a New Customer Portal Feature
225 +1. Add backend route with `customer_user` role scope.
226 +2. Add UI in `customer_portal/src/views` and wire in `customer_portal/src/router/index.ts`.
docs/architecture/DATABASE_SCHEMA.md new
+285
@@ -0,0 +1,285 @@
1 +# Database Schema (AI Agent-Oriented)
2 +
3 +This document summarizes the **current schema** for AI-agent workflows, using **Alembic migrations as source of truth** in `backend/alembic/versions/*.py`.
4 +
5 +- Current migration head in this repo: `fb51d610b306` (`backend/alembic/versions/fb51d610b306_add_github_audit_tables.py`)
6 +- Base migration: `bdf40d064ed1` (`backend/alembic/versions/bdf40d064ed1_initial_database_migration.py`)
7 +
8 +## Practical Domain Map
9 +
10 +For typical agent change work, these domains are most relevant:
11 +
12 +- Connectors and integration metadata: `connectors*`, `available_*`, `customer_*_connectors*`, `customer_*integrations*`, `integration_*`, `network_connectors_*`, `custom_alert_creation_*`, `monitoring_alerts`, `sigma_queries`, `github_audit_*`
13 +- Auth / users / roles: `user`, `role`, `smtp`, `user_customer_access`, `user_tag_access`, `role_tag_access`
14 +- Incidents (alerts/cases/tags/comments): all `incident_management_*` tables
15 +- Scheduler/job metadata: `scheduled_job_metadata`, `schedulerjob`, `index_snapshot_schedules`
16 +- Agent data store / artifacts / reports: `agent_datastore`, `incident_management_case_datastore`, `incident_management_case_report_template_datastore`, `vulnerability_reports`, `sca_reports`, `agent_vulnerabilities`
17 +
18 +## Critical relationship graph
19 +
20 +Compact relationship views for incident workflows and access controls.
21 +
22 +### Alerts -> assets (with link fields)
23 +
24 +```text
25 +incident_management_alert
26 + id (PK)
27 + |
28 + | 1-to-many via incident_management_asset.alert_linked
29 + v
30 +incident_management_asset
31 + alert_linked (FK -> incident_management_alert.id)
32 + alert_context_id (FK -> incident_management_alertcontext.id)
33 + index_name, index_id (origin pointer into SIEM index document)
34 +```
35 +
36 +### Cases <-> case-alert links <-> alerts
37 +
38 +```text
39 +incident_management_case incident_management_alert
40 + id (PK) id (PK)
41 + \ /
42 + \ /
43 + +-- incident_management_casealertlink --+
44 + case_id (FK -> incident_management_case.id)
45 + alert_id (FK -> incident_management_alert.id)
46 + PK(case_id, alert_id)
47 +```
48 +
49 +### Tags (alert/tag join)
50 +
51 +```text
52 +incident_management_alert incident_management_alerttag
53 + id (PK) id (PK), tag
54 + \ /
55 + \ /
56 + +-- incident_management_alert_to_tag --+
57 + alert_id (FK -> incident_management_alert.id)
58 + tag_id (FK -> incident_management_alerttag.id)
59 + PK(alert_id, tag_id)
60 +```
61 +
62 +### IoCs (alert/ioc join)
63 +
64 +```text
65 +incident_management_alert incident_management_ioc
66 + id (PK) id (PK), value/type/description
67 + \ /
68 + \ /
69 + +-- incident_management_alert_to_ioc --+
70 + alert_id (FK -> incident_management_alert.id)
71 + ioc_id (FK -> incident_management_ioc.id)
72 + PK(alert_id, ioc_id)
73 +```
74 +
75 +### Comments (alert comments + case comments)
76 +
77 +```text
78 +incident_management_comment
79 + alert_id (FK -> incident_management_alert.id)
80 + comment, user_name, created_at
81 +
82 +incident_management_case_comment
83 + case_id (FK -> incident_management_case.id)
84 + comment, user_name, created_at
85 +```
86 +
87 +### Case datastore + report template datastore
88 +
89 +```text
90 +incident_management_case
91 + id (PK)
92 + |
93 + | 1-to-many via incident_management_case_datastore.case_id
94 + v
95 +incident_management_case_datastore
96 + case_id (FK -> incident_management_case.id)
97 + bucket_name, object_key, file_name, file_hash, upload_time
98 +
99 +incident_management_case_report_template_datastore
100 + (global report templates; no case FK)
101 + report_template_name, bucket_name, object_key, file_name, file_hash, upload_time
102 +```
103 +
104 +### Tag access control and alert visibility
105 +
106 +```text
107 +incident_management_tag_access_settings
108 + enabled, untagged_alert_behavior, default_tag_id (FK -> incident_management_alerttag.id)
109 +
110 +user_tag_access (user_id, tag_id) role_tag_access (role_id, tag_id)
111 + \ /
112 + +------ allowed tag ids per identity ------+
113 + |
114 +incident_management_alert_to_tag (alert_id, tag_id)
115 + |
116 + incident_management_alert
117 +```
118 +
119 +When tag access control is enabled, alert visibility is constrained by the tag IDs reachable through `user_tag_access` and/or `role_tag_access` joined through `incident_management_alert_to_tag`. `incident_management_tag_access_settings` controls whether this filtering is active and what happens for untagged alerts (`untagged_alert_behavior`, optional `default_tag_id` fallback).
120 +
121 +**Tag access enforcement location (code pointers)**
122 +- Core tag RBAC logic: `backend/app/incidents/middleware/tag_access.py` (`TagAccessHandler`)
123 + - `is_tag_rbac_enabled()` (global enable/disable)
124 + - `build_alert_query_filters()` (computes accessible tags + untagged behavior)
125 + - `check_alert_tag_access()` / `can_user_access_alert()` (per-alert decision)
126 +- Applied in incident DB query layer:
127 + - `backend/app/incidents/services/db_operations.py` uses `tag_access_handler.build_alert_query_filters()` to add SQL `exists()` conditions when counting/listing alerts.
128 +
129 +### SIEM data origin + query pattern
130 +
131 +- Graylog alerting uses the `gl-events` index pattern (for example `gl-events*` in query flows).
132 +- Those Graylog alert documents live in Wazuh indexer storage (OpenSearch-backed).
133 +- Wazuh indexer is the backing SIEM event store across event sources (endpoints, O365 integrations, network connectors, and other ingested streams).
134 +- CoPilot commonly resolves and displays SIEM records by querying Wazuh indexer with `index_name` plus `index_id`.
135 +- Code pointers:
136 + - `backend/app/connectors/wazuh_indexer/routes/alerts.py`
137 + - `backend/app/connectors/wazuh_indexer/services/alerts.py`
138 + - `backend/app/routers/wazuh_indexer.py`
139 + - `frontend/src/api/endpoints/alerts.ts`
140 +
141 +## Table Inventory (Alembic-Derived)
142 +
143 +### Customer, Auth, and Core Platform
144 +
145 +| Table | PK | Important columns | Foreign keys | Model file(s) |
146 +|---|---|---|---|---|
147 +| `customers` | `id` | `customer_code`, `customer_name`, contact/address fields, `created_at` | None | `backend/app/db/universal_models.py` (`Customers`) |
148 +| `customersmeta` | `id` | `customer_code`, Graylog/Grafana/Wazuh metadata, `customer_meta_portainer_stack_id` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`CustomersMeta`) |
149 +| `customer_provisioning_default_settings` | `id` | `cluster_name`, `cluster_key`, `master_ip`, `grafana_url`, `wazuh_worker_hostname` | None | `backend/app/customer_provisioning/models/default_settings.py` |
150 +| `user` | `id` | `username`, `password`, `email`, `created_at`, `role_id` | `role_id -> role.id` | `backend/app/auth/models/users.py` (`User`) |
151 +| `role` | `id` | `name`, `description` | None | `backend/app/auth/models/users.py` (`Role`) |
152 +| `smtp` | `id` | `email`, `smtp_server`, `smtp_port`, `user_id` | `user_id -> user.id` | `backend/app/auth/models/users.py` (`SMTP`) |
153 +| `user_customer_access` | `id` | `user_id`, `customer_code`, `created_at` | `user_id -> user.id`, `customer_code -> customers.customer_code` | `backend/app/auth/models/users.py` |
154 +| `user_tag_access` | `id` | `user_id`, `tag_id`, `created_at` | `user_id -> user.id`, `tag_id -> incident_management_alerttag.id` | `backend/app/auth/models/users.py` |
155 +| `role_tag_access` | `id` | `role_id`, `tag_id`, `created_at` | `role_id -> role.id`, `tag_id -> incident_management_alerttag.id` | `backend/app/auth/models/users.py` |
156 +| `license` | `id` | `license_key`, customer/company identity fields | None | `backend/app/db/universal_models.py` (`License`) |
157 +| `license_cache` | `id` | `license_key`, `feature_name`, `is_enabled`, `cached_at`, `expires_at`, `license_data` | None | `backend/app/db/universal_models.py` (`LicenseCache`) |
158 +| `log_entries` | `id` | `timestamp`, `event_type`, `user_id`, `route`, `status_code`, `message` | None | `backend/app/db/universal_models.py` (`LogEntry`) |
159 +| `customer_portal_settings` | `id` | `title`, `logo_base64`, `logo_mime_type`, `updated_at`, `updated_by` | None | `backend/app/db/universal_models.py` (`CustomerPortalSettings`) |
160 +
161 +### Agents, Vulnerability, and Artifact/Data Store
162 +
163 +| Table | PK | Important columns | Foreign keys | Model file(s) |
164 +|---|---|---|---|---|
165 +| `agents` | `id` | `agent_id`, host/OS/status fields, `velociraptor_*`, `customer_code`, `quarantined`, `velociraptor_org` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`Agents`) |
166 +| `agent_datastore` | `id` | `agent_id`, `velociraptor_id`, `artifact_name`, `flow_id`, storage columns (`bucket_name`,`object_key`,`file_name`), `file_hash`, `status` | `agent_id -> agents.agent_id` | `backend/app/db/universal_models.py` (`AgentDataStore`) |
167 +| `agent_vulnerabilities` | `id` | `cve_id`, `severity`, `title`, `status`, `discovered_at`, `agent_id`, `customer_code` | `agent_id -> agents.agent_id`, `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`AgentVulnerabilities`) |
168 +| `vulnerability_reports` | `id` | `report_name`, `customer_code`, storage columns, `generated_at`, vulnerability counters, `status` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`VulnerabilityReport`) |
169 +| `sca_reports` | `id` | `report_name`, `customer_code`, storage columns, `generated_at`, SCA counters, `status` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`SCAReport`) |
170 +
171 +### Scheduler and Job Metadata
172 +
173 +| Table | PK | Important columns | Foreign keys | Model file(s) |
174 +|---|---|---|---|---|
175 +| `scheduled_job_metadata` | `id` | `job_id`, `last_success`, `time_interval`, `extra_data`, `enabled`, `job_description` | None | `backend/app/schedulers/models/scheduler.py` (`JobMetadata`) |
176 +| `schedulerjob` | `id` | `next_run_time`, `job_state` | None | `backend/app/db/universal_models.py` (`SchedulerJob`) |
177 +| `index_snapshot_schedules` | `id` | schedule metadata (`name`, `index_pattern`, `repository`), retention/last execution fields | None | `backend/app/connectors/wazuh_indexer/models/snapshot_and_restore.py` |
178 +
179 +### Connectors and Integrations
180 +
181 +| Table | PK | Important columns | Foreign keys | Model file(s) |
182 +|---|---|---|---|---|
183 +| `connectors` | `id` | connector identity/endpoint/auth fields, capability flags, `connector_enabled` | None | `backend/app/connectors/models.py` (`Connectors`) |
184 +| `connectorhistory` | `id` | `connector_id`, `change_timestamp`, `change_description` | `connector_id -> connectors.id` | `backend/app/connectors/models.py` (`ConnectorHistory`) |
185 +| `available_integrations` | `id` | `integration_name`, `description`, `integration_details` | None | `backend/app/integrations/models/customer_integration_settings.py` |
186 +| `available_integrations_auth_keys` | `id` | `integration_id`, `integration_name`, `auth_key_name` | `integration_id -> available_integrations.id` | `backend/app/integrations/models/customer_integration_settings.py` |
187 +| `customer_integrations` | `id` | `customer_code`, `integration_service_id`, `integration_service_name`, `deployed` | None | `backend/app/integrations/models/customer_integration_settings.py` |
188 +| `integration_services` | `id` | `service_name`, `auth_type` | None | `backend/app/integrations/models/customer_integration_settings.py` |
189 +| `integration_subscriptions` | `id` | `customer_id`, `integration_service_id` | `customer_id -> customer_integrations.id`, `integration_service_id -> integration_services.id` | `backend/app/integrations/models/customer_integration_settings.py` |
190 +| `integration_configs` | `id` | `integration_service_id`, `config_key`, `config_value` | `integration_service_id -> integration_services.id` | `backend/app/integrations/models/customer_integration_settings.py` |
191 +| `integration_auth_keys` | `id` | `subscription_id`, `auth_key_name`, `auth_value` | `subscription_id -> integration_subscriptions.id` | `backend/app/integrations/models/customer_integration_settings.py` |
192 +| `customer_integrations_meta` | `id` | Graylog/Grafana metadata (`graylog_*`, `grafana_*`, `grafana_datasource_uid`) | None | `backend/app/integrations/models/customer_integration_settings.py` |
193 +| `available_network_connectors` | `id` | `network_connector_name`, `description`, `network_connector_details` | None | `backend/app/network_connectors/models/network_connectors.py` |
194 +| `available_network_connectors_keys` | `id` | `network_connector_id`, `network_connector_name`, `auth_key_name` | `network_connector_id -> available_network_connectors.id` | `backend/app/network_connectors/models/network_connectors.py` |
195 +| `customer_network_connectors` | `id` | `customer_code`, `network_connector_service_id`, `network_connector_service_name`, `deployed` | None | `backend/app/network_connectors/models/network_connectors.py` |
196 +| `network_connectors_services` | `id` | `service_name`, `auth_type` | None | `backend/app/network_connectors/models/network_connectors.py` |
197 +| `network_connectors_subscriptions` | `id` | `customer_id`, `network_connectors_service_id` | `customer_id -> customer_network_connectors.id`, `network_connectors_service_id -> network_connectors_services.id` | `backend/app/network_connectors/models/network_connectors.py` |
198 +| `network_connectors_configs` | `id` | `network_connector_service_id`, `config_key`, `config_value` | `network_connector_service_id -> network_connectors_services.id` | `backend/app/network_connectors/models/network_connectors.py` |
199 +| `network_connectors_keys` | `id` | `subscription_id`, `auth_key_name`, `auth_value` | `subscription_id -> network_connectors_subscriptions.id` | `backend/app/network_connectors/models/network_connectors.py` |
200 +| `customer_network_connectors_meta` | `id` | Graylog/Grafana connector metadata (`graylog_*`, `grafana_*`, `grafana_datasource_uid`) | None | `backend/app/network_connectors/models/network_connectors.py` |
201 +| `custom_alert_creation_settings` | `id` | customer-wide alert-creation settings, `nvd_url`, custom integration URLs | None | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
202 +| `custom_alert_creation_event_order` | `id` | `alert_creation_settings_id`, `order_label` | `alert_creation_settings_id -> custom_alert_creation_settings.id` | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
203 +| `custom_alert_creation_condition` | `id` | `event_order_id`, `field_name`, `field_value` | `event_order_id -> custom_alert_creation_event_order.id` | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
204 +| `custom_alert_creation_event_config` | `id` | `event_order_id`, `event_id`, `field`, `value` | `event_order_id -> custom_alert_creation_event_order.id` | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
205 +| `monitoring_alerts` | `id` | `alert_id`, `alert_index`, `customer_code`, `alert_source` | None | `backend/app/integrations/monitoring_alert/models/monitoring_alert.py` |
206 +| `sigma_queries` | `id` | `rule_name`, `rule_query`, `active`, `time_interval`, execution timestamps | None | `backend/app/connectors/wazuh_indexer/models/sigma.py` |
207 +
208 +### Incident Management (Alerts, Cases, Tags, Comments)
209 +
210 +| Table | PK | Important columns | Foreign keys | Model file(s) |
211 +|---|---|---|---|---|
212 +| `incident_management_alert` | `id` | `alert_name`, `alert_description`, `status`, `alert_creation_time`, `customer_code`, `source`, `assigned_to`, `escalated` | None | `backend/app/incidents/models.py` (`Alert`) |
213 +| `incident_management_alertcontext` | `id` | `source`, `context` (JSON) | None | `backend/app/incidents/models.py` (`AlertContext`) |
214 +| `incident_management_asset` | `id` | `alert_linked`, `asset_name`, `alert_context_id`, `agent_id`, `customer_code`, `index_name`, `index_id` | `alert_linked -> incident_management_alert.id`, `alert_context_id -> incident_management_alertcontext.id` | `backend/app/incidents/models.py` (`Asset`) |
215 +| `incident_management_comment` | `id` | `alert_id`, `comment`, `user_name`, `created_at` | `alert_id -> incident_management_alert.id` | `backend/app/incidents/models.py` (`Comment`) |
216 +| `incident_management_case` | `id` | `case_name`, `case_description`, `case_creation_time`, `case_status`, `case_closed_time`, `assigned_to`, `customer_code`, `notification_invoked_number`, `escalated` | None | `backend/app/incidents/models.py` (`Case`) |
217 +| `incident_management_casealertlink` | composite: (`case_id`, `alert_id`) | link table case↔alert | `case_id -> incident_management_case.id`, `alert_id -> incident_management_alert.id` | `backend/app/incidents/models.py` (`CaseAlertLink`) |
218 +| `incident_management_case_comment` | `id` | `case_id`, `comment`, `user_name`, `created_at` | `case_id -> incident_management_case.id` | `backend/app/incidents/models.py` (`CaseComment`) |
219 +| `incident_management_alerttag` | `id` | `tag` | None | `backend/app/incidents/models.py` (`AlertTag`) |
220 +| `incident_management_alert_to_tag` | composite: (`alert_id`, `tag_id`) | link table alert↔tag | `alert_id -> incident_management_alert.id`, `tag_id -> incident_management_alerttag.id` | `backend/app/incidents/models.py` (`AlertToTag`) |
221 +| `incident_management_ioc` | `id` | `value`, `type`, `description` | None | `backend/app/incidents/models.py` (`IoC`) |
222 +| `incident_management_alert_to_ioc` | composite: (`alert_id`, `ioc_id`) | link table alert↔IoC | `alert_id -> incident_management_alert.id`, `ioc_id -> incident_management_ioc.id` | `backend/app/incidents/models.py` (`AlertToIoC`) |
223 +| `incident_management_fieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`FieldName`) |
224 +| `incident_management_assetfieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`AssetFieldName`) |
225 +| `incident_management_timestampfieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`TimestampFieldName`) |
226 +| `incident_management_alerttitlefieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`AlertTitleFieldName`) |
227 +| `incident_management_iocfieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`IoCFieldName`) |
228 +| `incident_management_customercodefieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`CustomerCodeFieldName`) |
229 +| `incident_management_notification` | `id` | `customer_code`, `shuffle_workflow_id`, `enabled` | None | `backend/app/incidents/models.py` (`Notification`) |
230 +| `incident_management_case_datastore` | `id` | `case_id`, storage fields (`bucket_name`,`object_key`,`file_name`), `upload_time`, `file_hash` | `case_id -> incident_management_case.id` | `backend/app/incidents/models.py` (`CaseDataStore`) |
231 +| `incident_management_case_report_template_datastore` | `id` | `report_template_name`, storage fields, `upload_time`, `file_hash` | None | `backend/app/incidents/models.py` (`CaseReportTemplateDataStore`) |
232 +| `incident_management_tag_access_settings` | `id` | `enabled`, `untagged_alert_behavior`, `default_tag_id`, `updated_at`, `updated_by` | `default_tag_id -> incident_management_alerttag.id` | `backend/app/incidents/models.py` (`TagAccessSettings`) |
233 +| `incident_management_velo_sigma_exclusion` | `id` | `name`, `field_matches` (JSON), `channel`, `title`, `customer_code`, `created_by`, `enabled` | None | `backend/app/incidents/models.py` (`VeloSigmaExclusion`) |
234 +
235 +### GitHub Audit (Added at Head)
236 +
237 +| Table | PK | Important columns | Foreign keys | Model file(s) |
238 +|---|---|---|---|---|
239 +| `github_audit_config` | `id` | customer/org token/configuration, filters (JSON), notification and score threshold fields | None | `backend/app/integrations/github_audit/model.py` |
240 +| `github_audit_check_exclusion` | `id` | `config_id`, `customer_code`, check/resource selectors, approval/expiry, `enabled` | `config_id -> github_audit_config.id` | `backend/app/integrations/github_audit/model.py` |
241 +| `github_audit_report` | `id` | `config_id`, organization/report metadata, summary counts, `status`, report JSON blobs | `config_id -> github_audit_config.id` | `backend/app/integrations/github_audit/model.py` |
242 +| `github_audit_baseline` | `id` | `config_id`, `customer_code`, baseline definition, expected checks (JSON), `baseline_report_id`, `is_active` | `config_id -> github_audit_config.id`, `baseline_report_id -> github_audit_report.id` | `backend/app/integrations/github_audit/model.py` |
243 +
244 +## Quick Model Scan: Tables Not Obvious From Alembic
245 +
246 +The following SQLModel tables are defined in code but do **not** appear in `backend/alembic/versions/*.py` migrations. Treat them as drift candidates / runtime-created tables unless there is an out-of-band migration process.
247 +
248 +| Table (in SQLModel) | Model file |
249 +|---|---|
250 +| `sublimealerts` | `backend/app/connectors/sublime/models/alerts.py` (`SublimeAlerts`) |
251 +| `flaggedrule` | `backend/app/connectors/sublime/models/alerts.py` (`FlaggedRule`) |
252 +| `mailbox` | `backend/app/connectors/sublime/models/alerts.py` (`Mailbox`) |
253 +| `triggeredaction` | `backend/app/connectors/sublime/models/alerts.py` (`TriggeredAction`) |
254 +| `sender` | `backend/app/connectors/sublime/models/alerts.py` (`Sender`) |
255 +| `recipient` | `backend/app/connectors/sublime/models/alerts.py` (`Recipient`) |
256 +| `disabledrule` | `backend/app/connectors/wazuh_manager/models/rules.py` (`DisabledRule`) |
257 +| `sap_siem_multiple_logins` | `backend/app/integrations/sap_siem/models/sap_siem.py` (`SapSiemMultipleLogins`) |
258 +
259 +## Where To Change Schema
260 +
261 +### Source of truth locations
262 +
263 +- Alembic migrations: `backend/alembic/versions/`
264 +- Alembic env/config: `backend/alembic/env.py`
265 +- SQLModel definitions commonly touched:
266 + - `backend/app/db/universal_models.py`
267 + - `backend/app/incidents/models.py`
268 + - `backend/app/auth/models/users.py`
269 + - `backend/app/network_connectors/models/network_connectors.py`
270 + - `backend/app/integrations/models/customer_integration_settings.py`
271 +
272 +### Practical workflow
273 +
274 +1. Update or add SQLModel fields/classes in the relevant model file.
275 +2. Generate a migration under `backend/alembic/versions/` (or author manually if needed).
276 +3. Review migration `upgrade()` and `downgrade()` carefully (FK names, nullable transitions, indexes).
277 +4. Apply migration locally and run tests.
278 +5. Update this document if table shape/ownership changes.
279 +
280 +### Notes for agent changes
281 +
282 +- Prefer extending existing domain tables over creating parallel tables when possible (especially incidents and connector metadata).
283 +- For incident workflows, changes usually involve: `incident_management_alert`, `incident_management_case`, link tables (`*_to_*`), and optional datastore tables.
284 +- For connector onboarding, changes usually involve: `available_*`, `customer_*`, `*_services`, `*_subscriptions`, `*_configs`, `*_keys`, and `*_meta` tables.
285 +- For scheduler automation, coordinate changes between `scheduled_job_metadata`, `schedulerjob`, and domain-specific tables storing job outcomes.
docs/architecture/DATA_FLOWS.md new
+97
@@ -0,0 +1,97 @@
1 +# Data Flows (AI Agent Quick Trace)
2 +
3 +This file is for fast debugging and change planning. Each flow includes the key files and the minimum execution path.
4 +
5 +## 1) Startup + Initialization
6 +
7 +Entry:
8 +- `backend/copilot.py` -> `@app.on_event("startup")`
9 +
10 +Flow:
11 +1. FastAPI app starts (`backend/copilot.py`).
12 +2. DB bootstrap/migration path runs (`backend/app/db/db_setup.py`):
13 + - `create_database_if_not_exists` (prod)
14 + - `create_copilot_user_if_not_exists` (prod)
15 + - `apply_migrations`
16 +3. Object storage buckets are ensured (`backend/app/data_store/data_store_setup.py:create_buckets`).
17 +4. Seed/reference data runs:
18 + - connectors (`add_connectors` -> `backend/app/db/db_populate.py`)
19 + - roles
20 + - available integrations/network connectors
21 +5. Admin + scheduler users ensured.
22 +6. APScheduler initialized and started (`backend/app/schedulers/scheduler.py`).
23 +
24 +## 2) Auth Request Flow
25 +
26 +Primary token endpoint:
27 +- `POST /api/auth/token` in `backend/app/auth/routes/auth.py`
28 +
29 +Flow:
30 +1. Frontend sign-in form submits credentials (`frontend/src/components/auth/SignIn.vue`).
31 +2. API wrapper sends form-data to `/auth/token` (`frontend/src/api/endpoints/auth.ts`).
32 +3. Backend authenticates user (`AuthHandler.authenticate_user` in `backend/app/auth/utils.py`).
33 +4. JWT is created with role scope(s) (`encode_token` in `backend/app/auth/utils.py`).
34 +5. Frontend stores token in auth store (`frontend/src/stores/auth.ts`).
35 +6. Axios interceptor adds `Authorization: Bearer <token>` on later calls (`frontend/src/api/httpClient.ts`).
36 +7. Protected backend routes validate token/scope via `AuthHandler.get_current_user` or `require_any_scope`.
37 +
38 +## 3) Scheduler Job Execution
39 +
40 +Core scheduler files:
41 +- `backend/app/schedulers/scheduler.py`
42 +- `backend/app/schedulers/routes/scheduler.py`
43 +
44 +Flow:
45 +1. Startup calls `init_scheduler`.
46 +2. `initialize_job_metadata` ensures known jobs exist in DB (`JobMetadata`).
47 +3. `schedule_enabled_jobs` loads enabled jobs and registers interval triggers.
48 +4. At run-time APScheduler calls mapped functions (`get_function_by_name`).
49 +5. Example job `invoke_alert_creation_collect`:
50 + - runs alert auto-create route logic (`backend/app/schedulers/services/invoke_alert_creation.py`)
51 + - updates `JobMetadata.last_success`.
52 +6. Manual operations (`/api/scheduler/...`) can run/pause/update/delete jobs.
53 +
54 +## 4) Connector Verify + Use
55 +
56 +Verify dispatch path:
57 +- `POST /api/connectors/verify/{id}` -> `backend/app/connectors/routes.py`
58 +- dispatch map in `backend/app/connectors/services.py:get_connector_service`
59 +
60 +Flow:
61 +1. Frontend calls verify (`frontend/src/api/endpoints/connectors.ts`).
62 +2. Backend fetches connector row by ID, builds response model.
63 +3. Connector name is mapped to a service class in `service_map`.
64 +4. Service class calls connector-specific verifier in `backend/app/connectors/<service>/utils/universal.py`.
65 +5. DB updates `connector_verified` + `connector_last_updated`.
66 +
67 +Use path (runtime connector client):
68 +1. Feature route/service calls a connector client factory in `utils/universal.py`.
69 +2. Factory pulls credentials via `get_connector_info_from_db` (`backend/app/connectors/utils.py`).
70 +3. Downstream API requests run with those connector settings.
71 +
72 +## 5) Alert -> Case
73 +
74 +Alert creation and case linking paths:
75 +- Auto/manual alert creation routes: `backend/app/incidents/routes/incident_alert.py`
76 +- Case creation routes: `backend/app/incidents/routes/db_operations.py`
77 +- Case creation service: `backend/app/incidents/services/db_operations.py`
78 +
79 +Flow:
80 +1. Alert is ingested/created (`/incident_alert/create/manual` or `/incident_alert/create/auto`).
81 +2. Analyst (or workflow) calls `/incident_management/case/from-alert`.
82 +3. Backend creates `Case` using alert fields (`create_case_from_alert`).
83 +4. Backend creates join record in `CaseAlertLink` (`create_case_alert_link`).
84 +5. Case now references the originating alert for SOC workflows and reporting.
85 +
86 +## 6) Artifact Upload to MinIO
87 +
88 +Two common paths:
89 +- Generic upload: `/api/agent_data_store/upload` (`backend/app/data_store/data_store_routes.py`)
90 +- Velociraptor collection upload: `backend/app/connectors/velociraptor/services/artifacts.py`
91 +
92 +Velociraptor-specific flow:
93 +1. Collection job runs and gets `flow_id`.
94 +2. `fetch_file_from_filestore` downloads zipped results locally.
95 +3. `upload_agent_artifact_file` uploads file to MinIO bucket `velociraptor-artifacts` with key `agent_id/flow_id/file.zip` (`backend/app/data_store/data_store_operations.py`).
96 +4. Metadata is stored in `AgentDataStore` table.
97 +5. UI/API can list/download/delete via `backend/app/data_store/data_store_routes.py`.
docs/architecture/DEPLOYMENT.md new
+74
@@ -0,0 +1,74 @@
1 +# Deployment (AI Agent View)
2 +
3 +Source of truth: `docker-compose.yml`.
4 +
5 +## Runtime Services
6 +
7 +| Service | Compose Name | Purpose | Exposed Ports | Persistent Storage |
8 +|---|---|---|---|---|
9 +| Backend API | `copilot-backend` | FastAPI app (`/api/*`) | `5000:5000` | `./data/copilot-backend-data/logs:/opt/logs`, `./data/data:/opt/copilot/backend/data` |
10 +| Frontend (Nginx) | `copilot-frontend` | UI + TLS termination + reverse proxy to backend | `80:80`, `443:443` | none by default |
11 +| MySQL | `copilot-mysql` | Primary relational DB | `3306:3306` | named volume `mysql-data:/var/lib/mysql` |
12 +| MinIO | `copilot-minio` | Object storage for case/artifact files | `9000:9000` (S3 API), container also uses `9001` console | `./data/data/minio-data:/data` |
13 +| Nuclei module | `copilot-nuclei-module` | External module container | none | none |
14 +| MCP service | `copilot-mcp` | MCP/OpenAI-adjacent service + optional subservers | none exposed by compose | `./data/copilot-mcp/api.config.yaml:/app/velociraptor-config.yaml:ro` |
15 +| Customer portal (optional) | `copilot-customer-portal` | Separate customer-facing UI | example `8443:443` (commented) | none by default |
16 +
17 +## Networking and Request Path
18 +
19 +- Browser -> `copilot-frontend` :443
20 +- `copilot-frontend` proxies `/api` to `http://copilot-backend:5000` (see `frontend/build/etc/nginx/sites-enabled/default.conf`)
21 +- Backend connects internally to:
22 + - MySQL via env (`MYSQL_URL`, defaults to `copilot-mysql`)
23 + - MinIO via env (`MINIO_URL`, defaults to `copilot-minio`)
24 +
25 +## Environment Variable Overview
26 +
27 +Primary env file: `.env` (template: `.env.example`).
28 +
29 +- Core runtime:
30 + - `SERVER_IP`, `SERVER_HOST`
31 +- MySQL:
32 + - `MYSQL_URL`, `MYSQL_ROOT_PASSWORD`, `MYSQL_USER`, `MYSQL_PASSWORD`, `MYSQL_DATABASE`
33 +- MinIO:
34 + - `MINIO_URL`, `MINIO_ROOT_USER`, `MINIO_ROOT_PASSWORD`, `MINIO_SECURE`
35 +- Connector bootstrap values:
36 + - e.g. `WAZUH_INDEXER_URL`, `GRAYLOG_URL`, `GRAFANA_URL`, etc. (loaded in `backend/app/db/db_populate.py`)
37 +- Header/shared-secret style values:
38 + - `GRAYLOG_API_HEADER_VALUE`, `VELOCIRAPTOR_API_HEADER_VALUE`
39 +- MCP/OpenAI values:
40 + - `OPENAI_API_KEY`, `OPENAI_MODEL`, `MCP_*`, `OPENSEARCH_*`, `WAZUH_PROD_*`, `VELOCIRAPTOR_*`
41 +
42 +## TLS
43 +
44 +Frontend TLS behavior is implemented in:
45 +- `frontend/build/docker-entrypoint.d/90-copilot-ssl.sh`
46 +- `frontend/build/etc/nginx/sites-enabled/default.conf`
47 +
48 +Behavior:
49 +- If `TLS_CERT_PATH`/`TLS_KEY_PATH` files exist, Nginx uses them.
50 +- If missing, startup script auto-generates self-signed certs (365 days).
51 +- Port 80 redirects to HTTPS (443).
52 +
53 +## Persistence Model
54 +
55 +- MySQL durable data:
56 + - Docker named volume `mysql-data`.
57 +- MinIO durable objects:
58 + - Host path `./data/data/minio-data`.
59 +- Backend local files/logs:
60 + - Host paths under `./data/...` bind-mounted into backend.
61 +- Buckets auto-created at startup:
62 + - See `backend/app/data_store/data_store_setup.py` (`copilot-cases`, `copilot-case-report-templates`, `sysmon-configs`, `velociraptor-artifacts`).
63 +
64 +## Startup Initialization Hooks (Deployment-Relevant)
65 +
66 +`backend/copilot.py` startup event performs:
67 +- DB creation/user bootstrap in production
68 +- Alembic migrations (`backend/app/db/db_setup.py`)
69 +- MinIO bucket creation
70 +- connector + integration seed data
71 +- admin/scheduler user ensure
72 +- scheduler init/start
73 +
74 +If deployment seems healthy but features fail, validate this startup chain first.
docs/architecture/MAP.md new
+137
@@ -0,0 +1,137 @@
1 +# CoPilot Repo Map
2 +
3 +## Root
4 +- `README.md`
5 + High‑level product overview, install steps, and TLS notes.
6 +- `docker-compose.yml`
7 + Deployment stack for backend, frontend, MySQL, MinIO, MCP, and Nuclei.
8 +- `.env.example`
9 + Canonical list of backend environment variables and connector settings.
10 +- `build-dockers.sh`
11 + Build helper for Docker images.
12 +
13 +## Backend Entry & Config
14 +- `backend/copilot.py`
15 + FastAPI app initialization, router mounting, startup/shutdown orchestration.
16 +- `backend/settings.py`
17 + Local env loading; legacy settings for SQLAlchemy URI.
18 +- `backend/requirements.txt`
19 + Backend dependencies including FastAPI, SQLModel, APScheduler, integrations.
20 +
21 +## Database & Migrations
22 +- `backend/app/db/db_session.py`
23 + Async and sync SQLAlchemy engines, session management.
24 +- `backend/app/db/db_setup.py`
25 + DB creation, migrations, seeding, admin/scheduler user creation.
26 +- `backend/app/db/db_populate.py`
27 + Default connectors, integrations, roles, auth keys.
28 +- `backend/alembic/`
29 + Alembic migrations and config.
30 +
31 +## Auth & Middleware
32 +- `backend/app/auth/utils.py`
33 + JWT auth, scopes/roles enforcement.
34 +- `backend/app/auth/models/users.py`
35 + User and role models.
36 +- `backend/app/middleware/*`
37 + License gating, logging, customer access control, exception handling.
38 +
39 +## Core Routing
40 +- `backend/app/routers/`
41 + Route modules for every domain (connectors, agents, incidents, integrations, etc.).
42 +
43 +## Scheduler & Jobs
44 +- `backend/app/schedulers/scheduler.py`
45 + APScheduler setup, job metadata, scheduling logic.
46 +- `backend/app/schedulers/routes/scheduler.py`
47 + API endpoints to list/update jobs.
48 +- `backend/app/schedulers/services/`
49 + Collectors and scheduled tasks (alert creation, Cato, Duo, Darktrace, etc.).
50 +
51 +## Connectors (Platform Services)
52 +- `backend/app/connectors/routes.py`
53 + Connector CRUD and verification API.
54 +- `backend/app/connectors/services.py`
55 + Connector verification dispatch map and file upload handling.
56 +- `backend/app/connectors/utils.py`
57 + Shared DB lookup helpers for connectors.
58 +- `backend/app/connectors/wazuh_manager/`
59 + Wazuh Manager auth/token caching and request utilities.
60 +- `backend/app/connectors/wazuh_indexer/`
61 + Wazuh Indexer connection utilities.
62 +- `backend/app/connectors/graylog/`
63 + Graylog API helpers and routing.
64 +- `backend/app/connectors/grafana/`
65 + Grafana connection utilities and folder/datasource management.
66 +- `backend/app/connectors/velociraptor/`
67 + Velociraptor connection and API helpers.
68 +- `backend/app/connectors/shuffle/`
69 + Shuffle connection verification.
70 +- `backend/app/connectors/event_shipper/`
71 + GELF TCP logger for Graylog event shipping.
72 +- `backend/app/connectors/portainer/`
73 + Portainer connection utilities.
74 +
75 +## Integrations (Per‑Customer)
76 +- `backend/app/integrations/routes.py`
77 + Customer integration CRUD and validation.
78 +- `backend/app/integrations/models/customer_integration_settings.py`
79 + Integration config and auth key models.
80 +- `backend/app/integrations/modules/`
81 + Data collection modules for Duo, Darktrace, Mimecast, Huntress, etc.
82 +- `backend/app/integrations/copilot_mcp/`
83 + MCP query routing (local and cloud services).
84 +- `backend/app/integrations/nuclei/`
85 + Web vulnerability assessment.
86 +- `backend/app/integrations/scoutsuite/`
87 + Cloud security assessment.
88 +- `backend/app/integrations/github_audit/`
89 + GitHub audit reports and metadata.
90 +
91 +## Network Connectors
92 +- `backend/app/network_connectors/routes.py`
93 + Customer‑scoped “network connector” management and auth keys.
94 +- `backend/app/network_connectors/models/network_connectors.py`
95 + Network connector DB schema and relations.
96 +
97 +## Provisioning
98 +- `backend/app/customer_provisioning/services/`
99 + Provision/decommission Graylog, Grafana, Wazuh, Portainer for customers.
100 +- `backend/app/stack_provisioning/graylog/`
101 + Graylog content packs, pipelines, streams, inputs templates.
102 +
103 +## Incidents & SOC Features
104 +- `backend/app/incidents/`
105 + Incident alerts/cases, reports, tags, and case data store.
106 +- `backend/app/agents/`
107 + Wazuh/Velociraptor agents, SCA, vulnerabilities, data store.
108 +
109 +## Data Store
110 +- `backend/app/data_store/data_store_session.py`
111 + MinIO client factory.
112 +- `backend/app/data_store/data_store_setup.py`
113 + Buckets for cases, templates, sysmon configs, Velociraptor artifacts.
114 +
115 +## Active Response
116 +- `backend/app/active_response/`
117 + Active response routes and scripts (Windows/Linux).
118 +
119 +## Threat Intel
120 +- `backend/app/threat_intel/`
121 + EPSS, VirusTotal, SOCFortress threat intel routes/services.
122 +
123 +## Frontend (Admin UI)
124 +- `frontend/src/router/index.ts`
125 + Primary UI routes and feature pages.
126 +- `frontend/src/api/endpoints/`
127 + Typed API clients for backend endpoints.
128 +- `frontend/src/components/`
129 + Feature components: alerts, cases, agents, connectors, integrations, reports.
130 +- `frontend/.env.example`
131 + Vite environment defaults.
132 +
133 +## Customer Portal
134 +- `customer_portal/src/router/index.ts`
135 + Customer portal routes (login, alerts, cases, agents).
136 +- `customer_portal/src/views/`
137 + Customer‑facing views with limited features.
docs/integrations/ADDING_A_CONNECTOR.md new
+130
@@ -0,0 +1,130 @@
1 +# Adding a Connector (AI Agent Checklist)
2 +
3 +Use this when introducing a new backend connector integration and wiring it through UI/API.
4 +
5 +## Scope
6 +
7 +This checklist covers:
8 +- connector bootstrap record
9 +- verify dispatch wiring
10 +- connector utility client + verifier
11 +- route wiring
12 +- frontend endpoint/UI integration
13 +
14 +## 1) Add Connector to Seed Data
15 +
16 +File: `backend/app/db/db_populate.py`
17 +
18 +Actions:
19 +1. Update `get_connectors_list()` with your new connector tuple.
20 +2. Pick exactly one auth mode flag via `accepts_key`:
21 + - `host_only`
22 + - `api_key`
23 + - `username_password`
24 + - `file`
25 +3. If needed, define `extra_data_key` env var for `connector_extra_data`.
26 +4. Ensure env var naming matches `load_connector_data()` prefix rule:
27 + - `connector_name.upper().replace("-", "_").replace(" ", "_")`
28 +
29 +Result:
30 +- connector appears in `/api/connectors` after startup seed.
31 +
32 +## 2) Implement Connector Utility Module
33 +
34 +File path (new): `backend/app/connectors/<service>/utils/universal.py`
35 +
36 +Minimum functions to provide:
37 +1. `verify_<service>_connection(connector_name: str)`
38 +2. `create_<service>_client(connector_name: str = "<Connector-Display-Name>")`
39 +
40 +Implementation requirements:
41 +- Pull credentials via `get_connector_info_from_db` from `backend/app/connectors/utils.py`.
42 +- Return consistent verify payload:
43 + - `{"connectionSuccessful": bool, "message": str}`
44 +- Raise `HTTPException` for hard failures in runtime client creation.
45 +
46 +## 3) Register Verify Dispatch
47 +
48 +File: `backend/app/connectors/services.py`
49 +
50 +Actions:
51 +1. Import your verifier function.
52 +2. Add service class (pattern: `<ServiceName>Service`) implementing `verify_authentication`.
53 +3. Add mapping in `get_connector_service()` `service_map`:
54 + - key must match DB `connector_name` exactly.
55 +
56 +If omitted, `/api/connectors/verify/{id}` will return unsupported/None behavior.
57 +
58 +## 4) Add Connector Routes (If Exposing Feature APIs)
59 +
60 +Typical files:
61 +- `backend/app/connectors/<service>/routes/*.py`
62 +- `backend/app/connectors/<service>/services/*.py`
63 +- `backend/app/connectors/<service>/schema/*.py`
64 +
65 +Router wiring:
66 +1. Add/modify router module `backend/app/routers/<service>.py`.
67 +2. Include route groups with appropriate prefixes/tags.
68 +3. Add top-level include in `backend/copilot.py`:
69 + - `from app.routers import <service>`
70 + - `api_router.include_router(<service>.router)`
71 +
72 +Without step 3, routes compile but are unreachable.
73 +
74 +## 5) Frontend Endpoint Wiring
75 +
76 +Primary files:
77 +- `frontend/src/api/endpoints/connectors.ts`
78 +- optional new endpoint file if connector has dedicated APIs (pattern in `frontend/src/api/endpoints/*.ts`)
79 +- `frontend/src/api/index.ts` (export)
80 +
81 +Checklist:
82 +1. Reuse generic `/connectors` endpoints if only configuring/verifying credentials.
83 +2. Add dedicated endpoint wrapper(s) for new connector-specific backend routes.
84 +3. Ensure payload type definitions exist/update in `frontend/src/types/*.d.ts`.
85 +
86 +## 6) Frontend UI Wiring
87 +
88 +Connector configuration UI already exists:
89 +- View: `frontend/src/views/Connectors.vue`
90 +- List: `frontend/src/components/connectors/ConnectorsList.vue`
91 +- Item: `frontend/src/components/connectors/ConnectorItem.vue`
92 +- Form: `frontend/src/components/connectors/ConfigForm/ConfigForm.vue`
93 +
94 +Checklist:
95 +1. Ensure DB flags (`connector_accepts_*`) drive the correct form type.
96 +2. Add connector logo asset if needed (`frontend/public/images/connectors/<lowercase-name>.svg`).
97 +3. Add any connector-specific screens/routes only if required:
98 + - route map: `frontend/src/router/index.ts`
99 +
100 +## 7) Environment + Secrets
101 +
102 +Update `.env.example` with required connector vars so bootstrap is deterministic.
103 +
104 +Rules:
105 +- Do not hardcode secrets in code.
106 +- Read credentials from DB connector config at runtime.
107 +- Keep placeholder defaults non-production.
108 +
109 +## 8) Validation Steps
110 +
111 +1. Start stack and call `GET /api/connectors` to confirm seed row exists.
112 +2. Configure connector in UI (`/connectors`).
113 +3. Call `POST /api/connectors/verify/{id}` and check `connectionSuccessful`.
114 +4. Exercise at least one runtime endpoint that uses `create_<service>_client`.
115 +
116 +## Common Pitfalls
117 +
118 +- TLS verification mismatch:
119 + - many existing connectors use `verify=False`/`verify_certs=False` for self-signed deployments.
120 + - if you enable strict TLS, make it explicit and configurable.
121 +- Timeout defaults too low/high:
122 + - define per-connector timeouts; include retries only when safe.
123 +- Wrong auth header format:
124 + - token connectors often require exact header names/prefixes (`Authorization: Bearer ...`, custom headers, etc.).
125 +- Connector name mismatch:
126 + - DB seed name and `service_map` key must be identical.
127 +- Router not included at top level:
128 + - adding route files alone is insufficient; include in `backend/copilot.py`.
129 +- Missing frontend export/wiring:
130 + - endpoint file exists but not exported in `frontend/src/api/index.ts`.