| 1 | # SOCFortress CoPilot Architecture |
| 2 | |
| 3 | ## Overview |
| 4 | CoPilot is a “single pane of glass” security operations platform. It centralizes data from Wazuh, Graylog, Velociraptor, Grafana, InfluxDB, and other tools, provides alert/case management, and adds automation such as scheduled collectors, active response, and report generation. The backend is a FastAPI service with a MySQL database and MinIO object storage. The UI is a Vue 3 SPA, with an optional customer-facing portal. |
| 5 | |
| 6 | Key entry points: |
| 7 | - Backend runtime: `backend/copilot.py` |
| 8 | - Frontend app: `frontend/src` |
| 9 | - Customer portal app: `customer-portal/src` |
| 10 | |
| 11 | ## Tech Stack |
| 12 | |
| 13 | ### Backend |
| 14 | - FastAPI, Starlette, Uvicorn: API server and routing. `backend/copilot.py` |
| 15 | - SQLModel + SQLAlchemy: ORM and DB access. `backend/app/db` |
| 16 | - Alembic: DB migrations. `backend/alembic` |
| 17 | - APScheduler: background job scheduling. `backend/app/schedulers/scheduler.py` |
| 18 | - MySQL: primary database. `docker-compose.yml`, `backend/app/db/db_session.py` |
| 19 | - MinIO: object storage for case/report artifacts, sysmon configs, Velociraptor artifacts. `backend/app/data_store` |
| 20 | - Loguru: logging. `backend` modules |
| 21 | - Requests/HTTPX: integrations with external systems. `backend/app/connectors/*/utils`, `backend/app/integrations/*` |
| 22 | |
| 23 | ### Frontend |
| 24 | - Vue 3 + TypeScript + Vite: SPA. `frontend/package.json`, `frontend/src` |
| 25 | - Pinia: state management. `frontend/src/stores` |
| 26 | - Naive UI + Tailwind CSS: UI components and styling. `frontend/package.json`, `frontend/tailwind.config.js` |
| 27 | - Axios: API client. `frontend/src/api` |
| 28 | - Cypress + Vitest: tests. `frontend/cypress`, `frontend/vitest.config.ts` |
| 29 | |
| 30 | ### Customer Portal (Optional) |
| 31 | - Vue 3 + Vite + Pinia + Naive UI. `customer-portal/package.json`, `customer-portal/src` |
| 32 | |
| 33 | ## Directory Layout & Responsibilities |
| 34 | |
| 35 | ### Root |
| 36 | - `docker-compose.yml`: default deployment stack (backend, frontend, MySQL, MinIO, MCP, Nuclei). |
| 37 | - `.env.example`: required runtime configuration for connectors and services. |
| 38 | - `build-dockers.sh`: build utility script. |
| 39 | |
| 40 | ### Backend |
| 41 | - `backend/copilot.py`: application entrypoint, API router mount, startup/shutdown hooks. |
| 42 | - `backend/settings.py`: local env loading (not the primary runtime env in Docker). |
| 43 | - `backend/app/routers/*`: API route modules for each feature/integration. |
| 44 | - `backend/app/auth/*`: authentication, user/role management. |
| 45 | - `backend/app/db/*`: sessions, migrations, bootstrapping, and data seeding. |
| 46 | - `backend/app/schedulers/*`: APScheduler instance, scheduled jobs, job metadata models. |
| 47 | - `backend/app/connectors/*`: connectors for platform services (Wazuh, Graylog, Grafana, etc.). |
| 48 | - `backend/app/integrations/*`: third‑party integrations and per‑customer integration config. |
| 49 | - `backend/app/network_connectors/*`: “network connectors” with customer‑scoped auth keys and configs. |
| 50 | - `backend/app/customer_provisioning/*`: provisioning workflows for external services (Graylog, Grafana, Portainer, Wazuh manager). |
| 51 | - `backend/app/stack_provisioning/graylog/*`: content packs, pipelines, streams, and input templates for Graylog. |
| 52 | - `backend/app/agents/*`: Wazuh and Velociraptor agent management, SCA, vulnerabilities. |
| 53 | - `backend/app/incidents/*`: incident management (cases, alerts, tags, reports). |
| 54 | - `backend/app/active_response/*`: automation scripts and invoke endpoints. |
| 55 | - `backend/app/data_store/*`: MinIO storage (cases, templates, artifacts). |
| 56 | - `backend/app/threat_intel/*`: EPSS, VirusTotal, SOCFortress threat intel. |
| 57 | - `backend/app/integrations/copilot_mcp/*`: local and cloud MCP queries. |
| 58 | |
| 59 | ### Frontend |
| 60 | - `frontend/src/router/index.ts`: primary navigation and feature routes. |
| 61 | - `frontend/src/api/endpoints/*`: typed API clients for each backend domain. |
| 62 | - `frontend/src/components/*`: feature UI modules (alerts, cases, agents, connectors, integrations, etc.). |
| 63 | - `frontend/src/views/*`: route views; corresponds closely to backend feature areas. |
| 64 | - `frontend/.env.example`: API base URL and UI behavior. |
| 65 | |
| 66 | ### Customer Portal |
| 67 | - `customer-portal/src/router/index.ts`: simple login + alerts/cases/agents views. |
| 68 | - `customer-portal/src/views/*`: customer‑limited UI. |
| 69 | |
| 70 | ## Core Runtime Flows |
| 71 | |
| 72 | ### Startup |
| 73 | 1. `backend/copilot.py` loads env vars and initializes FastAPI. |
| 74 | 2. On startup event: |
| 75 | - Creates MySQL database and user if needed. `backend/app/db/db_setup.py` |
| 76 | - Applies Alembic migrations. `backend/app/db/db_setup.py` |
| 77 | - Creates MinIO buckets. `backend/app/data_store/data_store_setup.py` |
| 78 | - Seeds connectors, roles, available integrations, available network connectors. `backend/app/db/db_setup.py`, `backend/app/db/db_populate.py` |
| 79 | - Ensures admin and scheduler users. `backend/app/db/db_setup.py` |
| 80 | - Initializes APScheduler and schedules enabled jobs. `backend/app/schedulers/scheduler.py` |
| 81 | 3. Static mount: `scoutsuite-report` directory for cloud security assessment outputs. `backend/copilot.py` |
| 82 | |
| 83 | ### Auth & Authorization |
| 84 | - JWT auth with OAuth2 password flow. `backend/app/auth/utils.py` |
| 85 | - Roles/scopes: `admin`, `analyst`, `scheduler`, `customer_user`. |
| 86 | - Routes use `Security(AuthHandler().get_current_user, scopes=[...])` or `require_any_scope(...)`. |
| 87 | |
| 88 | ### API Routing |
| 89 | - `FastAPI` app mounts a single APIRouter at `/api`. |
| 90 | - Each domain lives in `backend/app/routers/*.py` and delegates to feature modules in `backend/app/<domain>/*`. |
| 91 | - Frontend API clients match route structure. `frontend/src/api/endpoints/*` |
| 92 | |
| 93 | ### Background Jobs |
| 94 | - APScheduler runs inside the FastAPI app. |
| 95 | - Job metadata is stored in MySQL and loaded on startup. `backend/app/schedulers/scheduler.py` |
| 96 | - Jobs invoke integration collectors and internal maintenance: |
| 97 | - Agent sync (Wazuh + Velociraptor) |
| 98 | - Wazuh index resize |
| 99 | - Alert creation collection |
| 100 | - Snapshot schedule execution |
| 101 | - Integration collectors (Duo, Darktrace, Cato, Huntress, Carbon Black, etc.) |
| 102 | - Job definitions live in `backend/app/schedulers/services/*`. |
| 103 | |
| 104 | ### Integrations & Connectors |
| 105 | - Connectors are core service connections (Wazuh, Graylog, Grafana, etc.) stored in `Connectors` DB table. |
| 106 | - Connectors are seeded from env vars on startup. `backend/app/db/db_populate.py` |
| 107 | - Each connector implements a verify function in `backend/app/connectors/<service>/utils/universal.py` and is mapped in `backend/app/connectors/services.py`. |
| 108 | - Integration settings are customer‑scoped and stored in `CustomerIntegrations` / `CustomerIntegrationsMeta` tables. |
| 109 | |
| 110 | ### Data Storage |
| 111 | - MySQL for all operational data (users, integrations, connectors, alerts/cases, scheduler metadata). |
| 112 | - MinIO for case artifacts, report templates, sysmon configs, Velociraptor artifacts. `backend/app/data_store` |
| 113 | - Local filesystem: `scoutsuite-report` directory for report outputs. |
| 114 | |
| 115 | ## Major User‑Facing Features (Admin UI) |
| 116 | - Overview dashboard and system health. |
| 117 | - Connectors management (test/verify/update). |
| 118 | - Wazuh management: rules, groups, Sysmon config, MITRE browsing. |
| 119 | - Graylog management, metrics, pipelines, streams, inputs. |
| 120 | - Alerts and SIEM views, MITRE/Atomic Red Team views. |
| 121 | - Incident management: sources, alerts, cases, tags, comments, reports, data store. |
| 122 | - Agents (Wazuh + Velociraptor), SCA, vulnerabilities, data store. |
| 123 | - Artifacts and file collection. |
| 124 | - Active response actions. |
| 125 | - External services: third‑party integrations and network connectors. |
| 126 | - Reporting (Grafana dashboards, case reports, vuln/SCA reports). |
| 127 | - Scheduler management. |
| 128 | - Cloud security assessment (ScoutSuite). |
| 129 | - Web vulnerability assessment (Nuclei). |
| 130 | - GitHub Audit. |
| 131 | - Customer portal branding/settings. |
| 132 | - License management. |
| 133 | |
| 134 | ## Customer Portal Features |
| 135 | - Login and role‑restricted access for `customer_user`. |
| 136 | - Views for alerts, cases, case details, and agents. |
| 137 | - Separate SPA served by `copilot-customer-portal` container. |
| 138 | |
| 139 | ## Integrations & Open Source Services |
| 140 | |
| 141 | ### Wazuh |
| 142 | - Connector: `backend/app/connectors/wazuh_manager/*` and `backend/app/connectors/wazuh_indexer/*` |
| 143 | - Auth: API token cached in memory with TTL; requests via `requests`. |
| 144 | - Wazuh manager routes: `backend/app/connectors/wazuh_manager/routes/*` and `backend/app/routers/wazuh_manager.py`. |
| 145 | - Wazuh indexer routes: `backend/app/connectors/wazuh_indexer/routes/*`. |
| 146 | |
| 147 | ### Graylog |
| 148 | - Connector: `backend/app/connectors/graylog/*` |
| 149 | - Event shipper: GELF TCP to Graylog input. `backend/app/connectors/event_shipper/*`, `backend/app/integrations/utils/event_shipper.py` |
| 150 | - Graylog provisioning: content packs, pipelines, streams, inputs. `backend/app/stack_provisioning/graylog/*` |
| 151 | - Graylog API management used in routes and service modules. |
| 152 | |
| 153 | ### Grafana |
| 154 | - Connector: `backend/app/connectors/grafana/*` |
| 155 | - Reporting endpoints for orgs/dashboards/panels and iframe generation. `backend/app/routers/grafana.py`, `frontend/src/components/reportCreation` |
| 156 | |
| 157 | ### Velociraptor |
| 158 | - Connector: `backend/app/connectors/velociraptor/*` |
| 159 | - Agent management + artifacts; artifacts also stored in MinIO. `backend/app/agents/velociraptor/*`, `backend/app/data_store` |
| 160 | |
| 161 | ### Shuffle |
| 162 | - Connector: `backend/app/connectors/shuffle/*` |
| 163 | - Endpoints for Shuffle metadata. `backend/app/routers/shuffle.py` |
| 164 | |
| 165 | ### InfluxDB |
| 166 | - Connector: `backend/app/connectors/influxdb/*` |
| 167 | - Healthcheck and monitoring endpoints. `backend/app/routers/influxdb.py`, `frontend/src/components/healthcheck` |
| 168 | |
| 169 | ### Portainer |
| 170 | - Connector: `backend/app/connectors/portainer/*` |
| 171 | - Customer provisioning workflows can call Portainer. `backend/app/customer_provisioning/services/portainer.py` |
| 172 | |
| 173 | ### Nuclei |
| 174 | - Integration: `backend/app/integrations/nuclei/*` and `backend/app/routers/nuclei.py` |
| 175 | - Container present in `docker-compose.yml` as `copilot-nuclei-module`. |
| 176 | |
| 177 | ### ScoutSuite |
| 178 | - Integration: `backend/app/integrations/scoutsuite/*` and `backend/app/routers/scoutsuite.py` |
| 179 | - Outputs served from `scoutsuite-report` static mount. |
| 180 | |
| 181 | ### Threat Intel |
| 182 | - VirusTotal, EPSS, SOCFortress. `backend/app/threat_intel/*` |
| 183 | |
| 184 | ### MCP (CoPilot AI) |
| 185 | - Local MCP service container configured in Docker. |
| 186 | - Backend routes call `backend/app/integrations/copilot_mcp/services/copilot_mcp.py` to query local OpenSearch/MySQL/Wazuh/Velociraptor or cloud threat intel endpoints. |
| 187 | |
| 188 | ## Configuration Management & Secrets |
| 189 | - `.env` provides connector URLs, API keys, DB creds, MinIO creds, MCP settings. `.env.example` documents expected values. |
| 190 | - Frontend uses `VITE_API_URL` to target the backend. |
| 191 | - TLS handled by `copilot-frontend` container; TLS cert/key paths are configurable in `docker-compose.yml` and documented in README. |
| 192 | |
| 193 | ## Deployment (Docker Compose) |
| 194 | - `copilot-backend`: FastAPI service on port 5000. |
| 195 | - `copilot-frontend`: Vue app with TLS, ports 80/443. |
| 196 | - `copilot-mysql`: MySQL 8. |
| 197 | - `copilot-minio`: object storage. |
| 198 | - `copilot-nuclei-module`: web vulnerability scanner module. |
| 199 | - `copilot-mcp`: MCP service for AI queries. |
| 200 | - Optional `copilot-customer-portal` for customer‑facing UI. |
| 201 | |
| 202 | ## Extension Points |
| 203 | |
| 204 | ### Add a New Connector |
| 205 | 1. Add connector metadata in `backend/app/db/db_populate.py`. |
| 206 | 2. Add verification logic in `backend/app/connectors/<new_service>/utils/universal.py`. |
| 207 | 3. Map the connector name in `backend/app/connectors/services.py`. |
| 208 | 4. Provide routes in `backend/app/connectors/<new_service>/routes` and `backend/app/routers/<new_service>.py` as needed. |
| 209 | 5. Add frontend UI and API client in `frontend/src/components` and `frontend/src/api/endpoints`. |
| 210 | |
| 211 | ### Add a New Integration (Per‑Customer) |
| 212 | 1. Add integration metadata in `backend/app/db/db_populate.py`. |
| 213 | 2. Add models/schema in `backend/app/integrations/models` and `backend/app/integrations/schema`. |
| 214 | 3. Add integration service/routes in `backend/app/integrations/<integration_name>`. |
| 215 | 4. Expose route in `backend/app/routers/<integration_name>.py`. |
| 216 | 5. Add scheduled collection jobs if needed: `backend/app/schedulers/services/*` and `backend/app/schedulers/scheduler.py`. |
| 217 | |
| 218 | ### Add a New Scheduler Job |
| 219 | 1. Implement job function in `backend/app/schedulers/services`. |
| 220 | 2. Add to `known_jobs` in `backend/app/schedulers/scheduler.py`. |
| 221 | 3. Add function mapping in `get_function_by_name`. |
| 222 | 4. Expose job control in `backend/app/schedulers/routes/scheduler.py`. |
| 223 | |
| 224 | ### Add a New Customer Portal Feature |
| 225 | 1. Add backend route with `customer_user` role scope. |
| 226 | 2. Add UI in `customer-portal/src/views` and wire in `customer-portal/src/router/index.ts`. |