| 1 | # Data Flows (AI Agent Quick Trace) |
| 2 | |
| 3 | This file is for fast debugging and change planning. Each flow includes the key files and the minimum execution path. |
| 4 | |
| 5 | ## 1) Startup + Initialization |
| 6 | |
| 7 | Entry: |
| 8 | - `backend/copilot.py` -> `@app.on_event("startup")` |
| 9 | |
| 10 | Flow: |
| 11 | 1. FastAPI app starts (`backend/copilot.py`). |
| 12 | 2. DB bootstrap/migration path runs (`backend/app/db/db_setup.py`): |
| 13 | - `create_database_if_not_exists` (prod) |
| 14 | - `create_copilot_user_if_not_exists` (prod) |
| 15 | - `apply_migrations` |
| 16 | 3. Object storage buckets are ensured (`backend/app/data_store/data_store_setup.py:create_buckets`). |
| 17 | 4. Seed/reference data runs: |
| 18 | - connectors (`add_connectors` -> `backend/app/db/db_populate.py`) |
| 19 | - roles |
| 20 | - available integrations/network connectors |
| 21 | 5. Admin + scheduler users ensured. |
| 22 | 6. APScheduler initialized and started (`backend/app/schedulers/scheduler.py`). |
| 23 | |
| 24 | ## 2) Auth Request Flow |
| 25 | |
| 26 | Primary token endpoint: |
| 27 | - `POST /api/auth/token` in `backend/app/auth/routes/auth.py` |
| 28 | |
| 29 | Flow: |
| 30 | 1. Frontend sign-in form submits credentials (`frontend/src/components/auth/SignIn.vue`). |
| 31 | 2. API wrapper sends form-data to `/auth/token` (`frontend/src/api/endpoints/auth.ts`). |
| 32 | 3. Backend authenticates user (`AuthHandler.authenticate_user` in `backend/app/auth/utils.py`). |
| 33 | 4. JWT is created with role scope(s) (`encode_token` in `backend/app/auth/utils.py`). |
| 34 | 5. Frontend stores token in auth store (`frontend/src/stores/auth.ts`). |
| 35 | 6. Axios interceptor adds `Authorization: Bearer <token>` on later calls (`frontend/src/api/httpClient.ts`). |
| 36 | 7. Protected backend routes validate token/scope via `AuthHandler.get_current_user` or `require_any_scope`. |
| 37 | |
| 38 | ## 3) Scheduler Job Execution |
| 39 | |
| 40 | Core scheduler files: |
| 41 | - `backend/app/schedulers/scheduler.py` |
| 42 | - `backend/app/schedulers/routes/scheduler.py` |
| 43 | |
| 44 | Flow: |
| 45 | 1. Startup calls `init_scheduler`. |
| 46 | 2. `initialize_job_metadata` ensures known jobs exist in DB (`JobMetadata`). |
| 47 | 3. `schedule_enabled_jobs` loads enabled jobs and registers interval triggers. |
| 48 | 4. At run-time APScheduler calls mapped functions (`get_function_by_name`). |
| 49 | 5. Example job `invoke_alert_creation_collect`: |
| 50 | - runs alert auto-create route logic (`backend/app/schedulers/services/invoke_alert_creation.py`) |
| 51 | - updates `JobMetadata.last_success`. |
| 52 | 6. Manual operations (`/api/scheduler/...`) can run/pause/update/delete jobs. |
| 53 | |
| 54 | ## 4) Connector Verify + Use |
| 55 | |
| 56 | Verify dispatch path: |
| 57 | - `POST /api/connectors/verify/{id}` -> `backend/app/connectors/routes.py` |
| 58 | - dispatch map in `backend/app/connectors/services.py:get_connector_service` |
| 59 | |
| 60 | Flow: |
| 61 | 1. Frontend calls verify (`frontend/src/api/endpoints/connectors.ts`). |
| 62 | 2. Backend fetches connector row by ID, builds response model. |
| 63 | 3. Connector name is mapped to a service class in `service_map`. |
| 64 | 4. Service class calls connector-specific verifier in `backend/app/connectors/<service>/utils/universal.py`. |
| 65 | 5. DB updates `connector_verified` + `connector_last_updated`. |
| 66 | |
| 67 | Use path (runtime connector client): |
| 68 | 1. Feature route/service calls a connector client factory in `utils/universal.py`. |
| 69 | 2. Factory pulls credentials via `get_connector_info_from_db` (`backend/app/connectors/utils.py`). |
| 70 | 3. Downstream API requests run with those connector settings. |
| 71 | |
| 72 | ## 5) Alert -> Case |
| 73 | |
| 74 | Alert creation and case linking paths: |
| 75 | - Auto/manual alert creation routes: `backend/app/incidents/routes/incident_alert.py` |
| 76 | - Case creation routes: `backend/app/incidents/routes/db_operations.py` |
| 77 | - Case creation service: `backend/app/incidents/services/db_operations.py` |
| 78 | |
| 79 | Flow: |
| 80 | 1. Alert is ingested/created (`/incident_alert/create/manual` or `/incident_alert/create/auto`). |
| 81 | 2. Analyst (or workflow) calls `/incident_management/case/from-alert`. |
| 82 | 3. Backend creates `Case` using alert fields (`create_case_from_alert`). |
| 83 | 4. Backend creates join record in `CaseAlertLink` (`create_case_alert_link`). |
| 84 | 5. Case now references the originating alert for SOC workflows and reporting. |
| 85 | |
| 86 | ## 6) Artifact Upload to MinIO |
| 87 | |
| 88 | Two common paths: |
| 89 | - Generic upload: `/api/agent_data_store/upload` (`backend/app/data_store/data_store_routes.py`) |
| 90 | - Velociraptor collection upload: `backend/app/connectors/velociraptor/services/artifacts.py` |
| 91 | |
| 92 | Velociraptor-specific flow: |
| 93 | 1. Collection job runs and gets `flow_id`. |
| 94 | 2. `fetch_file_from_filestore` downloads zipped results locally. |
| 95 | 3. `upload_agent_artifact_file` uploads file to MinIO bucket `velociraptor-artifacts` with key `agent_id/flow_id/file.zip` (`backend/app/data_store/data_store_operations.py`). |
| 96 | 4. Metadata is stored in `AgentDataStore` table. |
| 97 | 5. UI/API can list/download/delete via `backend/app/data_store/data_store_routes.py`. |