main
md 284 lines 22.5 KB
Rendered Raw
1 # Database Schema (AI Agent-Oriented)
2
3 This document summarizes the **current schema** for AI-agent workflows, using **Alembic migrations as source of truth** in `backend/alembic/versions/*.py`.
4
5 - Current migration head in this repo: `fb51d610b306` (`backend/alembic/versions/fb51d610b306_add_github_audit_tables.py`)
6 - Base migration: `bdf40d064ed1` (`backend/alembic/versions/bdf40d064ed1_initial_database_migration.py`)
7
8 ## Practical Domain Map
9
10 For typical agent change work, these domains are most relevant:
11
12 - Connectors and integration metadata: `connectors*`, `available_*`, `customer_*_connectors*`, `customer_*integrations*`, `integration_*`, `network_connectors_*`, `custom_alert_creation_*`, `monitoring_alerts`, `sigma_queries`, `github_audit_*`
13 - Auth / users / roles: `user`, `role`, `user_customer_access`, `user_tag_access`, `role_tag_access`
14 - Incidents (alerts/cases/tags/comments): all `incident_management_*` tables
15 - Scheduler/job metadata: `scheduled_job_metadata`, `schedulerjob`, `index_snapshot_schedules`
16 - Agent data store / artifacts / reports: `agent_datastore`, `incident_management_case_datastore`, `incident_management_case_report_template_datastore`, `vulnerability_reports`, `sca_reports`, `agent_vulnerabilities`
17
18 ## Critical relationship graph
19
20 Compact relationship views for incident workflows and access controls.
21
22 ### Alerts -> assets (with link fields)
23
24 ```text
25 incident_management_alert
26 id (PK)
27 |
28 | 1-to-many via incident_management_asset.alert_linked
29 v
30 incident_management_asset
31 alert_linked (FK -> incident_management_alert.id)
32 alert_context_id (FK -> incident_management_alertcontext.id)
33 index_name, index_id (origin pointer into SIEM index document)
34 ```
35
36 ### Cases ↔ case-alert links ↔ alerts
37
38 ```text
39 incident_management_case incident_management_alert
40 id (PK) id (PK)
41 \ /
42 \ /
43 +-- incident_management_casealertlink --+
44 case_id (FK -> incident_management_case.id)
45 alert_id (FK -> incident_management_alert.id)
46 PK(case_id, alert_id)
47 ```
48
49 ### Tags (alert/tag join)
50
51 ```text
52 incident_management_alert incident_management_alerttag
53 id (PK) id (PK), tag
54 \ /
55 \ /
56 +-- incident_management_alert_to_tag --+
57 alert_id (FK -> incident_management_alert.id)
58 tag_id (FK -> incident_management_alerttag.id)
59 PK(alert_id, tag_id)
60 ```
61
62 ### IoCs (alert/ioc join)
63
64 ```text
65 incident_management_alert incident_management_ioc
66 id (PK) id (PK), value/type/description
67 \ /
68 \ /
69 +-- incident_management_alert_to_ioc --+
70 alert_id (FK -> incident_management_alert.id)
71 ioc_id (FK -> incident_management_ioc.id)
72 PK(alert_id, ioc_id)
73 ```
74
75 ### Comments (alert comments + case comments)
76
77 ```text
78 incident_management_comment
79 alert_id (FK -> incident_management_alert.id)
80 comment, user_name, created_at
81
82 incident_management_case_comment
83 case_id (FK -> incident_management_case.id)
84 comment, user_name, created_at
85 ```
86
87 ### Case datastore + report template datastore
88
89 ```text
90 incident_management_case
91 id (PK)
92 |
93 | 1-to-many via incident_management_case_datastore.case_id
94 v
95 incident_management_case_datastore
96 case_id (FK -> incident_management_case.id)
97 bucket_name, object_key, file_name, file_hash, upload_time
98
99 incident_management_case_report_template_datastore
100 (global report templates; no case FK)
101 report_template_name, bucket_name, object_key, file_name, file_hash, upload_time
102 ```
103
104 ### Tag access control and alert visibility
105
106 ```text
107 incident_management_tag_access_settings
108 enabled, untagged_alert_behavior, default_tag_id (FK -> incident_management_alerttag.id)
109
110 user_tag_access (user_id, tag_id) role_tag_access (role_id, tag_id)
111 \ /
112 +------ allowed tag ids per identity ------+
113 |
114 incident_management_alert_to_tag (alert_id, tag_id)
115 |
116 incident_management_alert
117 ```
118
119 When tag access control is enabled, alert visibility is constrained by the tag IDs reachable through `user_tag_access` and/or `role_tag_access` joined through `incident_management_alert_to_tag`. `incident_management_tag_access_settings` controls whether this filtering is active and what happens for untagged alerts (`untagged_alert_behavior`, optional `default_tag_id` fallback).
120
121 **Tag access enforcement location (code pointers)**
122 - Core tag RBAC logic: `backend/app/incidents/middleware/tag_access.py` (`TagAccessHandler`)
123 - `is_tag_rbac_enabled()` (global enable/disable)
124 - `build_alert_query_filters()` (computes accessible tags + untagged behavior)
125 - `check_alert_tag_access()` / `can_user_access_alert()` (per-alert decision)
126 - Applied in incident DB query layer:
127 - `backend/app/incidents/services/db_operations.py` uses `tag_access_handler.build_alert_query_filters()` to add SQL `exists()` conditions when counting/listing alerts.
128
129 ### SIEM data origin + query pattern
130
131 - Graylog alerting uses the `gl-events` index pattern (for example `gl-events*` in query flows).
132 - Those Graylog alert documents live in Wazuh indexer storage (OpenSearch-backed).
133 - Wazuh indexer is the backing SIEM event store across event sources (endpoints, O365 integrations, network connectors, and other ingested streams).
134 - CoPilot commonly resolves and displays SIEM records by querying Wazuh indexer with `index_name` plus `index_id`.
135 - Code pointers:
136 - `backend/app/connectors/wazuh_indexer/routes/alerts.py`
137 - `backend/app/connectors/wazuh_indexer/services/alerts.py`
138 - `backend/app/routers/wazuh_indexer.py`
139 - `frontend/src/api/endpoints/alerts.ts`
140
141 ## Table Inventory (Alembic-Derived)
142
143 ### Customer, Auth, and Core Platform
144
145 | Table | PK | Important columns | Foreign keys | Model file(s) |
146 |---|---|---|---|---|
147 | `customers` | `id` | `customer_code`, `customer_name`, contact/address fields, `created_at` | None | `backend/app/db/universal_models.py` (`Customers`) |
148 | `customersmeta` | `id` | `customer_code`, Graylog/Grafana/Wazuh metadata, `customer_meta_portainer_stack_id` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`CustomersMeta`) |
149 | `customer_provisioning_default_settings` | `id` | `cluster_name`, `cluster_key`, `master_ip`, `grafana_url`, `wazuh_worker_hostname` | None | `backend/app/customer_provisioning/models/default_settings.py` |
150 | `user` | `id` | `username`, `password`, `email`, `created_at`, `role_id` | `role_id -> role.id` | `backend/app/auth/models/users.py` (`User`) |
151 | `role` | `id` | `name`, `description` | None | `backend/app/auth/models/users.py` (`Role`) |
152 | `user_customer_access` | `id` | `user_id`, `customer_code`, `created_at` | `user_id -> user.id`, `customer_code -> customers.customer_code` | `backend/app/auth/models/users.py` |
153 | `user_tag_access` | `id` | `user_id`, `tag_id`, `created_at` | `user_id -> user.id`, `tag_id -> incident_management_alerttag.id` | `backend/app/auth/models/users.py` |
154 | `role_tag_access` | `id` | `role_id`, `tag_id`, `created_at` | `role_id -> role.id`, `tag_id -> incident_management_alerttag.id` | `backend/app/auth/models/users.py` |
155 | `license` | `id` | `license_key`, customer/company identity fields | None | `backend/app/db/universal_models.py` (`License`) |
156 | `license_cache` | `id` | `license_key`, `feature_name`, `is_enabled`, `cached_at`, `expires_at`, `license_data` | None | `backend/app/db/universal_models.py` (`LicenseCache`) |
157 | `log_entries` | `id` | `timestamp`, `event_type`, `user_id`, `route`, `status_code`, `message` | None | `backend/app/db/universal_models.py` (`LogEntry`) |
158 | `customer_portal_settings` | `id` | `title`, `logo_base64`, `logo_mime_type`, `updated_at`, `updated_by` | None | `backend/app/db/universal_models.py` (`CustomerPortalSettings`) |
159
160 ### Agents, Vulnerability, and Artifact/Data Store
161
162 | Table | PK | Important columns | Foreign keys | Model file(s) |
163 |---|---|---|---|---|
164 | `agents` | `id` | `agent_id`, host/OS/status fields, `velociraptor_*`, `customer_code`, `quarantined`, `velociraptor_org` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`Agents`) |
165 | `agent_datastore` | `id` | `agent_id`, `velociraptor_id`, `artifact_name`, `flow_id`, storage columns (`bucket_name`,`object_key`,`file_name`), `file_hash`, `status` | `agent_id -> agents.agent_id` | `backend/app/db/universal_models.py` (`AgentDataStore`) |
166 | `agent_vulnerabilities` | `id` | `cve_id`, `severity`, `title`, `status`, `discovered_at`, `agent_id`, `customer_code` | `agent_id -> agents.agent_id`, `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`AgentVulnerabilities`) |
167 | `vulnerability_reports` | `id` | `report_name`, `customer_code`, storage columns, `generated_at`, vulnerability counters, `status` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`VulnerabilityReport`) |
168 | `sca_reports` | `id` | `report_name`, `customer_code`, storage columns, `generated_at`, SCA counters, `status` | `customer_code -> customers.customer_code` | `backend/app/db/universal_models.py` (`SCAReport`) |
169
170 ### Scheduler and Job Metadata
171
172 | Table | PK | Important columns | Foreign keys | Model file(s) |
173 |---|---|---|---|---|
174 | `scheduled_job_metadata` | `id` | `job_id`, `last_success`, `time_interval`, `extra_data`, `enabled`, `job_description` | None | `backend/app/schedulers/models/scheduler.py` (`JobMetadata`) |
175 | `schedulerjob` | `id` | `next_run_time`, `job_state` | None | `backend/app/db/universal_models.py` (`SchedulerJob`) |
176 | `index_snapshot_schedules` | `id` | schedule metadata (`name`, `index_pattern`, `repository`), retention/last execution fields | None | `backend/app/connectors/wazuh_indexer/models/snapshot_and_restore.py` |
177
178 ### Connectors and Integrations
179
180 | Table | PK | Important columns | Foreign keys | Model file(s) |
181 |---|---|---|---|---|
182 | `connectors` | `id` | connector identity/endpoint/auth fields, capability flags, `connector_enabled` | None | `backend/app/connectors/models.py` (`Connectors`) |
183 | `connectorhistory` | `id` | `connector_id`, `change_timestamp`, `change_description` | `connector_id -> connectors.id` | `backend/app/connectors/models.py` (`ConnectorHistory`) |
184 | `available_integrations` | `id` | `integration_name`, `description`, `integration_details` | None | `backend/app/integrations/models/customer_integration_settings.py` |
185 | `available_integrations_auth_keys` | `id` | `integration_id`, `integration_name`, `auth_key_name` | `integration_id -> available_integrations.id` | `backend/app/integrations/models/customer_integration_settings.py` |
186 | `customer_integrations` | `id` | `customer_code`, `integration_service_id`, `integration_service_name`, `deployed` | None | `backend/app/integrations/models/customer_integration_settings.py` |
187 | `integration_services` | `id` | `service_name`, `auth_type` | None | `backend/app/integrations/models/customer_integration_settings.py` |
188 | `integration_subscriptions` | `id` | `customer_id`, `integration_service_id` | `customer_id -> customer_integrations.id`, `integration_service_id -> integration_services.id` | `backend/app/integrations/models/customer_integration_settings.py` |
189 | `integration_configs` | `id` | `integration_service_id`, `config_key`, `config_value` | `integration_service_id -> integration_services.id` | `backend/app/integrations/models/customer_integration_settings.py` |
190 | `integration_auth_keys` | `id` | `subscription_id`, `auth_key_name`, `auth_value` | `subscription_id -> integration_subscriptions.id` | `backend/app/integrations/models/customer_integration_settings.py` |
191 | `customer_integrations_meta` | `id` | Graylog/Grafana metadata (`graylog_*`, `grafana_*`, `grafana_datasource_uid`) | None | `backend/app/integrations/models/customer_integration_settings.py` |
192 | `available_network_connectors` | `id` | `network_connector_name`, `description`, `network_connector_details` | None | `backend/app/network_connectors/models/network_connectors.py` |
193 | `available_network_connectors_keys` | `id` | `network_connector_id`, `network_connector_name`, `auth_key_name` | `network_connector_id -> available_network_connectors.id` | `backend/app/network_connectors/models/network_connectors.py` |
194 | `customer_network_connectors` | `id` | `customer_code`, `network_connector_service_id`, `network_connector_service_name`, `deployed` | None | `backend/app/network_connectors/models/network_connectors.py` |
195 | `network_connectors_services` | `id` | `service_name`, `auth_type` | None | `backend/app/network_connectors/models/network_connectors.py` |
196 | `network_connectors_subscriptions` | `id` | `customer_id`, `network_connectors_service_id` | `customer_id -> customer_network_connectors.id`, `network_connectors_service_id -> network_connectors_services.id` | `backend/app/network_connectors/models/network_connectors.py` |
197 | `network_connectors_configs` | `id` | `network_connector_service_id`, `config_key`, `config_value` | `network_connector_service_id -> network_connectors_services.id` | `backend/app/network_connectors/models/network_connectors.py` |
198 | `network_connectors_keys` | `id` | `subscription_id`, `auth_key_name`, `auth_value` | `subscription_id -> network_connectors_subscriptions.id` | `backend/app/network_connectors/models/network_connectors.py` |
199 | `customer_network_connectors_meta` | `id` | Graylog/Grafana connector metadata (`graylog_*`, `grafana_*`, `grafana_datasource_uid`) | None | `backend/app/network_connectors/models/network_connectors.py` |
200 | `custom_alert_creation_settings` | `id` | customer-wide alert-creation settings, `nvd_url`, custom integration URLs | None | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
201 | `custom_alert_creation_event_order` | `id` | `alert_creation_settings_id`, `order_label` | `alert_creation_settings_id -> custom_alert_creation_settings.id` | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
202 | `custom_alert_creation_condition` | `id` | `event_order_id`, `field_name`, `field_value` | `event_order_id -> custom_alert_creation_event_order.id` | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
203 | `custom_alert_creation_event_config` | `id` | `event_order_id`, `event_id`, `field`, `value` | `event_order_id -> custom_alert_creation_event_order.id` | `backend/app/integrations/alert_creation_settings/models/alert_creation_settings.py` |
204 | `monitoring_alerts` | `id` | `alert_id`, `alert_index`, `customer_code`, `alert_source` | None | `backend/app/integrations/monitoring_alert/models/monitoring_alert.py` |
205 | `sigma_queries` | `id` | `rule_name`, `rule_query`, `active`, `time_interval`, execution timestamps | None | `backend/app/connectors/wazuh_indexer/models/sigma.py` |
206
207 ### Incident Management (Alerts, Cases, Tags, Comments)
208
209 | Table | PK | Important columns | Foreign keys | Model file(s) |
210 |---|---|---|---|---|
211 | `incident_management_alert` | `id` | `alert_name`, `alert_description`, `status`, `alert_creation_time`, `customer_code`, `source`, `assigned_to`, `escalated` | None | `backend/app/incidents/models.py` (`Alert`) |
212 | `incident_management_alertcontext` | `id` | `source`, `context` (JSON) | None | `backend/app/incidents/models.py` (`AlertContext`) |
213 | `incident_management_asset` | `id` | `alert_linked`, `asset_name`, `alert_context_id`, `agent_id`, `customer_code`, `index_name`, `index_id` | `alert_linked -> incident_management_alert.id`, `alert_context_id -> incident_management_alertcontext.id` | `backend/app/incidents/models.py` (`Asset`) |
214 | `incident_management_comment` | `id` | `alert_id`, `comment`, `user_name`, `created_at` | `alert_id -> incident_management_alert.id` | `backend/app/incidents/models.py` (`Comment`) |
215 | `incident_management_case` | `id` | `case_name`, `case_description`, `case_creation_time`, `case_status`, `case_closed_time`, `assigned_to`, `customer_code`, `notification_invoked_number`, `escalated` | None | `backend/app/incidents/models.py` (`Case`) |
216 | `incident_management_casealertlink` | composite: (`case_id`, `alert_id`) | link table case↔alert | `case_id -> incident_management_case.id`, `alert_id -> incident_management_alert.id` | `backend/app/incidents/models.py` (`CaseAlertLink`) |
217 | `incident_management_case_comment` | `id` | `case_id`, `comment`, `user_name`, `created_at` | `case_id -> incident_management_case.id` | `backend/app/incidents/models.py` (`CaseComment`) |
218 | `incident_management_alerttag` | `id` | `tag` | None | `backend/app/incidents/models.py` (`AlertTag`) |
219 | `incident_management_alert_to_tag` | composite: (`alert_id`, `tag_id`) | link table alert↔tag | `alert_id -> incident_management_alert.id`, `tag_id -> incident_management_alerttag.id` | `backend/app/incidents/models.py` (`AlertToTag`) |
220 | `incident_management_ioc` | `id` | `value`, `type`, `description` | None | `backend/app/incidents/models.py` (`IoC`) |
221 | `incident_management_alert_to_ioc` | composite: (`alert_id`, `ioc_id`) | link table alert↔IoC | `alert_id -> incident_management_alert.id`, `ioc_id -> incident_management_ioc.id` | `backend/app/incidents/models.py` (`AlertToIoC`) |
222 | `incident_management_fieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`FieldName`) |
223 | `incident_management_assetfieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`AssetFieldName`) |
224 | `incident_management_timestampfieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`TimestampFieldName`) |
225 | `incident_management_alerttitlefieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`AlertTitleFieldName`) |
226 | `incident_management_iocfieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`IoCFieldName`) |
227 | `incident_management_customercodefieldname` | `id` | `source`, `field_name` | None | `backend/app/incidents/models.py` (`CustomerCodeFieldName`) |
228 | `incident_management_notification` | `id` | `customer_code`, `shuffle_workflow_id`, `enabled` | None | `backend/app/incidents/models.py` (`Notification`) |
229 | `incident_management_case_datastore` | `id` | `case_id`, storage fields (`bucket_name`,`object_key`,`file_name`), `upload_time`, `file_hash` | `case_id -> incident_management_case.id` | `backend/app/incidents/models.py` (`CaseDataStore`) |
230 | `incident_management_case_report_template_datastore` | `id` | `report_template_name`, storage fields, `upload_time`, `file_hash` | None | `backend/app/incidents/models.py` (`CaseReportTemplateDataStore`) |
231 | `incident_management_tag_access_settings` | `id` | `enabled`, `untagged_alert_behavior`, `default_tag_id`, `updated_at`, `updated_by` | `default_tag_id -> incident_management_alerttag.id` | `backend/app/incidents/models.py` (`TagAccessSettings`) |
232 | `incident_management_velo_sigma_exclusion` | `id` | `name`, `field_matches` (JSON), `channel`, `title`, `customer_code`, `created_by`, `enabled` | None | `backend/app/incidents/models.py` (`VeloSigmaExclusion`) |
233
234 ### GitHub Audit (Added at Head)
235
236 | Table | PK | Important columns | Foreign keys | Model file(s) |
237 |---|---|---|---|---|
238 | `github_audit_config` | `id` | customer/org token/configuration, filters (JSON), notification and score threshold fields | None | `backend/app/integrations/github_audit/model.py` |
239 | `github_audit_check_exclusion` | `id` | `config_id`, `customer_code`, check/resource selectors, approval/expiry, `enabled` | `config_id -> github_audit_config.id` | `backend/app/integrations/github_audit/model.py` |
240 | `github_audit_report` | `id` | `config_id`, organization/report metadata, summary counts, `status`, report JSON blobs | `config_id -> github_audit_config.id` | `backend/app/integrations/github_audit/model.py` |
241 | `github_audit_baseline` | `id` | `config_id`, `customer_code`, baseline definition, expected checks (JSON), `baseline_report_id`, `is_active` | `config_id -> github_audit_config.id`, `baseline_report_id -> github_audit_report.id` | `backend/app/integrations/github_audit/model.py` |
242
243 ## Quick Model Scan: Tables Not Obvious From Alembic
244
245 The following SQLModel tables are defined in code but do **not** appear in `backend/alembic/versions/*.py` migrations. Treat them as drift candidates / runtime-created tables unless there is an out-of-band migration process.
246
247 | Table (in SQLModel) | Model file |
248 |---|---|
249 | `sublimealerts` | `backend/app/connectors/sublime/models/alerts.py` (`SublimeAlerts`) |
250 | `flaggedrule` | `backend/app/connectors/sublime/models/alerts.py` (`FlaggedRule`) |
251 | `mailbox` | `backend/app/connectors/sublime/models/alerts.py` (`Mailbox`) |
252 | `triggeredaction` | `backend/app/connectors/sublime/models/alerts.py` (`TriggeredAction`) |
253 | `sender` | `backend/app/connectors/sublime/models/alerts.py` (`Sender`) |
254 | `recipient` | `backend/app/connectors/sublime/models/alerts.py` (`Recipient`) |
255 | `disabledrule` | `backend/app/connectors/wazuh_manager/models/rules.py` (`DisabledRule`) |
256 | `sap_siem_multiple_logins` | `backend/app/integrations/sap_siem/models/sap_siem.py` (`SapSiemMultipleLogins`) |
257
258 ## Where To Change Schema
259
260 ### Source of truth locations
261
262 - Alembic migrations: `backend/alembic/versions/`
263 - Alembic env/config: `backend/alembic/env.py`
264 - SQLModel definitions commonly touched:
265 - `backend/app/db/universal_models.py`
266 - `backend/app/incidents/models.py`
267 - `backend/app/auth/models/users.py`
268 - `backend/app/network_connectors/models/network_connectors.py`
269 - `backend/app/integrations/models/customer_integration_settings.py`
270
271 ### Practical workflow
272
273 1. Update or add SQLModel fields/classes in the relevant model file.
274 2. Generate a migration under `backend/alembic/versions/` (or author manually if needed).
275 3. Review migration `upgrade()` and `downgrade()` carefully (FK names, nullable transitions, indexes).
276 4. Apply migration locally and run tests.
277 5. Update this document if table shape/ownership changes.
278
279 ### Notes for agent changes
280
281 - Prefer extending existing domain tables over creating parallel tables when possible (especially incidents and connector metadata).
282 - For incident workflows, changes usually involve: `incident_management_alert`, `incident_management_case`, link tables (`*_to_*`), and optional datastore tables.
283 - For connector onboarding, changes usually involve: `available_*`, `customer_*`, `*_services`, `*_subscriptions`, `*_configs`, `*_keys`, and `*_meta` tables.
284 - For scheduler automation, coordinate changes between `scheduled_job_metadata`, `schedulerjob`, and domain-specific tables storing job outcomes.