@cryptotaxi247 / CoPilot / commits / 9b9c9b9c

mkdocs (#36)

taylor_socfortress committed Jul 15, 2023 at 16:20 UTC 9b9c9b9c0b485a02f93caa5e41e30b19f1a3bb1d
3 files changed +253 -4
backend/app/models/models.py
+1
@@ -109,6 +109,7 @@ class Connectors(db.Model):
109 "dfir-irs": True,
110 "velociraptor": True,
111 "sublime": True,
112 + "influxdb": True,
113 }
114
115 def __init__(
backend/docs/influxdb.md
+135 -2
@@ -2,20 +2,153 @@
2
3 ### <span style="color:blue">Alerts Model</span>
4
5 +This Python script is part of a backend service that interacts with a database to store and manage alerts coming from InfluxDB. It uses SQLAlchemy, a SQL toolkit and Object-Relational Mapping (ORM) system for Python, and Marshmallow, an ORM/ODM/framework-agnostic library for complex data types serialization, to define and manipulate a database model for the alerts.
6 +
7 +Here is a detailed breakdown of the code:
8 +
9 +## Import Statements
10 +
11 +The necessary libraries and modules are imported. These include `datetime` for handling date and time information, various types from `sqlalchemy` for defining database table structure, and the `db` and `ma` instances from the `app` module.
12 +
13 +## InfluxDBAlerts Class
14 +
15 +This class inherits from the SQLAlchemy's `Model` class and represents the `InfluxDBAlerts` table in the database. It has four columns: `id`, `check_name`, `message`, and `timestamp`.
16 +
17 +- `id`: This is the primary key of the table. It is an integer.
18 +- `check_name`: This column stores the name of the check that triggered the alert. It is a string of up to 1000 characters.
19 +- `message`: This column stores the message associated with the alert. It is also a string of up to 1000 characters.
20 +- `timestamp`: This column stores the time when the alert was generated. It is a `DateTime` object and by default, it's set to the time when a new row is created.
21 +
22 +The `__init__` method initializes a new instance of the class with `check_name` and `message`. The `__repr__` method returns a string representation of an instance of the class.
23 +
24 +## InfluxDBAlertsSchema Class
25 +
26 +This class inherits from Marshmallow's `Schema` class. It is used to serialize and deserialize instances of the `InfluxDBAlerts` class to and from Python dictionaries. This is useful for converting the model instances into a format that can be used in JSON APIs.
27 +
28 +The `Meta` class within `InfluxDBAlertsSchema` specifies the fields to include in the serialized output.
29 +
30 +- `InfluxDB_alert_schema` and `InfluxDB_alerts_schema`: These are instances of the `InfluxDBAlertsSchema` class. `InfluxDB_alert_schema` is used to serialize a single `InfluxDBAlerts` instance, while `InfluxDB_alerts_schema` is used to serialize a list of `InfluxDBAlerts` instances (note the `many=True` argument).
31 +
32 ::: app.models.influxdb_alerts
33 <br>
34
8 -### <span style="color:green">Checks Routes</span>
35 +### <span style="color:green">Routes</span>
36 +
37 +This Python script, `influxdb.py`, uses Flask to create a web server with several endpoints for handling HTTP requests related to InfluxDB checks and alerts.
38 +
39 +Here's a detailed description:
40 +
41 +## Import Statements
42 +
43 +It begins by importing required modules. These include types from the `typing` module for type annotations, `Blueprint` from Flask for creating a set of routes, `jsonify` and `request` from Flask for handling JSON responses and requests, respectively, `logger` from loguru for logging, and `InfluxDBAlertsService` and `InfluxDBChecksService` from `app.services.InfluxDB`.
44 +
45 +## Blueprint Creation
46
10 -### <span style="color:green">Alerts Routes</span>
47 +It then creates a `Blueprint` named `influxdb`. A Blueprint is a way to organize a group of related routes, and it's being used here to create routes for the InfluxDB-related endpoints.
48 +
49 +## Route Definitions
50 +
51 +Following this, it defines several routes:
52 +
53 +- `@bp.route("/influxdb/checks", methods=["GET"])`: This route responds to HTTP GET requests at the `/influxdb/checks` endpoint. It retrieves a list of all InfluxDB checks using the `InfluxDBChecksService` and returns them as a JSON response.
54 +
55 +- `@bp.route("/influxdb/checks/<check_id>", methods=["GET"])`: This route responds to HTTP GET requests at the `/influxdb/checks/<check_id>` endpoint. It retrieves the query of a specific InfluxDB check using the `InfluxDBChecksService` and returns it as a JSON response.
56 +
57 +- `@bp.route("/influxdb/alerts", methods=["GET"])`: This route responds to HTTP GET requests at the `/influxdb/alerts` endpoint. It retrieves a list of all alerts from the `influxdb_alerts` table using the `InfluxDBAlertsService` and returns them as a JSON response.
58 +
59 +- `@bp.route("/influxdb/alert", methods=["POST"])`: This route responds to HTTP POST requests at the `/influxdb/alert` endpoint. It stores an alert in the `influxdb_alerts` table. The endpoint is invoked by the InfluxDB alert webhook, which is configured in the InfluxDB UI. It validates the payload from the request, stores the alert if the payload is valid, and returns a JSON response indicating whether the storage was successful.
60 +
61 +## Error Handling
62 +
63 +If the payload of the alert to store is invalid, an exception is caught, an error message is logged, and returned as a JSON response with a 400 status code.
64
65 ::: app.routes.influxdb
66 <br>
67
68 ### <span style="color:red">Checks Services</span>
69
70 +This Python script, `checks.py`, is focused on interacting with InfluxDB to manage and retrieve checks. InfluxDB checks are a part of the monitoring and alerting system, which are scripts or queries that run at regular intervals to determine the health of your data.
71 +
72 +Here's a detailed description:
73 +
74 +## Import Statements
75 +
76 +The necessary libraries and modules are imported. These include types from the `typing` module for type annotations, the `requests` library for making HTTP requests, `logger` from loguru for logging, and the `UniversalService` from `app.services.InfluxDB`.
77 +
78 +## Custom Exceptions
79 +
80 +The script defines two custom exceptions, `InvalidPayloadError` and `ChecksCollectionError`, that are used for error handling in specific cases.
81 +
82 +## InfluxDBSession Class
83 +
84 +This class manages the connection and session to the InfluxDB server. It has methods to initialize the connection and to send GET requests to the server.
85 +
86 +## InfluxDBChecksService Class
87 +
88 +This class handles operations related to InfluxDB checks and interacts with the InfluxDB server via an `InfluxDBSession` object.
89 +
90 +The class has methods to initialize itself with the session, connector URL, and API key. It also has a class method to create an instance of itself using connector details.
91 +
92 +### collect_checks
93 +
94 +This method collects all checks from InfluxDB. It sends a GET request to the InfluxDB server and processes the response to return a list of checks. Each check is a dictionary with information such as the check's ID, name, type, status, and the time it was last triggered.
95 +
96 +### collect_check_query
97 +
98 +This method retrieves the query for a specific check from InfluxDB. It sends a GET request to the server and processes the response to return a dictionary with the query information.
99 +
100 +These classes and their methods provide a structured and organized way to interact with the InfluxDB server and manage InfluxDB checks.
101 +
102 ::: app.services.InfluxDB.checks
103
104 ### <span style="color:red">Alerts Services</span>
105
106 +This Python script, `alerts.py`, is focused on interacting with a database to manage and retrieve alerts that come from InfluxDB.
107 +
108 +Here's a detailed description:
109 +
110 +## Import Statements
111 +
112 +The necessary libraries and modules are imported. These include types from the `typing` module for type annotations, the `requests` library for making HTTP requests, `logger` from loguru for logging, `InfluxDBAlerts` from `app.models.influxdb_alerts`, and `UniversalService` from `app.services.InfluxDB`.
113 +
114 +## Custom Exceptions
115 +
116 +The script defines two custom exceptions, `InvalidPayloadError` and `ChecksCollectionError`, that are used for error handling in specific scenarios.
117 +
118 +## InfluxDBSession Class
119 +
120 +This class manages the connection and session to the InfluxDB server. It has methods to initialize the connection and to send GET requests to the server.
121 +
122 +## InfluxDBAlertsService Class
123 +
124 +This class handles operations related to InfluxDB alerts and interacts with the InfluxDB server via an `InfluxDBSession` object.
125 +
126 +The class has methods to initialize itself with the session, connector URL, and API key. It also has a class method to create an instance of itself using connector details.
127 +
128 +### validate_payload
129 +
130 +This method validates the payload received from the InfluxDB alert webhook. If the payload is valid, it returns the check name and message. If it is invalid, it raises an `InvalidPayloadError`.
131 +
132 +### store_alerts
133 +
134 +This method stores the alerts in the `influxdb_alerts` table in the database.
135 +
136 +### collect_alerts
137 +
138 +This method collects alerts from the `influxdb_alerts` table in the database. It checks whether the InfluxDB connector details were successfully collected, then collects the alerts from the database. It returns a dictionary containing the success status, a message, and potentially the alert details.
139 +
140 +### \_are_influxdb_details_collected
141 +
142 +This private method checks whether the details for the InfluxDB connector were successfully collected.
143 +
144 +### \_collect_alerts_from_db
145 +
146 +This private method collects alerts from the `influxdb_alerts` table in the database. It returns a dictionary containing the success status, a message, and potentially the alert details.
147 +
148 +These classes and their methods provide a structured and organized way to interact with the InfluxDB server and manage InfluxDB alerts.
149 +
150 ::: app.services.InfluxDB.alerts
151 +
152 +### <span style="color:red">Universal Services</span>
153 +
154 +::: app.services.InfluxDB.universal
backend/docs/smtp.md
+117 -2
@@ -2,22 +2,137 @@
2
3 ### <span style="color:blue">EmailCredentials Model</span>
4
5 +This Python script, `smtp.py`, is centered around creating a SQLAlchemy model for storing email credentials and SMTP settings, and a Marshmallow schema for serializing and deserializing instances of the model.
6 +
7 +Here is a detailed breakdown:
8 +
9 +## Import Statements
10 +
11 +The necessary libraries and modules are imported. These include `datetime` for handling date and time information, various types from `sqlalchemy` for defining database table structure, and the `db` and `ma` instances from the `app` module.
12 +
13 +## EmailCredentials Class
14 +
15 +This class inherits from SQLAlchemy's `Model` class and represents the `EmailCredentials` table in the database. It has six columns: `id`, `email`, `password`, `smtp_server`, `smtp_port`, and `timestamp`.
16 +
17 +- `id`: This is the primary key of the table. It is an integer.
18 +- `email`: This column stores the user's email. It is a string of up to 100 characters, and it cannot be null.
19 +- `password`: This column stores the password of the user's email. It is a string of up to 100 characters, and it cannot be null.
20 +- `smtp_server`: This column stores the SMTP server address. It is a string of up to 100 characters, and it cannot be null.
21 +- `smtp_port`: This column stores the SMTP port. It is an integer, and it cannot be null.
22 +- `timestamp`: This column stores the time when the email credential was created. It is a `DateTime` object and by default, it's set to the time when a new row is created.
23 +
24 +The `__init__` method initializes a new instance of the class with `email`, `password`, `smtp_server`, and `smtp_port`. The `__repr__` method returns a string representation of an instance of the class.
25 +
26 +## EmailCredentialsSchema Class
27 +
28 +This class inherits from Marshmallow's `Schema` class. It is used to serialize and deserialize instances of the `EmailCredentials` class to and from Python dictionaries. This is useful for converting the model instances into a format that can be used in JSON APIs.
29 +
30 +The `Meta` class within `EmailCredentialsSchema` specifies the fields to include in the serialized output.
31 +
32 +- `email_credentials_schema` and `email_credentials_schemas`: These are instances of the `EmailCredentialsSchema` class. `email_credentials_schema` is used to serialize a single `EmailCredentials` instance, while `email_credentials_schemas` is used to serialize a list of `EmailCredentials` instances (note the `many=True` argument).
33 +
34 ::: app.models.smtp
35 <br>
36
8 -### <span style="color:green">Checks Routes</span>
37 +### <span style="color:green">SMTP Routes</span>
38 +
39 +This Python script, `smtp.py`, uses Flask to create a web server with several endpoints for handling HTTP requests related to SMTP (Simple Mail Transfer Protocol) credentials and sending reports via email.
40 +
41 +Here's a detailed description:
42
10 -### <span style="color:green">Alerts Routes</span>
43 +## Import Statements
44 +
45 +It begins by importing required modules. These include `Blueprint`, `jsonify`, and `request` from Flask for creating a set of routes and handling JSON responses and requests, `logger` from loguru for logging, and `EmailReportSender` and `UniversalEmailCredentials` from `app.services.smtp`.
46 +
47 +## Blueprint Creation
48 +
49 +It then creates a `Blueprint` named `smtp`. A Blueprint is a way to organize a group of related routes, and it's being used here to create routes for the SMTP-related endpoints.
50 +
51 +## Route Definitions
52 +
53 +Following this, it defines several routes:
54 +
55 +- `@bp.route("/smtp/credential", methods=["POST"])`: This route responds to HTTP POST requests at the `/smtp/credential` endpoint. It stores SMTP credentials in the `smtp_credentials` table. The endpoint expects a JSON payload with the email, password, SMTP server, and SMTP port. If the payload is valid, it uses `UniversalEmailCredentials.create()` to store the credentials and returns a JSON response indicating whether the storage was successful.
56 +
57 +- `@bp.route("/smtp/credentials", methods=["GET"])`: This route responds to HTTP GET requests at the `/smtp/credentials` endpoint. It retrieves a list of all SMTP credentials from the `smtp_credentials` table using the `UniversalEmailCredentials.read_all()` method and returns them as a JSON response.
58 +
59 +- `@bp.route("/smtp/report", methods=["POST"])`: This route responds to HTTP POST requests at the `/smtp/report` endpoint. It sends a report via email. The endpoint expects a JSON payload with the recipient's email address. If the payload is valid, it uses `EmailReportSender(to_email).send_email_with_pdf()` to send the report and returns a JSON response indicating whether the sending was successful.
60 +
61 +## Error Handling
62 +
63 +If the payload of the request to store SMTP credentials or send a report is invalid, an error message is logged and returned as a JSON response with a 400 status code.
64
65 ::: app.routes.smtp
66 <br>
67
68 ### <span style="color:red">Create Report Services</span>
69
70 +This Python script, `create_report.py`, generates a PDF report about alerts with two types of charts (bar and pie) and exports the report to a PDF file. It uses matplotlib for creating charts, reportlab for creating the PDF, and loguru for logging. It fetches the alerts data from an `AlertsService`.
71 +
72 +Here's a detailed description:
73 +
74 +## Import Statements
75 +
76 +It begins by importing required modules. These include `urllib.request` for downloading files, `matplotlib` for creating charts, `loguru` for logging, and `types` from the `typing` module for type annotations. `reportlab` is used to create the PDF.
77 +
78 +## Functions
79 +
80 +### fetch_alert_data(service, fetch_func)
81 +
82 +Fetches alert data using the provided function and logs the data.
83 +
84 +### create_bar_chart(alerts: dict, title: str, output_filename: str)
85 +
86 +Creates a horizontal bar chart of alerts by host and saves it to a file.
87 +
88 +### create_pie_chart(alerts: dict, title: str, output_filename: str)
89 +
90 +Creates a pie chart of alerts by rule and saves it to a file.
91 +
92 +### create_pdf(title: str, image_filenames: List[str], pdf_filename: str)
93 +
94 +Creates a PDF containing images (the bar chart and the pie chart) and the SOC Fortress logo, which is downloaded from a URL. The PDF is saved to a file.
95 +
96 +### create_alerts_report_pdf()
97 +
98 +This function uses the `AlertsService` to fetch alerts data, then creates a bar chart of alerts by host and a pie chart of alerts by rule. It finally creates a PDF report that includes these charts.
99 +
100 +## Matplotlib Backend Setting
101 +
102 +The Agg backend of matplotlib is used, which is a non-interactive backend suitable for scripts and web servers. This should resolve the "main thread is not in main loop" issue as it bypasses the need for tkinter.
103 +
104 +## Execution of Report Generation
105 +
106 +The last line of the script calls the `create_alerts_report_pdf()` function to generate the report when the script is run.
107 +
108 ::: app.services.smtp.create_report
109
110 ### <span style="color:red">Send Report Services</span>
111
112 +This Python script, `send_report.py`, is designed to send an email report with PDF attachments. It uses the `smtplib` library for sending emails, the `email` library for creating email messages with attachments, and the `create_alerts_report_pdf` function from the `app.services.smtp.create_report` module for generating the PDF report.
113 +
114 +Here's a detailed description:
115 +
116 +## Import Statements
117 +
118 +It begins by importing required modules. These include `smtplib` for the SMTP client session object that can be used to send mail, `email` for managing email messages, and `typing` for type annotations.
119 +
120 +## EmailReportSender Class
121 +
122 +This class is used to send an email report with PDF attachments. It has the following methods:
123 +
124 +- `__init__(self, to_email: str)`: Constructor for the `EmailReportSender` class. It initializes the class with the recipient's email address.
125 +
126 +- `_get_credentials(self) -> dict`: Fetches the email credentials.
127 +
128 +- `create_email_message(self, subject: str, body: str) -> MIMEMultipart`: Creates an email message with the provided subject and body.
129 +
130 +- `attach_pdfs(self, msg: MIMEMultipart, filenames: List[str]) -> MIMEMultipart`: Attaches PDF files to an email message.
131 +
132 +- `send_email_with_pdf(self)`: Sends an email with a PDF report. It generates the PDF report, creates an email message, attaches the report to the message, and then sends the email. It returns a dictionary containing a message describing the result of the operation and a success indicator.
133 +
134 +These methods collectively allow the `EmailReportSender` class to generate a report, create an email message with the report attached, and send the email message. They use helper methods to fetch email credentials, create the email message, attach the PDFs, and send the email.
135 +
136 ::: app.services.smtp.send_report
137
138 ### <span style="color:red">Universal Services</span>