@cryptotaxi247 / CoPilot / commits / 611ceda1

chore: reconcile requirements.in with packages the code actually needs (#843)

Today, requirements.in is a partial wishlist — it lists 25 packages while requirements.txt installs 175. Many top-level deps the code imports directly (alembic, aiomysql, miniopy-async, cryptography, jinja2, pyotp, qrcode, pdfkit, …) are missing from .in and only land in .txt because someone added them to requirements.txt by hand at some point. Drift made requirements.in untrustworthy as input to pip-compile (which is the whole point of having a .in file). This PR fills in the gaps so pip-compile against requirements.in produces a sensible closure. No changes to requirements.txt — the running deployment is unaffected. Methodology: 1. AST-walked every .py file in backend/ to enumerate top-level imports (excluding stdlib + internal `app.*`). 2. Mapped each import to its installed pip distribution via importlib.metadata.packages_distributions() inside the running backend container. 3. Cross-checked: 16 distributions are imported but missing from .in. 4. Added one CLI-only dep that imports never reveal: ScoutSuite (the `scout` CLI is invoked via subprocess from app/integrations/scoutsuite/services/scoutsuite.py). 5. Added two config-required deps that the code references via SQLAlchemy URL strings rather than `import`: - aiomysql (the `mysql+aiomysql://` async driver) - PyMySQL (used by alembic + DB bootstrap) 6. Promoted `httpx` -> `httpx[http2]` and `uvicorn` -> `uvicorn[standard]` to preserve the extras already pinned in the current closure (h2/qh3 for httpx; uvloop/httptools/websockets/watchfiles for uvicorn). Without these, pip-compile would silently drop them. 7. Verified by running `pip install -r requirements.in -c <stripped requirements.txt>` in a clean python:3.11 container and comparing the resulting `pip freeze` against requirements.txt. After this PR, the only packages a future pip-compile would *drop* from requirements.txt are pre-existing dead deps: deepdiff, licensing, pySigma, redis, shufflepy, singul, numpy, Pillow, openpyxl, plus the orphan transitives left over from previously-removed Celery and STIX-1 stack (amqp, kombu, billiard, vine, click-didyoumean, click-plugins, click-repl, cybox, maec, mixbox, stix*, stixmarx, taxii2-client, …). Cleaning those up is intentionally left to a follow-up PR so the diff here stays focused on the .in reconcile. Verified locally: - The new .in installs cleanly in a fresh container. - The freeze output is a strict superset of every package the backend currently imports. - requirements.txt is untouched, so the running build pipeline is unaffected. Co-authored-by: taylor_socfortress <taylor.walton@socfortress.co> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

taylorcopilot committed May 7, 2026 at 15:01 UTC 611ceda13a131824bbff78f9075c60ed83f58285
1 file changed +21 -2
backend/requirements.in
+21 -2
@@ -1,26 +1,45 @@
1 aiofiles
2 +aiohttp
3 +aiomysql
4 aiosqlite
5 +alembic
6 apscheduler
7 +asyncgelf
8 bcrypt
9 cortex4py
10 +cryptography
11 +docxtpl
12 elasticsearch7==7.10.1
13 environs
14 fastapi
15 grafana-client
10 -httpx
16 +grpcio
17 +httpx[http2]
18 influxdb-client[async]
19 +Jinja2
20 loguru
21 +miniopy-async
22 +packaging
23 passlib
24 passlib[bcrypt]
25 +pdfkit
26 playwright
27 pydantic[email]
28 PyJWT
29 +PyMySQL
30 +pyotp
31 +python-dotenv
32 python-magic
33 python-multipart
34 pyvelociraptor~=0.1
35 +PyYAML
36 +qrcode
37 regex
38 requests
39 +ScoutSuite
40 +SQLAlchemy
41 sqlmodel
24 -uvicorn
42 +starlette
43 +uvicorn[standard]
44 werkzeug
45 xmltodict