chore: reconcile requirements.in with packages the code actually needs (#843)
Today, requirements.in is a partial wishlist — it lists 25 packages while requirements.txt installs 175. Many top-level deps the code imports directly (alembic, aiomysql, miniopy-async, cryptography, jinja2, pyotp, qrcode, pdfkit, …) are missing from .in and only land in .txt because someone added them to requirements.txt by hand at some point. Drift made requirements.in untrustworthy as input to pip-compile (which is the whole point of having a .in file). This PR fills in the gaps so pip-compile against requirements.in produces a sensible closure. No changes to requirements.txt — the running deployment is unaffected. Methodology: 1. AST-walked every .py file in backend/ to enumerate top-level imports (excluding stdlib + internal `app.*`). 2. Mapped each import to its installed pip distribution via importlib.metadata.packages_distributions() inside the running backend container. 3. Cross-checked: 16 distributions are imported but missing from .in. 4. Added one CLI-only dep that imports never reveal: ScoutSuite (the `scout` CLI is invoked via subprocess from app/integrations/scoutsuite/services/scoutsuite.py). 5. Added two config-required deps that the code references via SQLAlchemy URL strings rather than `import`: - aiomysql (the `mysql+aiomysql://` async driver) - PyMySQL (used by alembic + DB bootstrap) 6. Promoted `httpx` -> `httpx[http2]` and `uvicorn` -> `uvicorn[standard]` to preserve the extras already pinned in the current closure (h2/qh3 for httpx; uvloop/httptools/websockets/watchfiles for uvicorn). Without these, pip-compile would silently drop them. 7. Verified by running `pip install -r requirements.in -c <stripped requirements.txt>` in a clean python:3.11 container and comparing the resulting `pip freeze` against requirements.txt. After this PR, the only packages a future pip-compile would *drop* from requirements.txt are pre-existing dead deps: deepdiff, licensing, pySigma, redis, shufflepy, singul, numpy, Pillow, openpyxl, plus the orphan transitives left over from previously-removed Celery and STIX-1 stack (amqp, kombu, billiard, vine, click-didyoumean, click-plugins, click-repl, cybox, maec, mixbox, stix*, stixmarx, taxii2-client, …). Cleaning those up is intentionally left to a follow-up PR so the diff here stays focused on the .in reconcile. Verified locally: - The new .in installs cleanly in a fresh container. - The freeze output is a strict superset of every package the backend currently imports. - requirements.txt is untouched, so the running build pipeline is unaffected. Co-authored-by: taylor_socfortress <taylor.walton@socfortress.co> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>