| 1 | --- |
| 2 | name: deployment |
| 3 | description: How to deploy, migrate, seed, and restart the sigit.si Rails app in production. Use this whenever shipping sigit-si, debugging a 500 after deploy, running a migration or seed on prod, or restarting Puma. |
| 4 | --- |
| 5 | |
| 6 | # Deploying sigit.si |
| 7 | |
| 8 | > **Product map:** siGit Code (local agent) · siGit Code Cloud (hosted chat + Cloud Sessions) · siGit Code Cloud Agent (autonomous task → PR; planning). `sigit.si` is Git hosting; `code.sigit.si` is the home of siGit Code. Full taxonomy: [product-overview](../../../docs/product/product-overview.md). |
| 9 | |
| 10 | The Rails app behind `https://sigit.si` (Git hosting for the AI era; also serves the model library and the `/api/v1` surface the siGit Code Cloud products sign in to). Deploys happen by pushing `main` to a bare repo whose `post-receive` hook checks out the work tree and restarts Puma. |
| 11 | |
| 12 | ## Server facts |
| 13 | |
| 14 | | Detail | Value | |
| 15 | | ------ | ----- | |
| 16 | | Domain | `sigit.si` (also the landing page `getsigit.5mb.app`) | |
| 17 | | Host | `api.splitfire.ai` = `65.21.240.91` (Hetzner Debian, `debian-4gb-hel1-2`) | |
| 18 | | SSH | `ssh smb1-deploy` (user `deploy`); the app runs as user `git` | |
| 19 | | App dir | `/home/git/apps/sigitsi` (deployed in place, not Capistrano releases) | |
| 20 | | Bare repo | `/home/git/sigitsi.git` (push target; `post-receive` hook deploys) | |
| 21 | | Puma port | 3015 (`SIGITSI_PORT`), single mode | |
| 22 | | Ruby | 3.4.2 via rbenv at `/home/git/.rbenv` | |
| 23 | | User repos | `/home/git/repos/users/<username>/<repo>.git` (`SIGITSI_REPOS_PATH`) | |
| 24 | | nginx vhost | `/etc/nginx/sites-enabled/getsigit.5mb.app` (proxies to 3015) | |
| 25 | |
| 26 | Note: this is a **different box** from `api.smbcloud.xyz` (`deploy-sigitweb`), which only hosts the static landing site. Do not look for the Rails app there. |
| 27 | |
| 28 | ### Acting as the app user |
| 29 | |
| 30 | The app, its files, and its Postgres role all belong to `git`. The `deploy` user has passwordless `sudo -u git`. A `git` login shell already has `RAILS_ENV=production`, the DB password (`SIGITSI_DATABASE_PASSWORD`), rbenv, and bundle on PATH, so run app commands like this: |
| 31 | |
| 32 | ```bash |
| 33 | ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && <command>"' |
| 34 | ``` |
| 35 | |
| 36 | There is no `.env` file; env vars come from `git`'s login shell (`~/.profile`). |
| 37 | |
| 38 | ## Deploying |
| 39 | |
| 40 | From a local clone with the `main` branch ready: |
| 41 | |
| 42 | ```bash |
| 43 | git push <prod-remote> main |
| 44 | ``` |
| 45 | |
| 46 | The `post-receive` hook (`/home/git/sigitsi.git/hooks/post-receive`) then: |
| 47 | |
| 48 | 1. `git checkout -f main` into `/home/git/apps/sigitsi` |
| 49 | 2. `rbenv local`, `nvm use`, `bundle install` |
| 50 | 3. `rake assets:precompile` |
| 51 | 4. `rake db:prepare` |
| 52 | 5. `kill -9 $(lsof -t -i:3015)` then `bundle exec puma -e production > output.log 2>&1 &` |
| 53 | 6. `bin/jobs start > output-jobs.log 2>&1 &` |
| 54 | |
| 55 | ### The hook does not reliably migrate (read this before trusting a deploy) |
| 56 | |
| 57 | The hook has **no `set -e`**, so a failed DB step does not stop the deploy: Puma restarts anyway, on whatever schema is live. Worse, `rake db:prepare` and `rake db:migrate` operate on **all four configured databases** (primary + solid `cache`/`queue`/`cable`). The cache/queue/cable databases do **not** exist in production and the `sigitsi` role lacks `CREATEDB`, so the task aborts with `permission denied to create database` before it ever migrates the primary DB. Its output goes to the pushing client, not to `output.log` (which Puma immediately truncates), so the failure is invisible afterward. |
| 58 | |
| 59 | Result: a deploy that adds a migration leaves the schema stale and every route touching the new column returns 500, while the home page and auth pages (which do not touch it) stay up. See "Run a migration" below for the fix, and "Known issues". |
| 60 | |
| 61 | ## Run a migration |
| 62 | |
| 63 | The deploy hook's `rake db:prepare` now self-migrates (all four databases exist; see "Postgres admin"). To migrate by hand: |
| 64 | |
| 65 | ```bash |
| 66 | # check what is pending first |
| 67 | ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails db:migrate:status | tail"' |
| 68 | |
| 69 | # apply |
| 70 | ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails db:migrate"' |
| 71 | ``` |
| 72 | |
| 73 | If the cache/queue/cable databases are ever missing again, `db:migrate` aborts trying to create them (the `sigitsi` role lacks `CREATEDB`). Scope to the primary to get past it: `bin/rails db:migrate:primary`. |
| 74 | |
| 75 | Then restart Puma (next section) so ActiveRecord regenerates attribute methods for the new columns. A live process with a stale schema raises `NameError: undefined method 'kind'` even after the column exists. |
| 76 | |
| 77 | ## Seed production data |
| 78 | |
| 79 | The model library ships with demo content (the `bartowski/Qwen2.5-3B-Instruct-GGUF` model and friends). Production needs this seeded once, e.g. so `https://sigit.si/bartowski/Qwen2.5-3B-Instruct-GGUF` resolves. |
| 80 | |
| 81 | With all four databases present, `bin/rails db:seed` works directly. If the cache/queue/cable DBs are missing, `db:seed`'s `db:abort_if_pending_migrations` prerequisite aborts on them; bypass it by loading the seed straight against the primary connection: |
| 82 | |
| 83 | ```bash |
| 84 | # normal |
| 85 | ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails db:seed"' |
| 86 | # fallback if the solid DBs are missing |
| 87 | ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails runner \"Rails.application.load_seed\""' |
| 88 | ``` |
| 89 | |
| 90 | What it creates (`db/seeds.rb`, idempotent — safe to re-run): |
| 91 | |
| 92 | - Demo owner users (`bartowski`, `meta-llama`, `sentence-transformers`) and one model under the existing `sigit` user (`SiGit-Coder-1.5B-GGUF`). |
| 93 | - Each model's bare git repo under `/home/git/repos/users/<owner>/<name>.git`, holding a YAML model card and Git LFS pointer files for the weights. |
| 94 | |
| 95 | The box has `git-lfs` installed, so the seed deliberately neutralises the local LFS filters (`filter.lfs.clean=cat`, `filter.lfs.process=`) before committing the pointer text. Without that, git-lfs tries to upload real objects and the push fails. This is already handled in `db/seeds.rb`. |
| 96 | |
| 97 | Verify: |
| 98 | |
| 99 | ```bash |
| 100 | curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Chrome/130)" https://sigit.si/bartowski/Qwen2.5-3B-Instruct-GGUF |
| 101 | ``` |
| 102 | |
| 103 | ## Restart Puma |
| 104 | |
| 105 | Single-mode Puma on 3015. Two options: |
| 106 | |
| 107 | ```bash |
| 108 | # Preferred: hot restart (re-exec, ~zero downtime). Get the master pid: |
| 109 | ssh smb1-deploy 'ss -ltnp | grep 3015' # or: sudo -u git lsof -t -i:3015 |
| 110 | ssh smb1-deploy 'sudo -u git kill -USR2 <master_pid>' |
| 111 | |
| 112 | # Full restart (matches the deploy hook), if a hot restart misbehaves: |
| 113 | ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && kill -9 \$(lsof -t -i:3015); sleep 1; setsid bundle exec puma -e production > output.log 2>&1 < /dev/null &"' |
| 114 | ``` |
| 115 | |
| 116 | Confirm it rebound and serves: |
| 117 | |
| 118 | ```bash |
| 119 | ssh smb1-deploy 'ss -ltnp | grep 3015' |
| 120 | ssh smb1-deploy 'curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Chrome/130)" http://127.0.0.1:3015/models' |
| 121 | ``` |
| 122 | |
| 123 | There is no systemd unit; Puma is a detached background process owned by `git`. |
| 124 | |
| 125 | ## nginx |
| 126 | |
| 127 | The vhost is `/etc/nginx/sites-enabled/sigit.si` (resolve symlinks before editing). It terminates TLS, serves static files from `/home/git/apps/sigitsi/public`, and proxies everything else to the `puma15` upstream (`127.0.0.1:3015`). |
| 128 | |
| 129 | ### The static-extension location must fall through to the app |
| 130 | |
| 131 | sigit.si serves **repo file contents** at arbitrary paths (`/:user/:repo/blob/:branch/*path` and `/raw/...`), so URLs like `/sigit/nord/blob/main/public/web-app-manifest-192x192.png` are app routes, not files on disk. The vhost has a catch-all static block: |
| 132 | |
| 133 | ```nginx |
| 134 | location ~ ^(?!/rails/).+\.(jpg|jpeg|gif|png|ico|json|txt|xml)$ { |
| 135 | ... |
| 136 | try_files $uri @puma15; # NOT =404 |
| 137 | } |
| 138 | ``` |
| 139 | |
| 140 | It **must** end in `try_files $uri @puma15;`. With the stock `try_files $uri =404;`, nginx looks for the file under `public/`, does not find it, and returns 404 without ever reaching Rails. The result: any blob/raw URL whose path ends in `.png/.jpg/.json/.ico/.txt/.xml` (etc.) 404s, while `.md` and other extensions work. The shared `server-nginx-rails` template ships the `=404` form, which is fine for normal apps but wrong here. |
| 141 | |
| 142 | ### Editing the vhost safely |
| 143 | |
| 144 | - `nginx` includes every file in `sites-enabled/`. Never leave a backup (`sigit.si.bak.*`) there or `nginx -t` fails with `duplicate upstream "puma15"`. Keep backups in `/root` or `/tmp`. |
| 145 | - Always `sudo nginx -t` before `sudo systemctl reload nginx`. |
| 146 | |
| 147 | ## Debugging a 500 after deploy |
| 148 | |
| 149 | 1. Read the app log (Puma stdout): `sudo -u git tail -80 /home/git/apps/sigitsi/output.log`. Look for `PG::UndefinedColumn` (pending migration) or `NameError` on a new attribute (stale schema, needs a Puma restart). |
| 150 | 2. `bin/rails db:migrate:status` to see if a migration is `down`. |
| 151 | 3. Note that `allow_browser versions: :modern` returns **403** to clients with no/old User-Agent, so always pass a modern UA when curling, or a healthy route looks broken. |
| 152 | |
| 153 | ## Postgres admin |
| 154 | |
| 155 | PostgreSQL 13, local. The app role `sigitsi` connects over the local socket with a password and has **no** `CREATEDB` or superuser. For any admin task (creating databases, granting roles) go through the `postgres` superuser, which authenticates by `peer`: |
| 156 | |
| 157 | ```bash |
| 158 | ssh smb1-deploy 'sudo -u postgres psql -c "<SQL>"' |
| 159 | ``` |
| 160 | |
| 161 | The four databases (`sigitsi_production` plus the solid `_cache` / `_queue` / `_cable`) now exist, each owned by `sigitsi`, created with: |
| 162 | |
| 163 | ```sql |
| 164 | CREATE DATABASE sigitsi_production_cache OWNER sigitsi; |
| 165 | CREATE DATABASE sigitsi_production_queue OWNER sigitsi; |
| 166 | CREATE DATABASE sigitsi_production_cable OWNER sigitsi; |
| 167 | ``` |
| 168 | |
| 169 | Owning the database lets `sigitsi` create its own tables there, so no extra `GRANT` is needed. Creating them (rather than `ALTER ROLE sigitsi CREATEDB`) keeps the app role least-privileged. `_cache`/`_cable` stay empty by design; `_queue` is live since the siGit Code GitHub App reviews shipped (see below). |
| 170 | |
| 171 | ## Known issues |
| 172 | |
| 173 | - **`_cache`/`_cable` are unused; `_queue` is not.** Production sets Action Cable to the `redis` adapter (`cable.yml`) and never sets `config.cache_store = :solid_cache_store`, so `_cache`/`_cable` exist only to satisfy the multi-database scaffold in `config/database.yml` and stay empty. Since the siGit Code GitHub App reviews shipped, production **does** run Solid Queue: `production.rb` sets `queue_adapter = :solid_queue` writing to the `queue` database, `bin/jobs` exists, and the hook's `bin/jobs start` starts a real supervisor. `sigitsi_production_queue` must contain the `solid_queue_*` tables — `db:prepare` loads them from `db/queue_schema.rb`; verify with `psql sigitsi_production_queue -c '\dt'`. See `docs/github-app-reviews.md`. |
| 174 | - **`bin/jobs` supervisors stack across deploys.** The hook starts a jobs supervisor on every push but never stops the previous one. Add `pkill -f 'solid_queue' || true` before the `bin/jobs start` line in the post-receive hook. |
| 175 | - **Deploy hook is not fail-safe.** No `set -e`, and Puma's stdout truncates the same `output.log` that captured the deploy steps, so migration failures ship silently. Hardening it (fail on a bad `db:prepare`, log deploy output to a separate file) is worthwhile. |