@setoelkahfi / homebrew-tap
Code Commits Pulls CI/CD 205
main
main
md 10.6 KB
View source Raw

name: deployment

description: How to deploy, migrate, seed, and restart the sigit.si Rails app in production. Use this whenever shipping sigit-si, debugging a 500 after deploy, running a migration or seed on prod, or restarting Puma.

Deploying sigit.si

Product map: siGit Code (local agent) · siGit Code Cloud (hosted chat + Cloud Sessions) · siGit Code Cloud Agent (autonomous task → PR; planning). sigit.si is Git hosting; code.sigit.si is the home of siGit Code. Full taxonomy: product-overview.

The Rails app behind https://sigit.si (Git hosting for the AI era; also serves the model library and the /api/v1 surface the siGit Code Cloud products sign in to). Deploys happen by pushing main to a bare repo whose post-receive hook checks out the work tree and restarts Puma.

Server facts

Detail Value
Domain sigit.si (also the landing page getsigit.5mb.app)
Host api.splitfire.ai = 65.21.240.91 (Hetzner Debian, debian-4gb-hel1-2)
SSH ssh smb1-deploy (user deploy); the app runs as user git
App dir /home/git/apps/sigitsi (deployed in place, not Capistrano releases)
Bare repo /home/git/sigitsi.git (push target; post-receive hook deploys)
Puma port 3015 (SIGITSI_PORT), single mode
Ruby 3.4.2 via rbenv at /home/git/.rbenv
User repos /home/git/repos/users/<username>/<repo>.git (SIGITSI_REPOS_PATH)
nginx vhost /etc/nginx/sites-enabled/getsigit.5mb.app (proxies to 3015)

Note: this is a different box from api.smbcloud.xyz (deploy-sigitweb), which only hosts the static landing site. Do not look for the Rails app there.

Acting as the app user

The app, its files, and its Postgres role all belong to git. The deploy user has passwordless sudo -u git. A git login shell already has RAILS_ENV=production, the DB password (SIGITSI_DATABASE_PASSWORD), rbenv, and bundle on PATH, so run app commands like this:

ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && <command>"'

There is no .env file; env vars come from git's login shell (~/.profile).

Deploying

From a local clone with the main branch ready:

git push <prod-remote> main

The post-receive hook (/home/git/sigitsi.git/hooks/post-receive) then:

  1. git checkout -f main into /home/git/apps/sigitsi
  2. rbenv local, nvm use, bundle install
  3. rake assets:precompile
  4. rake db:prepare
  5. kill -9 $(lsof -t -i:3015) then bundle exec puma -e production > output.log 2>&1 &
  6. bin/jobs start > output-jobs.log 2>&1 &

The hook does not reliably migrate (read this before trusting a deploy)

The hook has no set -e, so a failed DB step does not stop the deploy: Puma restarts anyway, on whatever schema is live. Worse, rake db:prepare and rake db:migrate operate on all four configured databases (primary + solid cache/queue/cable). The cache/queue/cable databases do not exist in production and the sigitsi role lacks CREATEDB, so the task aborts with permission denied to create database before it ever migrates the primary DB. Its output goes to the pushing client, not to output.log (which Puma immediately truncates), so the failure is invisible afterward.

Result: a deploy that adds a migration leaves the schema stale and every route touching the new column returns 500, while the home page and auth pages (which do not touch it) stay up. See "Run a migration" below for the fix, and "Known issues".

Run a migration

The deploy hook's rake db:prepare now self-migrates (all four databases exist; see "Postgres admin"). To migrate by hand:

# check what is pending first
ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails db:migrate:status | tail"'

# apply
ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails db:migrate"'

If the cache/queue/cable databases are ever missing again, db:migrate aborts trying to create them (the sigitsi role lacks CREATEDB). Scope to the primary to get past it: bin/rails db:migrate:primary.

Then restart Puma (next section) so ActiveRecord regenerates attribute methods for the new columns. A live process with a stale schema raises NameError: undefined method 'kind' even after the column exists.

Seed production data

The model library ships with demo content (the bartowski/Qwen2.5-3B-Instruct-GGUF model and friends). Production needs this seeded once, e.g. so https://sigit.si/bartowski/Qwen2.5-3B-Instruct-GGUF resolves.

With all four databases present, bin/rails db:seed works directly. If the cache/queue/cable DBs are missing, db:seed's db:abort_if_pending_migrations prerequisite aborts on them; bypass it by loading the seed straight against the primary connection:

# normal
ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails db:seed"'
# fallback if the solid DBs are missing
ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && bin/rails runner \"Rails.application.load_seed\""'

What it creates (db/seeds.rb, idempotent — safe to re-run):

  • Demo owner users (bartowski, meta-llama, sentence-transformers) and one model under the existing sigit user (SiGit-Coder-1.5B-GGUF).
  • Each model's bare git repo under /home/git/repos/users/<owner>/<name>.git, holding a YAML model card and Git LFS pointer files for the weights.

The box has git-lfs installed, so the seed deliberately neutralises the local LFS filters (filter.lfs.clean=cat, filter.lfs.process=) before committing the pointer text. Without that, git-lfs tries to upload real objects and the push fails. This is already handled in db/seeds.rb.

Verify:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Chrome/130)" https://sigit.si/bartowski/Qwen2.5-3B-Instruct-GGUF

Restart Puma

Single-mode Puma on 3015. Two options:

# Preferred: hot restart (re-exec, ~zero downtime). Get the master pid:
ssh smb1-deploy 'ss -ltnp | grep 3015'   # or: sudo -u git lsof -t -i:3015
ssh smb1-deploy 'sudo -u git kill -USR2 <master_pid>'

# Full restart (matches the deploy hook), if a hot restart misbehaves:
ssh smb1-deploy 'sudo -u git bash -lc "cd /home/git/apps/sigitsi && kill -9 \$(lsof -t -i:3015); sleep 1; setsid bundle exec puma -e production > output.log 2>&1 < /dev/null &"'

Confirm it rebound and serves:

ssh smb1-deploy 'ss -ltnp | grep 3015'
ssh smb1-deploy 'curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Chrome/130)" http://127.0.0.1:3015/models'

There is no systemd unit; Puma is a detached background process owned by git.

nginx

The vhost is /etc/nginx/sites-enabled/sigit.si (resolve symlinks before editing). It terminates TLS, serves static files from /home/git/apps/sigitsi/public, and proxies everything else to the puma15 upstream (127.0.0.1:3015).

The static-extension location must fall through to the app

sigit.si serves repo file contents at arbitrary paths (/:user/:repo/blob/:branch/*path and /raw/...), so URLs like /sigit/nord/blob/main/public/web-app-manifest-192x192.png are app routes, not files on disk. The vhost has a catch-all static block:

location ~ ^(?!/rails/).+\.(jpg|jpeg|gif|png|ico|json|txt|xml)$ {
    ...
    try_files $uri @puma15;   # NOT  =404
}

It must end in try_files $uri @puma15;. With the stock try_files $uri =404;, nginx looks for the file under public/, does not find it, and returns 404 without ever reaching Rails. The result: any blob/raw URL whose path ends in .png/.jpg/.json/.ico/.txt/.xml (etc.) 404s, while .md and other extensions work. The shared server-nginx-rails template ships the =404 form, which is fine for normal apps but wrong here.

Editing the vhost safely

  • nginx includes every file in sites-enabled/. Never leave a backup (sigit.si.bak.*) there or nginx -t fails with duplicate upstream "puma15". Keep backups in /root or /tmp.
  • Always sudo nginx -t before sudo systemctl reload nginx.

Debugging a 500 after deploy

  1. Read the app log (Puma stdout): sudo -u git tail -80 /home/git/apps/sigitsi/output.log. Look for PG::UndefinedColumn (pending migration) or NameError on a new attribute (stale schema, needs a Puma restart).
  2. bin/rails db:migrate:status to see if a migration is down.
  3. Note that allow_browser versions: :modern returns 403 to clients with no/old User-Agent, so always pass a modern UA when curling, or a healthy route looks broken.

Postgres admin

PostgreSQL 13, local. The app role sigitsi connects over the local socket with a password and has no CREATEDB or superuser. For any admin task (creating databases, granting roles) go through the postgres superuser, which authenticates by peer:

ssh smb1-deploy 'sudo -u postgres psql -c "<SQL>"'

The four databases (sigitsi_production plus the solid _cache / _queue / _cable) now exist, each owned by sigitsi, created with:

CREATE DATABASE sigitsi_production_cache OWNER sigitsi;
CREATE DATABASE sigitsi_production_queue OWNER sigitsi;
CREATE DATABASE sigitsi_production_cable OWNER sigitsi;

Owning the database lets sigitsi create its own tables there, so no extra GRANT is needed. Creating them (rather than ALTER ROLE sigitsi CREATEDB) keeps the app role least-privileged. _cache/_cable stay empty by design; _queue is live since the siGit Code GitHub App reviews shipped (see below).

Known issues

  • _cache/_cable are unused; _queue is not. Production sets Action Cable to the redis adapter (cable.yml) and never sets config.cache_store = :solid_cache_store, so _cache/_cable exist only to satisfy the multi-database scaffold in config/database.yml and stay empty. Since the siGit Code GitHub App reviews shipped, production does run Solid Queue: production.rb sets queue_adapter = :solid_queue writing to the queue database, bin/jobs exists, and the hook's bin/jobs start starts a real supervisor. sigitsi_production_queue must contain the solid_queue_* tables — db:prepare loads them from db/queue_schema.rb; verify with psql sigitsi_production_queue -c '\dt'. See docs/github-app-reviews.md.
  • bin/jobs supervisors stack across deploys. The hook starts a jobs supervisor on every push but never stops the previous one. Add pkill -f 'solid_queue' || true before the bin/jobs start line in the post-receive hook.
  • Deploy hook is not fail-safe. No set -e, and Puma's stdout truncates the same output.log that captured the deploy steps, so migration failures ship silently. Hardening it (fail on a bad db:prepare, log deploy output to a separate file) is worthwhile.