| 1 | # Import from GitHub (mirror + migrate) |
| 2 | |
| 3 | One-click import of a Git repository into siGit, in two modes: |
| 4 | |
| 5 | - **Migrate** — copy the repo once; siGit becomes the source of truth and you can |
| 6 | push to it. |
| 7 | - **Mirror** — keep GitHub (or any upstream) as the source of truth; siGit |
| 8 | auto-syncs on a schedule (and on webhook, if configured) and stays **read-only** |
| 9 | until you *detach* it. |
| 10 | |
| 11 | There is also **import by URL** for any public `http(s)` git URL, with no OAuth. |
| 12 | |
| 13 | ## How it fits the existing app |
| 14 | |
| 15 | - Repos are still bare repos on disk under `SIGITSI_REPOS_PATH/<user>/<name>.git`, |
| 16 | created via `GitRepositoryService` — the importer reuses that path, it does not |
| 17 | invent new storage. |
| 18 | - The long-running clone/push runs off the request thread in |
| 19 | `RepositoryImportJob` (Solid Queue). Mirror syncs run in `MirrorSyncJob`, |
| 20 | swept by the recurring `MirrorSyncSchedulerJob` (`config/recurring.yml`). |
| 21 | - GitHub API access uses a **dedicated GitHub OAuth app** (env |
| 22 | `GITHUB_IMPORT_CLIENT_ID` / `GITHUB_IMPORT_CLIENT_SECRET`), separate from the |
| 23 | smbCloud "Continue with GitHub" *sign-in* (which yields no GitHub API token). |
| 24 | The token is stored encrypted (`GithubConnection#access_token`, Active Record |
| 25 | Encryption) and never returned to the client. |
| 26 | |
| 27 | ## Key files |
| 28 | |
| 29 | | Area | File | |
| 30 | |------|------| |
| 31 | | SSRF URL guard | `app/services/import_url_validator.rb` | |
| 32 | | GitHub OAuth | `app/services/github_oauth_service.rb`, `app/controllers/github_connections_controller.rb` | |
| 33 | | GitHub REST | `app/services/github_api_client.rb` | |
| 34 | | Clone + push | `app/services/repository_import_service.rb` | |
| 35 | | Import orchestration | `app/jobs/repository_import_job.rb`, `app/controllers/imports_controller.rb` | |
| 36 | | Mirror sync | `app/services/mirror_sync_service.rb`, `app/jobs/mirror_sync_job.rb`, `app/jobs/mirror_sync_scheduler_job.rb` | |
| 37 | | Mirror read-only | `app/controllers/git_http_controller.rb` (`reject_mirror_push!`) | |
| 38 | | Detach / manual sync | `app/controllers/mirrors_controller.rb` | |
| 39 | | Webhook | `app/controllers/mirror_webhooks_controller.rb` (POST `/webhooks/github`) | |
| 40 | |
| 41 | ## Guardrails |
| 42 | |
| 43 | - **SSRF:** import-by-URL allows only `http(s)`; loopback / RFC1918 / link-local |
| 44 | (incl. `169.254.169.254`) / IPv6 ULA + mapped-v4 are rejected, for literal IPs |
| 45 | and every resolved address. Mirrors re-validate the upstream on every sync |
| 46 | (DNS-rebinding window). |
| 47 | - **Isolation:** clones run in a temp dir that's always removed; every git call |
| 48 | has a hard timeout and its process group is killed on overrun; |
| 49 | `GIT_TERMINAL_PROMPT=0` makes auth failures fail fast. |
| 50 | - **Size/abuse:** per-repo size cap, per-user in-flight volume cap, and a |
| 51 | concurrent-import cap (`IMPORT_MAX_*` env). Oversized repos are skipped with a |
| 52 | message. |
| 53 | - **Secrets:** OAuth + upstream tokens are encrypted at rest, scrubbed from |
| 54 | captured git output, never logged, never sent to the client, dropped on |
| 55 | disconnect/detach. |
| 56 | - **Idempotency:** a failed import leaves no half-created repo (rollback) and is |
| 57 | retryable; retries clear any partial repo; name collisions with a *different* |
| 58 | existing repo fail explicitly; bulk imports auto-suffix colliding names. |
| 59 | - **Rate limits:** `GithubApiClient` raises `RateLimited` with a reset time so |
| 60 | callers back off; the repo list is cached briefly and never blocks the page. |
| 61 | |
| 62 | Not imported in v1 (surfaced in the UI, not silently dropped): issues, pull |
| 63 | requests, wikis, releases, Actions. |
| 64 | |
| 65 | ## Configuration |
| 66 | |
| 67 | See `.env.example` (the "Import from GitHub" and "Active Record Encryption" |
| 68 | sections). Minimum for full functionality: |
| 69 | |
| 70 | ``` |
| 71 | GITHUB_IMPORT_CLIENT_ID=... |
| 72 | GITHUB_IMPORT_CLIENT_SECRET=... |
| 73 | # production only: |
| 74 | ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=... |
| 75 | ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=... |
| 76 | ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=... # bin/rails db:encryption:init |
| 77 | # optional: GITHUB_WEBHOOK_SECRET, IMPORT_MAX_*, MIRROR_SYNC_* |
| 78 | ``` |
| 79 | |
| 80 | Register the GitHub OAuth app with callback `<SIGITSI_URL>/import/github/callback`. |
| 81 | |
| 82 | ## Verify locally |
| 83 | |
| 84 | ```sh |
| 85 | bin/rails db:prepare |
| 86 | bin/rails tailwindcss:build # so views render in specs |
| 87 | |
| 88 | # Full suite (all green): |
| 89 | bundle exec rspec |
| 90 | |
| 91 | # Just this feature: |
| 92 | bundle exec rspec \ |
| 93 | spec/services/import_url_validator_spec.rb \ |
| 94 | spec/models/github_connection_spec.rb \ |
| 95 | spec/services/repository_import_service_spec.rb \ |
| 96 | spec/jobs/repository_import_job_spec.rb \ |
| 97 | spec/requests/github_connections_spec.rb \ |
| 98 | spec/requests/git_http_mirror_spec.rb |
| 99 | ``` |
| 100 | |
| 101 | Manual smoke test (no GitHub app needed — uses import-by-URL against a local |
| 102 | fixture, so nothing hits the network): |
| 103 | |
| 104 | ```sh |
| 105 | # 1. Build a tiny bare fixture repo with a branch + tag. |
| 106 | tmp=$(mktemp -d); git init -q -b main "$tmp/w" |
| 107 | git -C "$tmp/w" -c user.email=a@b.c -c user.name=t commit -q --allow-empty -m init |
| 108 | git -C "$tmp/w" tag v1 |
| 109 | git clone -q --bare "$tmp/w" "$tmp/src.git" |
| 110 | |
| 111 | # 2. In `bin/rails console`, queue and run an import synchronously: |
| 112 | # u = User.first |
| 113 | # imp = u.repository_imports.create!(source_url: "<tmp>/src.git", mode: "migrate", |
| 114 | # target_name: "fixture", status: "queued") |
| 115 | # RepositoryImportJob.perform_now(imp.id) |
| 116 | # imp.reload.status # => "done" |
| 117 | # imp.repository.default_branch # => "main"; branches/tags preserved |
| 118 | ``` |
| 119 | |
| 120 | For mirror mode, set `mode: "mirror"`; the resulting repo is read-only |
| 121 | (`repo.writable_by?(u) == false`) and a `git push` to it is rejected with a |
| 122 | message. Use the repo page's **Detach** to convert it to a normal writable repo |
| 123 | and stop syncing. |