@setoelkahfi / homebrew-tap
Code Commits Pulls CI/CD 205
main
main
md 5.3 KB
View source Raw

Import from GitHub (mirror + migrate)

One-click import of a Git repository into siGit, in two modes:

  • Migrate — copy the repo once; siGit becomes the source of truth and you can push to it.
  • Mirror — keep GitHub (or any upstream) as the source of truth; siGit auto-syncs on a schedule (and on webhook, if configured) and stays read-only until you detach it.

There is also import by URL for any public http(s) git URL, with no OAuth.

How it fits the existing app

  • Repos are still bare repos on disk under SIGITSI_REPOS_PATH/<user>/<name>.git, created via GitRepositoryService — the importer reuses that path, it does not invent new storage.
  • The long-running clone/push runs off the request thread in RepositoryImportJob (Solid Queue). Mirror syncs run in MirrorSyncJob, swept by the recurring MirrorSyncSchedulerJob (config/recurring.yml).
  • GitHub API access uses a dedicated GitHub OAuth app (env GITHUB_IMPORT_CLIENT_ID / GITHUB_IMPORT_CLIENT_SECRET), separate from the smbCloud "Continue with GitHub" sign-in (which yields no GitHub API token). The token is stored encrypted (GithubConnection#access_token, Active Record Encryption) and never returned to the client.

Key files

Area File
SSRF URL guard app/services/import_url_validator.rb
GitHub OAuth app/services/github_oauth_service.rb, app/controllers/github_connections_controller.rb
GitHub REST app/services/github_api_client.rb
Clone + push app/services/repository_import_service.rb
Import orchestration app/jobs/repository_import_job.rb, app/controllers/imports_controller.rb
Mirror sync app/services/mirror_sync_service.rb, app/jobs/mirror_sync_job.rb, app/jobs/mirror_sync_scheduler_job.rb
Mirror read-only app/controllers/git_http_controller.rb (reject_mirror_push!)
Detach / manual sync app/controllers/mirrors_controller.rb
Webhook app/controllers/mirror_webhooks_controller.rb (POST /webhooks/github)

Guardrails

  • SSRF: import-by-URL allows only http(s); loopback / RFC1918 / link-local (incl. 169.254.169.254) / IPv6 ULA + mapped-v4 are rejected, for literal IPs and every resolved address. Mirrors re-validate the upstream on every sync (DNS-rebinding window).
  • Isolation: clones run in a temp dir that's always removed; every git call has a hard timeout and its process group is killed on overrun; GIT_TERMINAL_PROMPT=0 makes auth failures fail fast.
  • Size/abuse: per-repo size cap, per-user in-flight volume cap, and a concurrent-import cap (IMPORT_MAX_* env). Oversized repos are skipped with a message.
  • Secrets: OAuth + upstream tokens are encrypted at rest, scrubbed from captured git output, never logged, never sent to the client, dropped on disconnect/detach.
  • Idempotency: a failed import leaves no half-created repo (rollback) and is retryable; retries clear any partial repo; name collisions with a different existing repo fail explicitly; bulk imports auto-suffix colliding names.
  • Rate limits: GithubApiClient raises RateLimited with a reset time so callers back off; the repo list is cached briefly and never blocks the page.

Not imported in v1 (surfaced in the UI, not silently dropped): issues, pull requests, wikis, releases, Actions.

Configuration

See .env.example (the "Import from GitHub" and "Active Record Encryption" sections). Minimum for full functionality:

GITHUB_IMPORT_CLIENT_ID=...
GITHUB_IMPORT_CLIENT_SECRET=...
# production only:
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=...
ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=...
ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=...   # bin/rails db:encryption:init
# optional: GITHUB_WEBHOOK_SECRET, IMPORT_MAX_*, MIRROR_SYNC_*

Register the GitHub OAuth app with callback <SIGITSI_URL>/import/github/callback.

Verify locally

bin/rails db:prepare
bin/rails tailwindcss:build        # so views render in specs

# Full suite (all green):
bundle exec rspec

# Just this feature:
bundle exec rspec \
  spec/services/import_url_validator_spec.rb \
  spec/models/github_connection_spec.rb \
  spec/services/repository_import_service_spec.rb \
  spec/jobs/repository_import_job_spec.rb \
  spec/requests/github_connections_spec.rb \
  spec/requests/git_http_mirror_spec.rb

Manual smoke test (no GitHub app needed — uses import-by-URL against a local fixture, so nothing hits the network):

# 1. Build a tiny bare fixture repo with a branch + tag.
tmp=$(mktemp -d); git init -q -b main "$tmp/w"
git -C "$tmp/w" -c user.email=a@b.c -c user.name=t commit -q --allow-empty -m init
git -C "$tmp/w" tag v1
git clone -q --bare "$tmp/w" "$tmp/src.git"

# 2. In `bin/rails console`, queue and run an import synchronously:
#    u = User.first
#    imp = u.repository_imports.create!(source_url: "<tmp>/src.git", mode: "migrate",
#            target_name: "fixture", status: "queued")
#    RepositoryImportJob.perform_now(imp.id)
#    imp.reload.status            # => "done"
#    imp.repository.default_branch # => "main"; branches/tags preserved

For mirror mode, set mode: "mirror"; the resulting repo is read-only (repo.writable_by?(u) == false) and a git push to it is rejected with a message. Use the repo page's Detach to convert it to a normal writable repo and stop syncing.