@setoelkahfi / homebrew-tap / commits / 1dd5a72

Add one-click "Import from GitHub" (mirror + migrate)

Let users bring repositories to siGit in two modes: - Migrate: clone the source once; siGit becomes the source of truth. - Mirror: keep the upstream as source of truth, auto-sync on a schedule (and on webhook), read-only on our side until detached. Also supports import-by-URL for any public http(s) git repo (no OAuth). Reuses the existing on-disk bare-repo storage (GitRepositoryService) and runs the long clone/push off the request thread in Solid Queue jobs. Highlights: - Dedicated GitHub OAuth app (separate from smbCloud sign-in) minting a repo-scoped token, stored encrypted at rest (Active Record Encryption), never returned to the client. - RepositoryImportService: sandboxed `git clone --mirror` + push of all refs/branches/tags with full history (no rewrites), hard timeouts with process-group kill, size caps, and best-effort Git LFS detect/fetch that reports clearly when skipped. - Async, resumable imports with a live progress UI (queued -> cloning -> pushing -> done/failed); failures roll back so no half-created repo is left, and are retryable. Name collisions handled explicitly. - Mirror mode: read-only enforcement over Git smart HTTP (push rejected with a helpful ERR message), scheduled + webhook sync, last-synced status, and a detach action that converts to a writable repo. - Guardrails: SSRF URL validator (blocks loopback/RFC1918/link-local/ metadata), per-repo and per-user size/volume caps, GitHub rate-limit backoff, secrets scrubbed from git output and never logged. Tests: SSRF validator, token encryption + OAuth callback, mirror clone-and-push against a local fixture, mirror read-only enforcement, and failure cleanup / idempotency. Docs in docs/import-from-github.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019URPN7b9Kwa3RUP2ubFNo3

Claude committed Jul 2, 2026 at 06:22 UTC 1dd5a72b2b27c0a32d2e09c7235266ab2f1e51f9

42 files changed +2702 -7

.env.example

+39

@@ -87,6 +87,45 @@ STRIPE_WEBHOOK_SECRET=whsec_your-webhook-signing-secret

87 # SIGITSI_URL=https://sigit.si

90 +# -----------------------------------------------------------------------------

91 +# Import from GitHub (mirror + migrate)

92 +# A GitHub OAuth App used ONLY for importing repos — separate from the smbCloud

93 +# "Continue with GitHub" sign-in above. Register one at

94 +# https://github.com/settings/developers with the callback:

95 +# <SIGITSI_URL>/import/github/callback

96 +# Without these, import-by-URL (public repos) still works; GitHub repo listing /

97 +# private import does not.

98 +# -----------------------------------------------------------------------------

99 +

100 +# GITHUB_IMPORT_CLIENT_ID=your-github-oauth-app-client-id

101 +# GITHUB_IMPORT_CLIENT_SECRET=your-github-oauth-app-client-secret

102 +

103 +# Shared secret for the GitHub push webhook (POST /webhooks/github) that triggers

104 +# immediate mirror syncs. Set the same value in the repo/org webhook config.

105 +# GITHUB_WEBHOOK_SECRET=a-long-random-string

106 +

107 +# Import guardrails (all optional; sensible defaults shown).

108 +# IMPORT_MAX_REPO_SIZE_KB=2000000 # per-repo cap (~2 GB)

109 +# IMPORT_MAX_USER_VOLUME_KB=10000000 # per-user in-flight cap (~10 GB)

110 +# IMPORT_MAX_ACTIVE_PER_USER=10 # concurrent imports per user

111 +# IMPORT_GIT_TIMEOUT=1800 # seconds a single clone/push may run

112 +# MIRROR_SYNC_TIMEOUT=900 # seconds a single mirror fetch may run

113 +# MIRROR_SYNC_INTERVAL_SECONDS=3600 # how stale a mirror may get before re-sync

114 +

115 +

116 +# -----------------------------------------------------------------------------

117 +# Active Record Encryption (required in production)

118 +# Encrypts tokens at rest: the GitHub import OAuth token and any private mirror

119 +# upstream credential. Generate a set with: bin/rails db:encryption:init

120 +# In development/test these are derived from SECRET_KEY_BASE if unset (never used

121 +# for real user data).

122 +# -----------------------------------------------------------------------------

123 +

124 +# ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=

125 +# ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=

126 +# ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=

127 +

128 +

129 # -----------------------------------------------------------------------------

130 # Rails

131 # -----------------------------------------------------------------------------

app/controllers/git_http_controller.rb

+23 -2

@@ -24,9 +24,10 @@ class GitHttpController < ActionController::API

24 def info_refs

25 service = params[:service]

26 return head(:forbidden) unless SERVICES.include?(service)

27 + return if reject_mirror_push!(service)

28 return unless authorize!(service)

29 - advertise = git_run([service.delete_prefix("git-"), "--stateless-rpc", "--advertise-refs", @repo.disk_path])

30 + advertise = git_run([ service.delete_prefix("git-"), "--stateless-rpc", "--advertise-refs", @repo.disk_path ])

31 return head(:internal_server_error) if advertise.nil?

33 no_cache

@@ -42,17 +43,37 @@ class GitHttpController < ActionController::API

44 # POST /:user/:repo.git/git-receive-pack (push)

45 def receive_pack

46 + return if reject_mirror_push!("git-receive-pack")

47 return unless authorize!("git-receive-pack")

48 rpc("receive-pack", "application/x-git-receive-pack-result")

49 end

51 private

53 + # Mirrors are read-only: their upstream is the source of truth, so accepting a

54 + # push would let siGit silently diverge. Reject any receive-pack (push) with a

55 + # message the user actually sees — a git `ERR` pkt-line in the ref

56 + # advertisement, which the client prints as `fatal: remote error: …`. Returns

57 + # true when it handled (rejected) the request. Applies to everyone, including

58 + # the owner, until the mirror is detached.

59 + def reject_mirror_push!(service)

60 + return false unless service == "git-receive-pack"

61 + return false unless @repo&.mirror?

62 +

63 + message = "siGit: #{@repo.full_name} is a read-only mirror of #{@repo.upstream_url}. " \

64 + "Detach it in the siGit UI to make it writable."

65 +

66 + no_cache

67 + response.content_type = "application/x-git-receive-pack-advertisement"

68 + render body: pkt_line("# service=git-receive-pack\n") + "0000" + pkt_line("ERR #{message}")

69 + true

70 + end

71 +

72 def rpc(service, content_type)

73 input = request.body.read.to_s

74 input = ActiveSupport::Gzip.decompress(input) if gzip_request?

55 - out = git_run([service, "--stateless-rpc", @repo.disk_path], stdin: input)

76 + out = git_run([ service, "--stateless-rpc", @repo.disk_path ], stdin: input)

77 return head(:internal_server_error) if out.nil?

79 no_cache

app/controllers/github_connections_controller.rb new

+68

@@ -0,0 +1,68 @@

1 +# frozen_string_literal: true

2 +

3 +# Connects a user's GitHub account for repository imports via our own GitHub

4 +# OAuth app (separate from smbCloud sign-in). The resulting token is stored

5 +# encrypted in GithubConnection and never leaves the server.

6 +class GithubConnectionsController < ApplicationController

7 + before_action :require_sign_in!

8 + before_action :noindex!

9 +

10 + # GET /import/github/connect

11 + # Kicks off the OAuth dance. `visibility=public` narrows the scope to public

12 + # repos only; the default requests `repo` so private repos can be imported.

13 + def connect

14 + unless GithubOauthService.configured?

15 + return redirect_to new_import_path, alert: "GitHub import isn't configured on this server yet."

16 + end

17 +

18 + state = GithubOauthService.generate_state

19 + session[:github_import_state] = state

20 +

21 + include_private = params[:visibility] != "public"

22 + redirect_to GithubOauthService.authorize_url(

23 + redirect_uri: github_import_callback_url,

24 + state: state,

25 + include_private: include_private

26 + ), allow_other_host: true

27 + end

28 +

29 + # GET /import/github/callback

30 + def callback

31 + if params[:error].present?

32 + return redirect_to new_import_path, alert: "GitHub connection was cancelled."

33 + end

34 +

35 + # CSRF: the state we set must round-trip back unchanged.

36 + expected = session.delete(:github_import_state)

37 + if expected.blank? || !ActiveSupport::SecurityUtils.secure_compare(params[:state].to_s, expected)

38 + return redirect_to new_import_path, alert: "GitHub connection failed a security check. Please try again."

39 + end

40 +

41 + token = GithubOauthService.exchange_code(

42 + code: params[:code].to_s,

43 + redirect_uri: github_import_callback_url

44 + )

45 +

46 + login = GithubApiClient.new(token[:access_token]).login

47 +

48 + connection = current_user.github_connection || current_user.build_github_connection

49 + connection.update!(

50 + access_token: token[:access_token],

51 + scope: token[:scope],

52 + token_type: token[:token_type],

53 + github_login: login,

54 + connected_at: Time.current

55 + )

56 +

57 + redirect_to new_import_path, notice: "Connected to GitHub#{login ? " as @#{login}" : ''}."

58 + rescue GithubOauthService::ExchangeError, GithubOauthService::ConfigurationError => e

59 + Rails.logger.warn("GitHub import connect failed: #{e.class}")

60 + redirect_to new_import_path, alert: "Couldn't connect to GitHub. Please try again."

61 + end

62 +

63 + # DELETE /import/github/disconnect

64 + def disconnect

65 + current_user.github_connection&.destroy

66 + redirect_to new_import_path, notice: "Disconnected from GitHub."

67 + end

68 +end

app/controllers/github_webhooks_controller.rb new

+53

@@ -0,0 +1,53 @@

1 +# frozen_string_literal: true

2 +

3 +# Receives GitHub push webhooks and triggers an immediate mirror sync for any

4 +# mirror whose upstream matches the pushed repo. Optional: mirrors also sync on

5 +# a schedule, so this is just a freshness optimization.

6 +#

7 +# Authenticated by the HMAC signature GitHub sends (X-Hub-Signature-256), keyed

8 +# by GITHUB_WEBHOOK_SECRET. Without a configured secret the endpoint refuses

9 +# requests rather than trusting unsigned input.

10 +class GithubWebhooksController < ActionController::API

11 + # POST /webhooks/github

12 + def create

13 + return head(:service_unavailable) if webhook_secret.blank?

14 +

15 + payload = request.body.read

16 + return head(:unauthorized) unless valid_signature?(payload)

17 +

18 + event = request.headers["X-GitHub-Event"]

19 + # Respond OK to ping so the webhook can be verified in GitHub's UI.

20 + return head(:ok) if event == "ping"

21 + return head(:ok) unless event == "push"

22 +

23 + data = JSON.parse(payload) rescue {}

24 + urls = candidate_urls(data)

25 + return head(:ok) if urls.empty?

26 +

27 + Repository.mirrors.where(upstream_url: urls).find_each do |repo|

28 + MirrorSyncJob.perform_later(repo.id)

29 + end

30 +

31 + head :ok

32 + end

33 +

34 + private

35 +

36 + # The clone URLs GitHub might have stored as our upstream for this repo.

37 + def candidate_urls(data)

38 + repo = data["repository"] || {}

39 + [ repo["clone_url"], repo["git_url"], repo["html_url"], repo["ssh_url"] ].compact.uniq

40 + end

41 +

42 + def valid_signature?(payload)

43 + sig = request.headers["X-Hub-Signature-256"].to_s

44 + return false if sig.blank?

45 +

46 + expected = "sha256=" + OpenSSL::HMAC.hexdigest("sha256", webhook_secret, payload)

47 + ActiveSupport::SecurityUtils.secure_compare(sig, expected)

48 + end

49 +

50 + def webhook_secret

51 + ENV["GITHUB_WEBHOOK_SECRET"].presence

52 + end

53 +end

app/controllers/imports_controller.rb new

+204

@@ -0,0 +1,204 @@

1 +# frozen_string_literal: true

2 +

3 +# The "Import from GitHub" experience: connect GitHub and pick repos, or paste a

4 +# public git URL. Imports run off the request thread in RepositoryImportJob; this

5 +# controller only validates, enforces limits, records RepositoryImport rows, and

6 +# enqueues the work.

7 +class ImportsController < ApplicationController

8 + before_action :require_sign_in!

9 + before_action :noindex!

10 +

11 + # Guardrails against abuse / one user saturating the workers.

12 + MAX_ACTIVE_IMPORTS = Integer(ENV.fetch("IMPORT_MAX_ACTIVE_PER_USER", 10))

13 + MAX_USER_VOLUME_KB = Integer(ENV.fetch("IMPORT_MAX_USER_VOLUME_KB", 10_000_000)) # ~10 GB in flight

14 + MAX_REPO_SIZE_KB = RepositoryImportService::DEFAULT_MAX_SIZE_KB

15 +

16 + # GET /import

17 + def new

18 + @connection = current_user.github_connection

19 + @recent_imports = current_user.repository_imports.recent.limit(20)

20 + @github_repos = load_github_repos(@connection) if @connection

21 + end

22 +

23 + # GET /import (list-only, e.g. progress polling of all imports)

24 + def index

25 + @imports = current_user.repository_imports.recent.limit(50)

26 + end

27 +

28 + # GET /import/:id — progress for a single import

29 + def show

30 + @import = current_user.repository_imports.find(params[:id])

31 + rescue ActiveRecord::RecordNotFound

32 + redirect_to new_import_path, alert: "Import not found."

33 + end

34 +

35 + # POST /import

36 + # Either:

37 + # source=github, repos[]=owner/name..., mode=migrate|mirror

38 + # source=url, url=..., name=..., mode=..., visibility=public|private

39 + def create

40 + mode = params[:mode].to_s == "mirror" ? "mirror" : "migrate"

41 +

42 + specs =

43 + if params[:source].to_s == "url"

44 + build_url_spec(mode)

45 + else

46 + build_github_specs(mode)

47 + end

48 +

49 + return if performed? # a builder already redirected with an error

50 + if specs.blank?

51 + return redirect_to new_import_path, alert: "Select at least one repository to import."

52 + end

53 +

54 + enqueue_imports(specs)

55 + end

56 +

57 + # POST /import/:id/retry

58 + def retry

59 + import = current_user.repository_imports.find(params[:id])

60 + unless import.retryable?

61 + return redirect_to import_path(import), alert: "This import can't be retried."

62 + end

63 +

64 + import.update!(status: "queued", error_message: nil, started_at: nil, finished_at: nil)

65 + RepositoryImportJob.perform_later(import.id)

66 + redirect_to import_path(import), notice: "Retrying import."

67 + rescue ActiveRecord::RecordNotFound

68 + redirect_to new_import_path, alert: "Import not found."

69 + end

70 +

71 + private

72 +

73 + # Builds a single import spec from the by-URL form. Validates the URL (SSRF

74 + # guard) and the target name. Redirects with an error on invalid input.

75 + def build_url_spec(mode)

76 + url = ImportUrlValidator.validate!(params[:url])

77 + host = URI.parse(url).host

78 + name = params[:name].presence || derive_name_from_url(url)

79 +

80 + [ { source_url: url, source_host: host, mode: mode,

81 + target_name: name, target_private: false, size_kb: nil,

82 + description: nil, default_branch: nil } ]

83 + rescue ImportUrlValidator::InvalidUrl => e

84 + redirect_to new_import_path, alert: e.message

85 + nil

86 + end

87 +

88 + # Builds import specs from selected GitHub repos. Reads fresh metadata from

89 + # GitHub so we capture the real visibility/size/default branch (never trusting

90 + # client-supplied values for security-relevant fields like private).

91 + def build_github_specs(mode)

92 + selected = Array(params[:repos]).reject(&:blank?).uniq

93 + return [] if selected.empty?

94 +

95 + connection = current_user.github_connection

96 + return redirect_to(new_import_path, alert: "Connect GitHub first.") && nil if connection.nil?

97 +

98 + client = connection.api_client

99 + selected.filter_map do |full_name|

100 + meta = client.repository(full_name)

101 + next if meta.nil?

102 +

103 + {

104 + source_url: meta[:clone_url],

105 + source_host: "github.com",

106 + mode: mode,

107 + target_name: meta[:name],

108 + target_private: !!meta[:private],

109 + size_kb: meta[:size_kb],

110 + description: meta[:description],

111 + default_branch: meta[:default_branch]

112 + }

113 + end

114 + rescue GithubApiClient::RateLimited

115 + redirect_to new_import_path, alert: "GitHub rate limit reached. Please try again shortly."

116 + nil

117 + rescue GithubApiClient::Error

118 + redirect_to new_import_path, alert: "Couldn't read repositories from GitHub. Please try again."

119 + nil

120 + end

121 +

122 + # Applies caps, resolves name collisions, creates the RepositoryImport rows,

123 + # and enqueues one job per import. Redirects with a summary.

124 + def enqueue_imports(specs)

125 + active = current_user.repository_imports.active.count

126 + volume = current_user.active_import_size_kb

127 +

128 + created = []

129 + rejected = []

130 +

131 + specs.each do |spec|

132 + if spec[:size_kb] && spec[:size_kb] > MAX_REPO_SIZE_KB

133 + rejected << "#{spec[:target_name]} (too large: #{(spec[:size_kb] / 1024.0).round} MB)"

134 + next

135 + end

136 + if active + created.length >= MAX_ACTIVE_IMPORTS

137 + rejected << "#{spec[:target_name]} (import queue full — try again later)"

138 + next

139 + end

140 + if volume + (spec[:size_kb] || 0) > MAX_USER_VOLUME_KB

141 + rejected << "#{spec[:target_name]} (would exceed your in-flight import limit)"

142 + next

143 + end

144 +

145 + import = current_user.repository_imports.create!(

146 + source_url: spec[:source_url],

147 + source_host: spec[:source_host],

148 + mode: spec[:mode],

149 + target_name: unique_repo_name(spec[:target_name]),

150 + target_private: spec[:target_private],

151 + description: spec[:description],

152 + default_branch: spec[:default_branch],

153 + source_size_kb: spec[:size_kb],

154 + status: "queued"

155 + )

156 + RepositoryImportJob.perform_later(import.id)

157 + created << import

158 + volume += (spec[:size_kb] || 0)

159 + end

160 +

161 + notice = created.any? ? "Queued #{created.length} import#{'s' if created.length != 1}." : nil

162 + alert = rejected.any? ? "Skipped: #{rejected.join('; ')}." : nil

163 +

164 + if created.length == 1 && rejected.empty?

165 + redirect_to import_path(created.first), notice: notice

166 + else

167 + redirect_to new_import_path, notice: notice, alert: alert

168 + end

169 + end

170 +

171 + # Ensures the target name doesn't collide with an existing repo (or another

172 + # queued import) by appending -1, -2, … This makes bulk imports and retries

173 + # idempotent rather than failing on the unique (user, name) index.

174 + def unique_repo_name(base)

175 + base = base.to_s.gsub(/[^a-zA-Z0-9._-]/, "-").gsub(/-{2,}/, "-").gsub(/\A[-.]+|[-.]+\z/, "")

176 + base = "imported-repo" if base.blank?

177 +

178 + taken = current_user.repositories.pluck(:name).map(&:downcase).to_set

179 + taken.merge(current_user.repository_imports.active.pluck(:target_name).map(&:downcase))

180 +

181 + return base unless taken.include?(base.downcase)

182 +

183 + (1..1000).each do |n|

184 + candidate = "#{base}-#{n}"

185 + return candidate unless taken.include?(candidate.downcase)

186 + end

187 + "#{base}-#{SecureRandom.hex(3)}"

188 + end

189 +

190 + def derive_name_from_url(url)

191 + File.basename(URI.parse(url).path.to_s).sub(/\.git\z/, "").presence || "imported-repo"

192 + end

193 +

194 + # Lists the user's GitHub repos for the picker. Cached briefly so re-rendering

195 + # the page (and Turbo revisits) don't re-hit GitHub. Never blocks the page on a

196 + # rate-limit/outage — returns nil and the view shows a fallback.

197 + def load_github_repos(connection)

198 + Rails.cache.fetch("github_repos/#{connection.id}/#{connection.updated_at.to_i}", expires_in: 5.minutes) do

199 + connection.api_client.list_repositories

200 + end

201 + rescue GithubApiClient::RateLimited, GithubApiClient::Error, GithubApiClient::Unauthorized

202 + nil

203 + end

204 +end

app/controllers/mirrors_controller.rb new

+50

@@ -0,0 +1,50 @@

1 +# frozen_string_literal: true

2 +

3 +# Actions on mirror repositories: detach (convert to a normal writable repo and

4 +# stop syncing) and a manual sync trigger. Owner-only.

5 +class MirrorsController < ApplicationController

6 + before_action :require_sign_in!

7 + before_action :noindex!

8 + before_action :load_repository

9 + before_action :ensure_owner!

10 +

11 + # POST /:username/:repository/mirror/detach

12 + # Flips the mirror into a normal writable repo and stops syncing. Idempotent.

13 + def detach

14 + if @repository.detach_mirror!

15 + redirect_to repository_path(@owner.username, @repository.name),

16 + notice: "Mirror detached. This is now a normal repository you can push to."

17 + else

18 + redirect_to repository_path(@owner.username, @repository.name),

19 + alert: "This repository isn't a mirror."

20 + end

21 + end

22 +

23 + # POST /:username/:repository/mirror/sync

24 + # Enqueue an immediate sync (in addition to the scheduled ones).

25 + def sync

26 + unless @repository.mirror?

27 + return redirect_to repository_path(@owner.username, @repository.name),

28 + alert: "This repository isn't a mirror."

29 + end

30 +

31 + MirrorSyncJob.perform_later(@repository.id)

32 + redirect_to repository_path(@owner.username, @repository.name),

33 + notice: "Sync queued."

34 + end

35 +

36 + private

37 +

38 + def load_repository

39 + @owner = User.find_by!(username: params[:username])

40 + @repository = @owner.repositories.find_by!(name: params[:repository])

41 + rescue ActiveRecord::RecordNotFound

42 + render file: Rails.public_path.join("404.html"), status: :not_found, layout: false

43 + end

44 +

45 + def ensure_owner!

46 + unless signed_in? && current_user == @owner

47 + render file: Rails.public_path.join("404.html"), status: :not_found, layout: false

48 + end

49 + end

50 +end

app/jobs/mirror_sync_job.rb new

+26

@@ -0,0 +1,26 @@

1 +# frozen_string_literal: true

2 +

3 +# Syncs one mirror repository from its upstream. Enqueued on a schedule by

4 +# MirrorSyncSchedulerJob and (optionally) by a GitHub push webhook. Marks the

5 +# repo's sync status so the UI can show last-synced time and surface failures.

6 +class MirrorSyncJob < ApplicationJob

7 + queue_as :default

8 +

9 + def perform(repository_id)

10 + repo = Repository.find_by(id: repository_id)

11 + return if repo.nil? || !repo.mirror?

12 +

13 + repo.update!(mirror_status: "syncing")

14 + MirrorSyncService.sync(repo)

15 + repo.update!(mirror_status: "ok", mirror_synced_at: Time.current, mirror_error: nil)

16 + rescue ImportUrlValidator::InvalidUrl => e

17 + # Upstream now resolves somewhere unsafe (e.g. DNS rebinding) — stop, don't

18 + # fetch, and surface it.

19 + repo&.update!(mirror_status: "failed", mirror_error: "Upstream is no longer a valid public URL: #{e.message}")

20 + rescue MirrorSyncService::SyncError => e

21 + repo&.update!(mirror_status: "failed", mirror_error: e.message)

22 + rescue StandardError => e

23 + Rails.logger.error("Mirror sync failed for repo #{repository_id}: #{e.class}: #{e.message}")

24 + repo&.update!(mirror_status: "failed", mirror_error: "Sync failed unexpectedly.")

25 + end

26 +end

app/jobs/mirror_sync_scheduler_job.rb new

+28

@@ -0,0 +1,28 @@

1 +# frozen_string_literal: true

2 +

3 +# Periodic sweep that enqueues a MirrorSyncJob for every mirror due for a

4 +# refresh. Runs from Solid Queue's recurring schedule (config/recurring.yml).

5 +#

6 +# Fanning out one job per repo (rather than syncing inline) keeps any single

7 +# slow/hung upstream from blocking the others and lets the queue spread the work

8 +# across workers. "Due" means never-synced or last synced before the interval.

9 +class MirrorSyncSchedulerJob < ApplicationJob

10 + queue_as :default

11 +

12 + # How stale a mirror may get before we re-sync it on the schedule. Webhooks,

13 + # when configured, sync sooner.

14 + SYNC_INTERVAL = ActiveSupport::Duration.build(Integer(ENV.fetch("MIRROR_SYNC_INTERVAL_SECONDS", 3600)))

15 +

16 + # Don't pile up on a mirror that's already syncing.

17 + def perform

18 + cutoff = Time.current - SYNC_INTERVAL

19 +

20 + due = Repository.mirrors

21 + .where.not(mirror_status: "syncing")

22 + .where("mirror_synced_at IS NULL OR mirror_synced_at < ?", cutoff)

23 +

24 + due.find_each do |repo|

25 + MirrorSyncJob.perform_later(repo.id)

26 + end

27 + end

28 +end

app/jobs/repository_import_job.rb new

+143

@@ -0,0 +1,143 @@

1 +# frozen_string_literal: true

2 +

3 +# Runs a repository import off the request thread: creates the destination repo,

4 +# mirror-clones the source and pushes its history in, records LFS/metadata, and

5 +# either finishes cleanly or rolls back so no half-created repo is left behind.

6 +#

7 +# Idempotency & retries: the job is safe to re-run for a failed import. It first

8 +# clears any partial destination from a previous attempt, so a retry starts from

9 +# a clean slate and can't duplicate or corrupt an existing repo. A name that

10 +# collides with a *different* existing repo fails explicitly rather than

11 +# clobbering it.

12 +class RepositoryImportJob < ApplicationJob

13 + queue_as :default

14 +

15 + # Raised when the target name collides with a different existing repo. Its

16 + # message is user-facing, so it's surfaced verbatim on the import.

17 + class NameCollisionError < StandardError; end

18 +

19 + # Don't auto-retry: an import failure is surfaced to the user as a retryable

20 + # state with a message, and retrying blindly could hammer a bad remote.

21 + def perform(import_id)

22 + import = RepositoryImport.find_by(id: import_id)

23 + return if import.nil?

24 + # Guard against double-delivery: only queued or (re-queued) failed imports

25 + # should run. An already-active/done import is left alone.

26 + return unless %w[queued failed].include?(import.status)

27 +

28 + reset_for_retry(import)

29 +

30 + user = import.user

31 + repo = create_destination!(import)

32 + import.update!(repository: repo)

33 +

34 + result = RepositoryImportService.clone_and_push(

35 + source_url: import.source_url,

36 + dest_path: repo.disk_path,

37 + credential: import_credential(import),

38 + progress: ->(phase) { import.transition_to(phase.to_s) }

39 + )

40 +

41 + import.transition_to("finalizing")

42 + finalize_repo!(repo, import, result)

43 +

44 + import.update!(lfs_detected: result.lfs_detected, lfs_skipped: result.lfs_skipped,

45 + source_size_kb: result.size_kb || import.source_size_kb)

46 + import.transition_to("done")

47 + rescue StandardError => e

48 + handle_failure(import, e)

49 + end

50 +

51 + private

52 +

53 + # Clear any leftover from a previous failed attempt so a retry is clean.

54 + def reset_for_retry(import)

55 + return if import.repository.nil?

56 +

57 + destroy_repo(import.repository)

58 + import.update!(repository: nil)

59 + end

60 +

61 + # Creates the Repository row and the bare repo on disk. Name collisions with a

62 + # different existing repo are rejected here, explicitly.

63 + def create_destination!(import)

64 + user = import.user

65 + existing = user.repositories.find_by("LOWER(name) = ?", import.target_name.downcase)

66 + if existing

67 + raise NameCollisionError,

68 + "A repository named \"#{import.target_name}\" already exists. Rename it or delete it, then retry."

69 + end

70 +

71 + repo = user.repositories.new(

72 + name: import.target_name,

73 + description: import.description,

74 + is_private: import.target_private,

75 + default_branch: import.default_branch.presence || "main",

76 + kind: "code"

77 + )

78 + repo.disk_path = GitRepositoryService.repo_path(user.username, repo.name)

79 + repo.save!

80 +

81 + GitRepositoryService.create_bare_repo(user.username, repo.name)

82 + repo

83 + end

84 +

85 + # After the push: point HEAD at the imported default branch (so the README

86 + # renders on the landing page) and, for mirror mode, wire up the read-only

87 + # upstream link and stamp the first successful sync.

88 + def finalize_repo!(repo, import, _result)

89 + head = GitRepositoryService.set_head(repo.disk_path, repo.default_branch)

90 + repo.update!(default_branch: head) if head.present? && head != repo.default_branch

91 +

92 + if import.mirror?

93 + repo.update!(

94 + mirror: true,

95 + upstream_url: import.source_url,

96 + upstream_token: import_credential(import),

97 + mirror_status: "ok",

98 + mirror_synced_at: Time.current

99 + )

100 + end

101 + end

102 +

103 + # The credential to clone/fetch the source with. GitHub private sources use the

104 + # user's connection token; URL imports are unauthenticated (public only), so a

105 + # private URL without credentials fails fast in the clone.

106 + def import_credential(import)

107 + return nil unless import.source_host == "github.com"

108 +

109 + import.user.github_connection&.access_token

110 + end

111 +

112 + def handle_failure(import, error)

113 + return if import.nil?

114 +

115 + # Leave no half-created repo behind.

116 + if import.repository

117 + destroy_repo(import.repository)

118 + import.update!(repository: nil)

119 + end

120 +

121 + message =

122 + case error

123 + when NameCollisionError,

124 + RepositoryImportService::SizeExceeded,

125 + RepositoryImportService::CloneTimeout,

126 + RepositoryImportService::ImportError

127 + error.message

128 + else

129 + Rails.logger.error("Import ##{import.id} failed: #{error.class}: #{error.message}")

130 + "The import failed unexpectedly. You can retry it."

131 + end

132 +

133 + import.transition_to("failed", error: message)

134 + end

135 +

136 + def destroy_repo(repo)

137 + path = repo.disk_path

138 + repo.destroy

139 + FileUtils.rm_rf(path) if path.present?

140 + rescue StandardError => e

141 + Rails.logger.error("Failed to clean up repo #{repo&.id}: #{e.message}")

142 + end

143 +end

app/models/github_connection.rb new

+27

@@ -0,0 +1,27 @@

1 +# frozen_string_literal: true

2 +

3 +# A user's connection to their GitHub account for importing repositories.

4 +#

5 +# Distinct from smbCloud "Continue with GitHub" sign-in: that authenticates the

6 +# user to siGit; this holds a GitHub API token (scope `repo` or `public_repo`)

7 +# used to list and clone the user's repos. One connection per user; the token is

8 +# encrypted at rest and never serialized to the client.

9 +class GithubConnection < ApplicationRecord

10 + belongs_to :user

11 +

12 + encrypts :access_token

13 +

14 + validates :access_token, presence: true

15 + validates :user_id, uniqueness: true

16 +

17 + # True when the connection can reach the user's private repositories.

18 + def private_scope?

19 + scope.to_s.split(/[ ,]/).include?("repo")

20 + end

21 +

22 + # A GithubApiClient bound to this connection's token. The token stays server

23 + # side; callers get repo data, never the credential.

24 + def api_client

25 + GithubApiClient.new(access_token)

26 + end

27 +end

app/models/repository.rb

+44 -1

@@ -6,6 +6,13 @@ class Repository < ApplicationRecord

6 has_many :stargazers, through: :stars, source: :user

7 has_many :pull_requests, dependent: :destroy

8 has_many :issues, dependent: :destroy

9 + has_many :repository_imports, dependent: :nullify

10 +

11 + # Upstream credential for a private mirror, encrypted at rest and never

12 + # returned to the client. Nil for public upstreams and non-mirror repos.

13 + encrypts :upstream_token

14 +

15 + scope :mirrors, -> { where(mirror: true) }

17 scope :models, -> { where(kind: "model") }

18 scope :code, -> { where(kind: "code") }

@@ -44,11 +51,47 @@ class Repository < ApplicationRecord

51 end

53 # True if +user+ may push to / open changes against this repo. siGit repos

47 - # have a single owner and no collaborators yet, so write == ownership.

54 + # have a single owner and no collaborators yet, so write == ownership — but a

55 + # mirror is read-only for everyone (including the owner) until it's detached,

56 + # so its upstream stays the single source of truth.

57 def writable_by?(user)

58 + return false if mirror?

59 user.present? && user_id == user.id

60 end

62 + # ── Mirror mode ─────────────────────────────────────────────────────────

63 +

64 + def mirror?

65 + mirror

66 + end

67 +

68 + # Flip a mirror into a normal writable repo and stop syncing. Idempotent: a

69 + # non-mirror repo is left untouched. The upstream URL is kept for reference,

70 + # but the credential is dropped since we no longer fetch with it.

71 + def detach_mirror!

72 + return false unless mirror?

73 +

74 + update!(

75 + mirror: false,

76 + upstream_token: nil,

77 + mirror_status: nil,

78 + mirror_error: nil

79 + )

80 + true

81 + end

82 +

83 + # A short human status for the mirror badge/health line.

84 + def mirror_state_label

85 + return nil unless mirror?

86 +

87 + case mirror_status

88 + when "syncing" then "Syncing…"

89 + when "failed" then "Sync failed"

90 + else

91 + mirror_synced_at ? "Synced #{mirror_synced_at.strftime('%b %-d, %H:%M UTC')}" : "Awaiting first sync"

92 + end

93 + end

94 +

95 # Read a file's contents at +ref+ (branch, tag, or SHA). Returns the content

96 # String, or nil when the path doesn't exist at that ref. This is the git read

97 # layer the MCP `get_file_contents` tool sits on.

app/models/repository_import.rb new

+74

@@ -0,0 +1,74 @@

1 +# frozen_string_literal: true

2 +

3 +# One repository import, from queued through to done or failed. Drives the

4 +# progress UI and makes a failed import retryable. The actual clone/push runs in

5 +# RepositoryImportJob; this record is the state it advances.

6 +class RepositoryImport < ApplicationRecord

7 + belongs_to :user

8 + belongs_to :repository, optional: true

9 +

10 + MODES = %w[migrate mirror].freeze

11 +

12 + # Ordered lifecycle. `cloning`/`pushing`/`fetching_lfs` are the long phases the

13 + # UI narrates; terminal states are `done` and `failed`.

14 + STATUSES = %w[queued cloning pushing fetching_lfs finalizing done failed].freeze

15 + TERMINAL = %w[done failed].freeze

16 + ACTIVE = STATUSES - TERMINAL

17 +

18 + validates :mode, inclusion: { in: MODES }

19 + validates :status, inclusion: { in: STATUSES }

20 + validates :source_url, presence: true

21 + validates :target_name, presence: true

22 +

23 + scope :active, -> { where(status: ACTIVE) }

24 + scope :recent, -> { order(created_at: :desc) }

25 +

26 + def mirror?

27 + mode == "mirror"

28 + end

29 +

30 + def migrate?

31 + mode == "migrate"

32 + end

33 +

34 + def terminal?

35 + TERMINAL.include?(status)

36 + end

37 +

38 + def failed?

39 + status == "failed"

40 + end

41 +

42 + def done?

43 + status == "done"

44 + end

45 +

46 + # A retry is allowed only from a failed state; re-running from any active or

47 + # done state would risk duplicate work against a live import.

48 + def retryable?

49 + failed?

50 + end

51 +

52 + # Human-friendly one-liner for the progress UI.

53 + def status_label

54 + {

55 + "queued" => "Queued",

56 + "cloning" => "Cloning source",

57 + "pushing" => "Pushing to siGit",

58 + "fetching_lfs" => "Fetching LFS objects",

59 + "finalizing" => "Finalizing",

60 + "done" => "Imported",

61 + "failed" => "Failed"

62 + }.fetch(status, status.humanize)

63 + end

64 +

65 + # Advance the state machine, stamping timing on entry/exit. Kept tiny so the

66 + # job body reads as a sequence of `transition_to(...)` calls.

67 + def transition_to(new_status, error: nil)

68 + attrs = { status: new_status }

69 + attrs[:started_at] = Time.current if new_status == "cloning" && started_at.nil?

70 + attrs[:finished_at] = Time.current if TERMINAL.include?(new_status)

71 + attrs[:error_message] = error if error

72 + update!(attrs)

73 + end

74 +end

app/models/user.rb

+13

@@ -8,6 +8,19 @@ class User < ApplicationRecord

8 has_one :subscription, dependent: :destroy

9 has_many :cloud_usages, dependent: :destroy

10 has_many :cloud_sessions, dependent: :destroy

11 + has_one :github_connection, dependent: :destroy

12 + has_many :repository_imports, dependent: :destroy

13 +

14 + # True when the user has linked their GitHub account for imports.

15 + def github_connected?

16 + github_connection.present?

17 + end

18 +

19 + # Total reported size (KB) of imports currently in flight, used to enforce the

20 + # per-user concurrent import volume cap without counting finished ones.

21 + def active_import_size_kb

22 + repository_imports.active.sum(:source_size_kb)

23 + end

25 # Whether this user may use siGit Code Cloud (the paid cloud tiers). Free /

26 # unsubscribed users run on-device only.

app/services/git_repository_service.rb

+13

@@ -31,6 +31,19 @@ class GitRepositoryService

31 end

32 end

34 + # Point HEAD at +branch+ so the landing page and default clone check out the

35 + # right branch. Falls back to the first available branch when +branch+ wasn't

36 + # among the pushed refs (e.g. an imported repo whose default differs). Returns

37 + # the branch HEAD ends up on, or nil if the repo has no branches at all.

38 + def self.set_head(path, branch)

39 + target = branch if branch.present? && branch_exists?(path, branch)

40 + target ||= branches(path).first

41 + return nil if target.nil?

42 +

43 + system("git", "--git-dir", path, "symbolic-ref", "HEAD", "refs/heads/#{target}", exception: false)

44 + target

45 + end

46 +

47 def self.default_branch(path)

48 out, _err, status = Open3.capture3("git", "--git-dir", path, "symbolic-ref", "--short", "HEAD")

49 return "main" unless status.success?

app/services/github_api_client.rb new

+126

@@ -0,0 +1,126 @@

1 +# frozen_string_literal: true

2 +

3 +require "net/http"

4 +require "json"

5 +

6 +# Thin GitHub REST client bound to one user's OAuth token, used to list repos

7 +# and read repo metadata for the import flow. Intentionally small — we only need

8 +# a few read endpoints, so this avoids pulling in Octokit.

9 +#

10 +# The token is held only in memory on this instance and sent as a bearer header;

11 +# it is never logged. Rate limits are respected: on a 403/429 with the limit

12 +# exhausted we raise RateLimited carrying the reset time so the caller can back

13 +# off rather than hammer GitHub.

14 +class GithubApiClient

15 + API_ROOT = "https://api.github.com"

16 +

17 + OPEN_TIMEOUT = 5

18 + READ_TIMEOUT = 15

19 +

20 + class Error < StandardError; end

21 + class Unauthorized < Error; end

22 + class RateLimited < Error

23 + attr_reader :reset_at

24 + def initialize(message, reset_at:)

25 + super(message)

26 + @reset_at = reset_at

27 + end

28 + end

29 +

30 + def initialize(access_token)

31 + @access_token = access_token

32 + end

33 +

34 + # The authenticated user's login, or nil if the token is bad. Used to label

35 + # the connection and build clone URLs.

36 + def login

37 + me = get("/user")

38 + me["login"]

39 + rescue Unauthorized

40 + nil

41 + end

42 +

43 + # Lists the authenticated user's repositories, newest-pushed first. Paginates

44 + # internally up to +max+ repos (a safety cap so a user with thousands of repos

45 + # can't make us page forever). Returns an array of normalized hashes.

46 + def list_repositories(max: 300, per_page: 50)

47 + repos = []

48 + page = 1

49 + loop do

50 + batch = get("/user/repos", affiliation: "owner,collaborator",

51 + sort: "pushed", per_page: per_page, page: page)

52 + break if batch.blank?

53 +

54 + repos.concat(batch.map { |r| normalize_repo(r) })

55 + break if batch.length < per_page || repos.length >= max

56 +

57 + page += 1

58 + end

59 + repos.first(max)

60 + end

61 +

62 + # Metadata for a single repo ("owner/name"), normalized. nil if not found.

63 + def repository(full_name)

64 + normalize_repo(get("/repos/#{full_name}"))

65 + rescue Error

66 + nil

67 + end

68 +

69 + private

70 +

71 + def normalize_repo(r)

72 + {

73 + full_name: r["full_name"],

74 + name: r["name"],

75 + description: r["description"],

76 + private: r["private"],

77 + default_branch: r["default_branch"],

78 + clone_url: r["clone_url"],

79 + homepage: r["homepage"],

80 + topics: r["topics"] || [],

81 + size_kb: r["size"], # GitHub reports size in KB

82 + pushed_at: r["pushed_at"],

83 + archived: r["archived"],

84 + fork: r["fork"]

85 + }

86 + end

87 +

88 + def get(path, **query)

89 + uri = URI.parse("#{API_ROOT}#{path}")

90 + uri.query = URI.encode_www_form(query) if query.any?

91 +

92 + req = Net::HTTP::Get.new(uri)

93 + req["Authorization"] = "Bearer #{@access_token}"

94 + req["Accept"] = "application/vnd.github+json"

95 + req["X-GitHub-Api-Version"] = "2022-11-28"

96 + req["User-Agent"] = "sigit-si-importer"

97 +

98 + res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true,

99 + open_timeout: OPEN_TIMEOUT, read_timeout: READ_TIMEOUT) do |http|

100 + http.request(req)

101 + end

102 +

103 + handle(res)

104 + end

105 +

106 + def handle(res)

107 + case res

108 + when Net::HTTPSuccess

109 + res.body.present? ? JSON.parse(res.body) : nil

110 + when Net::HTTPUnauthorized

111 + raise Unauthorized, "GitHub token is invalid or expired."

112 + when Net::HTTPForbidden, Net::HTTPTooManyRequests

113 + # Distinguish a real rate-limit from a permission error by the remaining

114 + # header, so we only tell the caller to back off when it actually should.

115 + if res["x-ratelimit-remaining"].to_i.zero? && res["x-ratelimit-reset"].present?

116 + reset_at = Time.at(res["x-ratelimit-reset"].to_i)

117 + raise RateLimited.new("GitHub API rate limit reached.", reset_at: reset_at)

118 + end

119 + raise Error, "GitHub denied the request (HTTP #{res.code})."

120 + else

121 + raise Error, "GitHub request failed (HTTP #{res.code})."

122 + end

123 + rescue JSON::ParserError

124 + raise Error, "GitHub returned an unreadable response."

125 + end

126 +end

app/services/github_oauth_service.rb new

+115

@@ -0,0 +1,115 @@

1 +# frozen_string_literal: true

2 +

3 +require "net/http"

4 +require "json"

5 +require "securerandom"

6 +

7 +# The GitHub OAuth app siGit uses to import repositories.

8 +#

9 +# This is a *separate* OAuth app from smbCloud "Continue with GitHub" sign-in.

10 +# Sign-in authenticates the user to siGit (via smbCloud) and gives us no GitHub

11 +# API access. Importing needs a GitHub token that can read the user's repos and

12 +# clone private ones, so we run our own minimal OAuth app:

13 +#

14 +# 1. #authorize_url — send the user to GitHub to grant access.

15 +# 2. GitHub redirects back to our callback with ?code=…&state=…

16 +# 3. #exchange_code — trade the code for an access token, stored encrypted in

17 +# GithubConnection.

18 +#

19 +# Requires:

20 +# GITHUB_IMPORT_CLIENT_ID

21 +# GITHUB_IMPORT_CLIENT_SECRET

22 +#

23 +# The token never reaches the browser; only the server holds it.

24 +class GithubOauthService

25 + class ConfigurationError < StandardError; end

26 + class ExchangeError < StandardError; end

27 +

28 + AUTHORIZE_URL = "https://github.com/login/oauth/authorize"

29 + TOKEN_URL = "https://github.com/login/oauth/access_token"

30 +

31 + # Full read/write to private repos vs. public repos only. We ask for the

32 + # narrowest scope that covers what the user chose to import.

33 + SCOPE_PRIVATE = "repo"

34 + SCOPE_PUBLIC = "public_repo"

35 +

36 + OPEN_TIMEOUT = 5

37 + READ_TIMEOUT = 15

38 +

39 + class << self

40 + def configured?

41 + client_id.present? && client_secret.present?

42 + end

43 +

44 + # A CSRF token to round-trip through the `state` param. Stored in the session

45 + # before redirecting and compared on callback.

46 + def generate_state

47 + SecureRandom.urlsafe_base64(24)

48 + end

49 +

50 + # URL to send the user's browser to. +include_private+ picks the scope.

51 + def authorize_url(redirect_uri:, state:, include_private: true)

52 + raise ConfigurationError, "GitHub import OAuth app is not configured." unless configured?

53 +

54 + params = {

55 + client_id: client_id,

56 + redirect_uri: redirect_uri,

57 + scope: include_private ? SCOPE_PRIVATE : SCOPE_PUBLIC,

58 + state: state,

59 + allow_signup: "false"

60 + }

61 + "#{AUTHORIZE_URL}?#{URI.encode_www_form(params)}"

62 + end

63 +

64 + # Exchanges the OAuth +code+ for an access token. Returns a hash:

65 + # { access_token:, scope:, token_type: }

66 + # Raises ExchangeError on any failure (never leaks the secret or the raw

67 + # GitHub error to callers/logs).

68 + def exchange_code(code:, redirect_uri:)

69 + raise ConfigurationError, "GitHub import OAuth app is not configured." unless configured?

70 +

71 + uri = URI.parse(TOKEN_URL)

72 + req = Net::HTTP::Post.new(uri)

73 + req["Accept"] = "application/json"

74 + req.set_form_data(

75 + client_id: client_id,

76 + client_secret: client_secret,

77 + code: code,

78 + redirect_uri: redirect_uri

79 + )

80 +

81 + res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true,

82 + open_timeout: OPEN_TIMEOUT, read_timeout: READ_TIMEOUT) do |http|

83 + http.request(req)

84 + end

85 +

86 + unless res.is_a?(Net::HTTPSuccess)

87 + raise ExchangeError, "GitHub rejected the authorization (HTTP #{res.code})."

88 + end

89 +

90 + body = JSON.parse(res.body)

91 + token = body["access_token"]

92 + if token.blank?

93 + # body["error"] is safe to surface (e.g. "bad_verification_code"); it

94 + # contains no secret.

95 + raise ExchangeError, "GitHub did not return an access token (#{body['error'] || 'unknown error'})."

96 + end

97 +

98 + {

99 + access_token: token,

100 + scope: body["scope"].to_s,

101 + token_type: body["token_type"].presence || "bearer"

102 + }

103 + rescue JSON::ParserError

104 + raise ExchangeError, "GitHub returned an unreadable token response."

105 + end

106 +

107 + def client_id

108 + ENV["GITHUB_IMPORT_CLIENT_ID"].presence

109 + end

110 +

111 + def client_secret

112 + ENV["GITHUB_IMPORT_CLIENT_SECRET"].presence

113 + end

114 + end

115 +end

app/services/import_url_validator.rb new

+135

@@ -0,0 +1,135 @@

1 +# frozen_string_literal: true

2 +

3 +require "uri"

4 +require "resolv"

5 +require "ipaddr"

6 +

7 +# Validates a user-supplied git URL before we hand it to `git clone`, closing

8 +# the SSRF hole that "import by URL" would otherwise open: without this, a user

9 +# could point the importer at `http://169.254.169.254/…` (cloud metadata) or an

10 +# internal `http://10.0.0.5/…` service and have the worker fetch it.

11 +#

12 +# Rules:

13 +# - scheme must be http or https (no ssh://, file://, git://, ftp://, …)

14 +# - a host must be present

15 +# - the host must not resolve to a private, loopback, link-local, or otherwise

16 +# non-public address (checked for every A/AAAA record, and for literal IPs)

17 +# - userinfo (user:pass@) is rejected — credentials belong in the token field,

18 +# not smuggled through the URL

19 +#

20 +# Usage:

21 +# ImportUrlValidator.validate!(url) # => normalized URL string, or raises

22 +# ImportUrlValidator.safe?(url) # => true/false

23 +#

24 +# DNS resolution here is advisory (it reduces the attack surface); the clone

25 +# itself still runs sandboxed with a timeout because DNS can rebind between this

26 +# check and the fetch.

27 +class ImportUrlValidator

28 + class InvalidUrl < StandardError; end

29 +

30 + ALLOWED_SCHEMES = %w[http https].freeze

31 +

32 + # CIDR blocks that must never be reachable from an import. Covers loopback,

33 + # RFC1918 private ranges, link-local (incl. 169.254.169.254 metadata),

34 + # carrier-grade NAT, and their IPv6 equivalents (ULA, link-local, mapped v4).

35 + BLOCKED_RANGES = [

36 + "0.0.0.0/8", # "this" network

37 + "10.0.0.0/8", # private

38 + "100.64.0.0/10", # carrier-grade NAT

39 + "127.0.0.0/8", # loopback

40 + "169.254.0.0/16", # link-local (AWS/GCP metadata lives at 169.254.169.254)

41 + "172.16.0.0/12", # private

42 + "192.0.0.0/24", # IETF protocol assignments

43 + "192.168.0.0/16", # private

44 + "198.18.0.0/15", # benchmarking

45 + "::1/128", # IPv6 loopback

46 + "fc00::/7", # IPv6 unique local

47 + "fe80::/10", # IPv6 link-local

48 + "::ffff:0:0/96" # IPv4-mapped IPv6 (re-checked as v4 below)

49 + ].map { |c| IPAddr.new(c) }.freeze

50 +

51 + class << self

52 + # Returns a normalized URL string when +raw+ is a safe public http(s) git

53 + # URL, otherwise raises InvalidUrl with a user-facing message.

54 + def validate!(raw)

55 + url = raw.to_s.strip

56 + raise InvalidUrl, "Enter a repository URL." if url.empty?

57 +

58 + uri = begin

59 + URI.parse(url)

60 + rescue URI::InvalidURIError

61 + raise InvalidUrl, "That doesn't look like a valid URL."

62 + end

63 +

64 + unless ALLOWED_SCHEMES.include?(uri.scheme)

65 + raise InvalidUrl, "Only http(s) git URLs are supported."

66 + end

67 +

68 + raise InvalidUrl, "The URL is missing a host." if uri.host.blank?

69 +

70 + if uri.userinfo.present?

71 + raise InvalidUrl, "Don't put credentials in the URL; connect the account or use a token instead."

72 + end

73 +

74 + resolve_and_guard!(uri.host)

75 +

76 + url

77 + end

78 +

79 + def safe?(raw)

80 + validate!(raw)

81 + true

82 + rescue InvalidUrl

83 + false

84 + end

85 +

86 + private

87 +

88 + # Rejects the host if it's a blocked literal IP, or if any address it

89 + # resolves to is blocked. A resolution failure is treated as unsafe.

90 + def resolve_and_guard!(host)

91 + addresses =

92 + if literal_ip?(host)

93 + [ host ]

94 + else

95 + resolve(host)

96 + end

97 +

98 + raise InvalidUrl, "Could not resolve that host." if addresses.empty?

99 +

100 + addresses.each do |addr|

101 + ip = IPAddr.new(addr)

102 + # Normalize IPv4-mapped IPv6 (::ffff:10.0.0.1) down to the v4 form so it

103 + # can't slip past the v4 blocklist.

104 + ip = ip.ipv4_mapped? ? ip.native : ip if ip.ipv6?

105 + if blocked?(ip)

106 + raise InvalidUrl, "That host points at a private or internal address, which isn't allowed."

107 + end

108 + end

109 + rescue IPAddr::InvalidAddressError

110 + raise InvalidUrl, "That host resolved to an address we couldn't verify."

111 + end

112 +

113 + def blocked?(ip)

114 + BLOCKED_RANGES.any? { |range| range.include?(ip) }

115 + end

116 +

117 + def literal_ip?(host)

118 + IPAddr.new(host)

119 + true

120 + rescue IPAddr::InvalidAddressError

121 + false

122 + end

123 +

124 + def resolve(host)

125 + Resolv::DNS.open do |dns|

126 + dns.timeouts = 3

127 + v4 = dns.getresources(host, Resolv::DNS::Resource::IN::A).map { |r| r.address.to_s }

128 + v6 = dns.getresources(host, Resolv::DNS::Resource::IN::AAAA).map { |r| r.address.to_s }

129 + v4 + v6

130 + end

131 + rescue Resolv::ResolvError, Resolv::ResolvTimeout

132 + []

133 + end

134 + end

135 +end

app/services/mirror_sync_service.rb new

+112

@@ -0,0 +1,112 @@

1 +# frozen_string_literal: true

2 +

3 +require "open3"

4 +require "tempfile"

5 +require "timeout"

6 +require "base64"

7 +

8 +# Re-syncs a mirror repository from its upstream: force-fetches all heads and

9 +# tags into the (read-only) bare repo so it tracks the upstream exactly. Because

10 +# mirrors reject local pushes, force-updating refs here can never clobber unique

11 +# local work — the upstream is the single source of truth by design.

12 +#

13 +# Shares the isolation posture of RepositoryImportService: hard timeout with

14 +# process-group kill, no interactive prompts, credentials passed as an HTTP

15 +# header (never the URL or stored config) and scrubbed from captured output.

16 +class MirrorSyncService

17 + class SyncError < StandardError; end

18 + class SyncTimeout < SyncError; end

19 +

20 + DEFAULT_TIMEOUT = Integer(ENV.fetch("MIRROR_SYNC_TIMEOUT", 900)) # 15 min

21 +

22 + def self.sync(repository, timeout: DEFAULT_TIMEOUT)

23 + new(repository, timeout:).call

24 + end

25 +

26 + def initialize(repository, timeout: DEFAULT_TIMEOUT)

27 + @repository = repository

28 + @timeout = timeout

29 + end

30 +

31 + # Fetches upstream into the repo. Raises SyncError on failure. On success the

32 + # caller stamps mirror_synced_at / mirror_status.

33 + def call

34 + raise SyncError, "Not a mirror." unless @repository.mirror?

35 + raise SyncError, "Mirror has no upstream URL." if @repository.upstream_url.blank?

36 +

37 + # Re-validate the upstream every sync: DNS could have been repointed at an

38 + # internal address since the import, so this closes the rebinding window.

39 + ImportUrlValidator.validate!(@repository.upstream_url)

40 +

41 + args = [ "git" ]

42 + args += credential_config

43 + args += [ "-C", @repository.disk_path, "fetch", "--prune", @repository.upstream_url,

44 + "+refs/heads/*:refs/heads/*", "+refs/tags/*:refs/tags/*" ]

45 +

46 + ok, output = run(args, timeout: @timeout)

47 + raise SyncError, "Fetch failed: #{first_error_line(output)}" unless ok

48 +

49 + true

50 + end

51 +

52 + private

53 +

54 + def credential_config

55 + token = @repository.upstream_token

56 + return [] if token.blank?

57 +

58 + basic = Base64.strict_encode64("x-access-token:#{token}")

59 + [ "-c", "http.extraHeader=Authorization: Basic #{basic}" ]

60 + end

61 +

62 + def run(cmd, timeout:)

63 + out = Tempfile.new("sigit-sync-out")

64 + begin

65 + pid = Process.spawn(git_env, *cmd, out: out.path, err: [ :child, :out ], pgroup: true)

66 + begin

67 + Timeout.timeout(timeout) { Process.wait(pid) }

68 + rescue Timeout::Error

69 + kill_group(pid)

70 + raise SyncTimeout, "mirror sync exceeded #{timeout}s and was terminated."

71 + end

72 + [ $?.success?, scrub(File.read(out.path)) ]

73 + ensure

74 + out.close!

75 + end

76 + end

77 +

78 + def kill_group(pid)

79 + begin

80 + Process.kill("TERM", -pid)

81 + sleep 2

82 + Process.kill("KILL", -pid)

83 + rescue Errno::ESRCH, Errno::EPERM

84 + nil

85 + end

86 + begin

87 + Process.wait(pid)

88 + rescue Errno::ECHILD

89 + nil

90 + end

91 + end

92 +

93 + def git_env

94 + {

95 + "GIT_TERMINAL_PROMPT" => "0",

96 + "GIT_ASKPASS" => "/bin/echo",

97 + "GCM_INTERACTIVE" => "never",

98 + "GIT_LFS_SKIP_SMUDGE" => "1"

99 + }

100 + end

101 +

102 + def scrub(text)

103 + token = @repository.upstream_token

104 + return text if token.blank?

105 +

106 + text.to_s.gsub(token, "***")

107 + end

108 +

109 + def first_error_line(output)

110 + output.to_s.each_line.map(&:strip).reject(&:empty?).last.presence || "unknown error"

111 + end

112 +end

app/services/repository_import_service.rb new

+240

@@ -0,0 +1,240 @@

1 +# frozen_string_literal: true

2 +

3 +require "open3"

4 +require "fileutils"

5 +require "tmpdir"

6 +require "tempfile"

7 +require "timeout"

8 +require "base64"

9 +

10 +# The git mechanics of an import: mirror-clone a source repo in a sandbox, then

11 +# push every ref (all branches and tags, full history, no rewrites) into an

12 +# existing bare siGit repo. LFS objects are fetched when git-lfs is available,

13 +# and clearly reported as skipped otherwise.

14 +#

15 +# This class is deliberately isolation-focused — every guardrail that keeps a

16 +# hung or malicious remote from wedging a worker lives here:

17 +# - the clone runs in a fresh temp dir that is always removed,

18 +# - every git invocation runs with a hard timeout and is killed (whole process

19 +# group) if it overruns,

20 +# - `GIT_TERMINAL_PROMPT=0` makes auth failures fail fast instead of hanging,

21 +# - the cloned size is capped, so one repo can't fill the disk,

22 +# - credentials are passed via an HTTP header (not the URL, not the stored

23 +# remote config) and scrubbed from any captured output before it is logged.

24 +#

25 +# It does not touch the database or create the destination repo — the caller

26 +# (RepositoryImportJob) owns lifecycle, status, and cleanup.

27 +class RepositoryImportService

28 + class ImportError < StandardError; end

29 + class SizeExceeded < ImportError; end

30 + class CloneTimeout < ImportError; end

31 +

32 + # A single git call may not pin a worker forever. Overridable for very large

33 + # legitimate repos via env.

34 + DEFAULT_TIMEOUT = Integer(ENV.fetch("IMPORT_GIT_TIMEOUT", 1800)) # 30 min

35 +

36 + # Per-repo on-disk cap. GitHub reports size in KB; we also re-check the actual

37 + # clone so URL imports (no reported size) are bounded too.

38 + DEFAULT_MAX_SIZE_KB = Integer(ENV.fetch("IMPORT_MAX_REPO_SIZE_KB", 2_000_000)) # ~2 GB

39 +

40 + Result = Struct.new(:lfs_detected, :lfs_skipped, :size_kb, keyword_init: true)

41 +

42 + # Clones +source_url+ (with optional +credential+ token for private sources)

43 + # and pushes all refs into the existing bare repo at +dest_path+. Yields a

44 + # status symbol (:cloning, :pushing, :fetching_lfs) to +progress+ as it moves

45 + # through the phases. Returns a Result. Raises ImportError (or a subclass) on

46 + # failure; the caller is responsible for cleaning up +dest_path+.

47 + def self.clone_and_push(source_url:, dest_path:, credential: nil,

48 + max_size_kb: DEFAULT_MAX_SIZE_KB, timeout: DEFAULT_TIMEOUT, progress: nil)

49 + new(source_url:, dest_path:, credential:, max_size_kb:, timeout:, progress:).call

50 + end

51 +

52 + def initialize(source_url:, dest_path:, credential:, max_size_kb:, timeout:, progress:)

53 + @source_url = source_url

54 + @dest_path = dest_path

55 + @credential = credential

56 + @max_size_kb = max_size_kb

57 + @timeout = timeout

58 + @progress = progress

59 + end

60 +

61 + def call

62 + Dir.mktmpdir("sigit-import-") do |workdir|

63 + mirror = File.join(workdir, "source.git")

64 +

65 + emit(:cloning)

66 + clone_mirror(mirror)

67 + enforce_size!(mirror)

68 +

69 + emit(:pushing)

70 + push_all_refs(mirror)

71 +

72 + emit(:fetching_lfs)

73 + lfs = handle_lfs(mirror)

74 +

75 + Result.new(

76 + lfs_detected: lfs[:detected],

77 + lfs_skipped: lfs[:skipped],

78 + size_kb: dir_size_kb(@dest_path)

79 + )

80 + end

81 + end

82 +

83 + private

84 +

85 + def emit(phase)

86 + @progress&.call(phase)

87 + end

88 +

89 + def clone_mirror(dest)

90 + args = [ "git" ]

91 + args += credential_config

92 + args += [ "clone", "--mirror", @source_url, dest ]

93 +

94 + ok, output = run(args, timeout: @timeout)

95 + raise ImportError, "Clone failed: #{first_error_line(output)}" unless ok

96 + end

97 +

98 + # Push every ref verbatim (heads + tags), preserving history exactly. The

99 + # refspec form (rather than `--mirror`) keeps siGit's internal refs, if any,

100 + # from being pruned, and never rewrites commits.

101 + def push_all_refs(mirror)

102 + ok, output = run([ "git", "-C", mirror, "push", @dest_path, "+refs/heads/*:refs/heads/*", "+refs/tags/*:refs/tags/*" ],

103 + timeout: @timeout)

104 + raise ImportError, "Push failed: #{first_error_line(output)}" unless ok

105 + end

106 +

107 + # Best-effort LFS handling. Detection is cheap (does any ref carry a

108 + # `.gitattributes` with `filter=lfs`); fetching requires the git-lfs binary.

109 + # When it's missing we don't fail — the LFS *pointer* files are ordinary git

110 + # blobs and are already imported with the history; only the large binaries are

111 + # skipped, which we report so the UI can say so plainly.

112 + def handle_lfs(mirror)

113 + return { detected: false, skipped: false } unless lfs_used?(mirror)

114 + return { detected: true, skipped: true } unless lfs_available?

115 +

116 + ok, _ = run([ "git", "-C", mirror, "lfs", "fetch", "--all" ], timeout: @timeout)

117 + return { detected: true, skipped: true } unless ok

118 +

119 + # Copy the fetched object store into the destination so siGit holds the

120 + # binaries too. Non-fatal if it doesn't land.

121 + src_lfs = File.join(mirror, "lfs")

122 + if Dir.exist?(src_lfs)

123 + FileUtils.cp_r(src_lfs, File.join(@dest_path, "lfs"))

124 + { detected: true, skipped: false }

125 + else

126 + { detected: true, skipped: true }

127 + end

128 + rescue StandardError => e

129 + Rails.logger.warn("LFS import step failed (continuing): #{e.class}")

130 + { detected: true, skipped: true }

131 + end

132 +

133 + def lfs_used?(mirror)

134 + # Check the tip of every branch for a .gitattributes that enables the lfs

135 + # filter. Cheap and covers the overwhelmingly common case. The blob read

136 + # goes through GitRepositoryService (the single git-read layer) rather than

137 + # shelling out again here.

138 + ok, refs = run([ "git", "-C", mirror, "for-each-ref", "--format=%(refname)", "refs/heads/" ], timeout: 30)

139 + return false unless ok

140 +

141 + refs.each_line.map(&:strip).reject(&:empty?).any? do |ref|

142 + content = GitRepositoryService.file_content(mirror, ref, ".gitattributes")

143 + content&.include?("filter=lfs")

144 + end

145 + end

146 +

147 + def lfs_available?

148 + _o, _e, st = Open3.capture3("git", "lfs", "version")

149 + st.success?

150 + rescue StandardError

151 + false

152 + end

153 +

154 + def enforce_size!(mirror)

155 + size = dir_size_kb(mirror)

156 + return if size.nil? || size <= @max_size_kb

157 +

158 + raise SizeExceeded,

159 + "Repository is #{(size / 1024.0).round} MB, over the #{(@max_size_kb / 1024.0).round} MB import limit."

160 + end

161 +

162 + # ── process execution ────────────────────────────────────────────────────

163 +

164 + # Runs +cmd+ with a hard timeout. On overrun the whole process group is killed

165 + # so a hung `git` (or a child it spawned) can't linger. Returns

166 + # [success_bool, output_string]; output has any credential scrubbed out.

167 + def run(cmd, timeout:)

168 + out = Tempfile.new("sigit-git-out")

169 + begin

170 + pid = Process.spawn(git_env, *cmd, out: out.path, err: [ :child, :out ], pgroup: true)

171 + begin

172 + Timeout.timeout(timeout) { Process.wait(pid) }

173 + rescue Timeout::Error

174 + kill_group(pid)

175 + raise CloneTimeout, "git operation exceeded #{timeout}s and was terminated."

176 + end

177 + success = $?.success?

178 + [ success, scrub(File.read(out.path)) ]

179 + ensure

180 + out.close!

181 + end

182 + end

183 +

184 + def kill_group(pid)

185 + begin

186 + Process.kill("TERM", -pid)

187 + sleep 2

188 + Process.kill("KILL", -pid)

189 + rescue Errno::ESRCH, Errno::EPERM

190 + # already gone

191 + end

192 +

193 + begin

194 + Process.wait(pid)

195 + rescue Errno::ECHILD

196 + nil

197 + end

198 + end

199 +

200 + def git_env

201 + {

202 + # Never block on an interactive auth/host prompt — fail fast instead.

203 + "GIT_TERMINAL_PROMPT" => "0",

204 + "GIT_ASKPASS" => "/bin/echo",

205 + "GCM_INTERACTIVE" => "never",

206 + # Don't try to download LFS blobs during the mirror clone; we handle LFS

207 + # explicitly afterwards.

208 + "GIT_LFS_SKIP_SMUDGE" => "1"

209 + }

210 + end

211 +

212 + # Auth for a private source, injected as an HTTP header via `-c` so the token

213 + # never lands in the URL, the stored remote config, or the clone on disk.

214 + def credential_config

215 + return [] if @credential.blank?

216 +

217 + basic = Base64.strict_encode64("x-access-token:#{@credential}")

218 + [ "-c", "http.extraHeader=Authorization: Basic #{basic}" ]

219 + end

220 +

221 + # Remove the credential from any captured text before it can be logged or

222 + # persisted to the import's error_message.

223 + def scrub(text)

224 + return text if @credential.blank?

225 +

226 + text.to_s.gsub(@credential, "***")

227 + end

228 +

229 + def first_error_line(output)

230 + line = output.to_s.each_line.map(&:strip).reject(&:empty?).last

231 + line.presence || "unknown error"

232 + end

233 +

234 + def dir_size_kb(path)

235 + return nil unless Dir.exist?(path)

236 +

237 + out, _e, st = Open3.capture3("du", "-sk", path)

238 + st.success? ? out.split("\t").first.to_i : nil

239 + end

240 +end

app/views/imports/_status_badge.html.erb new

+18

@@ -0,0 +1,18 @@

1 +<%#

2 + Small status pill for a RepositoryImport. Colors: terminal-done green,

3 + failed red, everything in-flight amber.

4 +%>

5 +<% classes =

6 + if import.done? then "bg-green-900/30 text-green-300 border border-green-800/40"

7 + elsif import.failed? then "bg-red-900/30 text-red-300 border border-red-800/40"

8 + else "bg-amber-900/30 text-amber-300 border border-amber-800/40"

9 + end %>

10 +<span class="inline-flex items-center gap-1.5 text-xs px-2 py-0.5 rounded <%= classes %>">

11 + <% unless import.terminal? %>

12 + <svg class="w-3 h-3 animate-spin" viewBox="0 0 24 24" fill="none">

13 + <circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"></circle>

14 + <path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.4 0 0 5.4 0 12h4z"></path>

15 + </svg>

16 + <% end %>

17 + <%= import.status_label %>

18 +</span>

app/views/imports/index.html.erb new

+28

@@ -0,0 +1,28 @@

1 +<% content_for :title, "Imports" %>

2 +

3 +<div class="max-w-2xl mx-auto px-4 sm:px-6 py-10">

4 + <div class="flex items-center justify-between mb-6">

5 + <h1 class="text-xl font-semibold text-gray-100">Imports</h1>

6 + <%= link_to "Import a repo", new_import_path, class: "btn-primary text-xs py-1.5 px-3" %>

7 + </div>

8 +

9 + <% if @imports.empty? %>

10 + <div class="card px-6 py-12 text-center">

11 + <p class="text-sm text-gray-400 mb-4">You haven't imported anything yet.</p>

12 + <%= link_to "Import from GitHub", new_import_path, class: "btn-primary" %>

13 + </div>

14 + <% else %>

15 + <div class="card divide-y divide-surface-600">

16 + <% @imports.each do |import| %>

17 + <%= link_to import_path(import), class: "flex items-center justify-between px-4 py-3 hover:bg-surface-600" do %>

18 + <div class="min-w-0">

19 + <span class="text-sm text-gray-100"><%= import.target_name %></span>

20 + <span class="text-xs text-gray-500 ml-2"><%= import.mirror? ? "mirror" : "migrate" %></span>

21 + <div class="text-xs text-gray-500"><%= time_ago_in_words(import.created_at) %> ago</div>

22 + </div>

23 + <%= render "imports/status_badge", import: import %>

24 + <% end %>

25 + <% end %>

26 + </div>

27 + <% end %>

28 +</div>

app/views/imports/new.html.erb new

+155

@@ -0,0 +1,155 @@

1 +<% content_for :title, "Import from GitHub" %>

2 +

3 +<div class="max-w-3xl mx-auto px-4 sm:px-6 py-10">

4 + <h1 class="text-xl font-semibold text-gray-100 mb-1">Import a repository</h1>

5 + <p class="text-sm text-gray-400 mb-8">

6 + Bring a repo to siGit with its full history — every branch and tag, no rewrites.

7 + </p>

8 +

9 + <%# ── MIGRATE vs MIRROR explainer ─────────────────────────────────────── %>

10 + <div class="grid grid-cols-1 sm:grid-cols-2 gap-3 mb-8">

11 + <div class="card px-4 py-3">

12 + <div class="text-sm font-medium text-gray-100 mb-0.5">Migrate</div>

13 + <p class="text-xs text-gray-400">Copy the repo once. siGit becomes the source of truth and you can push here.</p>

14 + </div>

15 + <div class="card px-4 py-3">

16 + <div class="text-sm font-medium text-gray-100 mb-0.5">Mirror <span class="badge-gray ml-1">risk-free</span></div>

17 + <p class="text-xs text-gray-400">Keep GitHub as upstream. siGit auto-syncs and stays read-only until you detach.</p>

18 + </div>

19 + </div>

20 +

21 + <%# ── GitHub section ───────────────────────────────────────────────────── %>

22 + <section class="mb-10">

23 + <div class="flex items-center justify-between mb-3">

24 + <h2 class="text-base font-semibold text-gray-100">From GitHub</h2>

25 + <% if @connection %>

26 + <div class="flex items-center gap-3 text-xs text-gray-400">

27 + <span>Connected<%= " as @#{@connection.github_login}" if @connection.github_login %></span>

28 + <%= button_to "Disconnect", github_import_disconnect_path, method: :delete,

29 + class: "text-red-400 hover:underline", form: { class: "inline" } %>

30 + </div>

31 + <% end %>

32 + </div>

33 +

34 + <% unless GithubOauthService.configured? %>

35 + <div class="card px-4 py-3 text-sm text-amber-300 bg-amber-900/10 border-amber-800/40">

36 + GitHub import isn't configured on this server. You can still import any public repo by URL below.

37 + </div>

38 + <% end %>

39 +

40 + <% if @connection.nil? %>

41 + <div class="card px-6 py-8 text-center">

42 + <p class="text-sm text-gray-400 mb-4">Connect GitHub to see and pick your repositories.</p>

43 + <div class="flex flex-col sm:flex-row items-center justify-center gap-3">

44 + <%= link_to github_import_connect_path, class: "btn-primary" do %>

45 + Connect GitHub (incl. private)

46 + <% end %>

47 + <%= link_to github_import_connect_path(visibility: "public"), class: "btn-ghost" do %>

48 + Public repos only

49 + <% end %>

50 + </div>

51 + <p class="text-xs text-gray-500 mt-3">We request the narrowest scope for what you choose. Your token is stored encrypted and never shown.</p>

52 + </div>

53 + <% elsif @github_repos.nil? %>

54 + <div class="card px-4 py-3 text-sm text-amber-300">

55 + Couldn't load your GitHub repositories right now (rate limit or a hiccup). Try again shortly.

56 + </div>

57 + <% elsif @github_repos.empty? %>

58 + <div class="card px-4 py-3 text-sm text-gray-400">No repositories found on your GitHub account.</div>

59 + <% else %>

60 + <%= form_with url: imports_path, method: :post, data: { turbo: false }, class: "space-y-4" do %>

61 + <input type="hidden" name="source" value="github">

62 +

63 + <div class="flex items-center gap-4">

64 + <span class="form-label mb-0">Mode</span>

65 + <label class="flex items-center gap-1.5 text-sm text-gray-200">

66 + <input type="radio" name="mode" value="migrate" checked> Migrate

67 + </label>

68 + <label class="flex items-center gap-1.5 text-sm text-gray-200">

69 + <input type="radio" name="mode" value="mirror"> Mirror

70 + </label>

71 + </div>

72 +

73 + <div class="card divide-y divide-surface-600 max-h-96 overflow-y-auto">

74 + <% @github_repos.each do |repo| %>

75 + <label class="flex items-center gap-3 px-4 py-2.5 hover:bg-surface-600 cursor-pointer">

76 + <input type="checkbox" name="repos[]" value="<%= repo[:full_name] %>" class="shrink-0">

77 + <div class="min-w-0 flex-1">

78 + <div class="flex items-center gap-2">

79 + <span class="text-sm font-medium text-gray-100 truncate"><%= repo[:full_name] %></span>

80 + <% if repo[:private] %><span class="badge-gray">Private</span><% end %>

81 + <% if repo[:archived] %><span class="badge-gray">Archived</span><% end %>

82 + </div>

83 + <% if repo[:description].present? %>

84 + <p class="text-xs text-gray-400 truncate"><%= repo[:description] %></p>

85 + <% end %>

86 + </div>

87 + <div class="shrink-0 text-right text-xs text-gray-500">

88 + <div><%= number_to_human_size((repo[:size_kb] || 0) * 1024) %></div>

89 + <% if repo[:pushed_at].present? %>

90 + <div>pushed <%= time_ago_in_words(Time.parse(repo[:pushed_at])) rescue "—" %> ago</div>

91 + <% end %>

92 + </div>

93 + </label>

94 + <% end %>

95 + </div>

96 +

97 + <div class="flex items-center gap-3">

98 + <%= submit_tag "Import selected", class: "btn-primary cursor-pointer" %>

99 + <span class="text-xs text-gray-500">Select multiple to queue a bulk import.</span>

100 + </div>

101 + <% end %>

102 + <% end %>

103 + </section>

104 +

105 + <%# ── Import by URL ────────────────────────────────────────────────────── %>

106 + <section class="mb-10">

107 + <h2 class="text-base font-semibold text-gray-100 mb-3">Or import by URL</h2>

108 + <%= form_with url: imports_path, method: :post, data: { turbo: false }, class: "space-y-4" do %>

109 + <input type="hidden" name="source" value="url">

110 + <div>

111 + <label class="form-label">Public git URL</label>

112 + <input type="url" name="url" class="form-input" placeholder="https://github.com/owner/repo.git" required>

113 + <p class="text-xs text-gray-500 mt-1">Any public http(s) git URL. Private URLs need the GitHub connection above.</p>

114 + </div>

115 + <div class="flex items-end gap-3">

116 + <div class="flex-1">

117 + <label class="form-label">Repository name <span class="text-gray-500">(optional)</span></label>

118 + <input type="text" name="name" class="form-input" placeholder="derived from the URL">

119 + </div>

120 + <div class="flex items-center gap-3 pb-2.5">

121 + <label class="flex items-center gap-1.5 text-sm text-gray-200">

122 + <input type="radio" name="mode" value="migrate" checked> Migrate

123 + </label>

124 + <label class="flex items-center gap-1.5 text-sm text-gray-200">

125 + <input type="radio" name="mode" value="mirror"> Mirror

126 + </label>

127 + </div>

128 + </div>

129 + <%= submit_tag "Import from URL", class: "btn-secondary cursor-pointer" %>

130 + <% end %>

131 + </section>

132 +

133 + <p class="text-xs text-gray-500 mb-8">

134 + Issues, pull requests, wikis, and releases are <strong>not imported</strong> in this version — only the

135 + git repository, its history, branches, and tags.

136 + </p>

137 +

138 + <%# ── Recent imports ───────────────────────────────────────────────────── %>

139 + <% if @recent_imports.present? %>

140 + <section>

141 + <h2 class="text-base font-semibold text-gray-100 mb-3">Recent imports</h2>

142 + <div class="card divide-y divide-surface-600">

143 + <% @recent_imports.each do |import| %>

144 + <%= link_to import_path(import), class: "flex items-center justify-between px-4 py-2.5 hover:bg-surface-600" do %>

145 + <div class="min-w-0">

146 + <span class="text-sm text-gray-100"><%= import.target_name %></span>

147 + <span class="text-xs text-gray-500 ml-2"><%= import.mirror? ? "mirror" : "migrate" %></span>

148 + </div>

149 + <%= render "imports/status_badge", import: import %>

150 + <% end %>

151 + <% end %>

152 + </div>

153 + </section>

154 + <% end %>

155 +</div>

app/views/imports/show.html.erb new

+84

@@ -0,0 +1,84 @@

1 +<% content_for :title, "Importing #{@import.target_name}" %>

2 +

3 +<%# Live progress: poll every few seconds until the import reaches a terminal

4 + state. A meta refresh keeps this dependency-free (no JS needed) and stops

5 + once done/failed. %>

6 +<% unless @import.terminal? %>

7 + <% content_for :head do %>

8 + <meta http-equiv="refresh" content="4">

9 + <% end %>

10 +<% end %>

11 +

12 +<div class="max-w-2xl mx-auto px-4 sm:px-6 py-12">

13 + <div class="flex items-center justify-between mb-6">

14 + <div>

15 + <h1 class="text-xl font-semibold text-gray-100"><%= @import.target_name %></h1>

16 + <p class="text-sm text-gray-400">

17 + <%= @import.mirror? ? "Mirror" : "Migrate" %> · from <%= @import.source_url %>

18 + </p>

19 + </div>

20 + <%= render "imports/status_badge", import: @import %>

21 + </div>

22 +

23 + <%# Ordered phase checklist. %>

24 + <% phases = [

25 + ["queued", "Queued"],

26 + ["cloning", "Cloning source"],

27 + ["pushing", "Pushing to siGit"],

28 + ["fetching_lfs", "Fetching LFS objects"],

29 + ["finalizing", "Finalizing"],

30 + ["done", "Done"]

31 + ]

32 + order = RepositoryImport::STATUSES

33 + current_idx = order.index(@import.status) || 0 %>

34 +

35 + <div class="card divide-y divide-surface-600">

36 + <% phases.each do |key, label|

37 + idx = order.index(key)

38 + done = !@import.failed? && idx && idx < current_idx

39 + active = @import.status == key %>

40 + <div class="flex items-center gap-3 px-4 py-3">

41 + <% if done || (@import.done? && key == "done") %>

42 + <svg class="w-4 h-4 text-green-400" viewBox="0 0 16 16" fill="currentColor"><path d="M13.78 4.22a.75.75 0 0 1 0 1.06l-7.25 7.25a.75.75 0 0 1-1.06 0L1.72 8.84a.75.75 0 1 1 1.06-1.06l3.22 3.22 6.72-6.72a.75.75 0 0 1 1.06 0Z"/></svg>

43 + <% elsif active && !@import.terminal? %>

44 + <svg class="w-4 h-4 text-amber-400 animate-spin" viewBox="0 0 24 24" fill="none"><circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"></circle><path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.4 0 0 5.4 0 12h4z"></path></svg>

45 + <% else %>

46 + <span class="w-4 h-4 rounded-full border border-surface-500 inline-block"></span>

47 + <% end %>

48 + <span class="text-sm <%= done || active ? 'text-gray-100' : 'text-gray-500' %>"><%= label %></span>

49 + </div>

50 + <% end %>

51 + </div>

52 +

53 + <% if @import.failed? %>

54 + <div class="border border-red-800/40 bg-red-900/20 rounded px-4 py-3 mt-6">

55 + <p class="text-sm font-medium text-red-300 mb-1">Import failed</p>

56 + <p class="text-sm text-red-400"><%= @import.error_message %></p>

57 + <div class="mt-3">

58 + <%= button_to "Retry import", retry_import_path(@import), method: :post, class: "btn-secondary cursor-pointer" %>

59 + </div>

60 + </div>

61 + <% end %>

62 +

63 + <% if @import.done? && @import.repository %>

64 + <div class="mt-6 space-y-3">

65 + <div class="flex items-center gap-3">

66 + <%= link_to "View repository", repository_path(@import.user.username, @import.repository.name), class: "btn-primary" %>

67 + <% if @import.mirror? %>

68 + <span class="badge-gray">Read-only mirror</span>

69 + <% end %>

70 + </div>

71 + <% if @import.lfs_detected %>

72 + <p class="text-xs <%= @import.lfs_skipped ? 'text-amber-400' : 'text-gray-400' %>">

73 + <% if @import.lfs_skipped %>

74 + Git LFS was detected but its large objects were <strong>not fetched</strong> (LFS unavailable on this server).

75 + Pointer files and full history were imported.

76 + <% else %>

77 + Git LFS objects were fetched and imported.

78 + <% end %>

79 + </p>

80 + <% end %>

81 + <p class="text-xs text-gray-500">Issues, pull requests, and releases were not imported.</p>

82 + </div>

83 + <% end %>

84 +</div>

app/views/layouts/application.html.erb

@@ -14,6 +14,7 @@

14 <%= stylesheet_link_tag "tailwind", "data-turbo-track": "reload" %>

15 <%= javascript_importmap_tags %>

16 <%= csrf_meta_tags %>

17 + <%= yield :head %>

18 </head>

20 <body class="h-full bg-surface-900">

app/views/pages/dashboard.html.erb

+9 -2

@@ -57,8 +57,15 @@

57 <path stroke-linecap="round" stroke-linejoin="round"

58 d="M3.75 9.776c.112-.017.227-.026.344-.026h15.812c.117 0 .232.009.344.026m-16.5 0a2.25 2.25 0 0 0-1.883 2.542l.857 6a2.25 2.25 0 0 0 2.227 1.932H19.05a2.25 2.25 0 0 0 2.227-1.932l.857-6a2.25 2.25 0 0 0-1.883-2.542m-16.5 0V6A2.25 2.25 0 0 1 6 3.75h3.879a1.5 1.5 0 0 1 1.06.44l2.122 2.12a1.5 1.5 0 0 0 1.06.44H18A2.25 2.25 0 0 1 20.25 9v.776"/>

59 </svg>

60 - <p class="text-sm text-gray-400 mb-4">No repositories yet.</p>

61 - <%= link_to new_repository_path, class: "btn-primary" do %>Create your first repository<% end %>

60 + <p class="text-sm text-gray-100 font-medium mb-1">Import your first repo from GitHub</p>

61 + <p class="text-sm text-gray-400 mb-5 max-w-sm mx-auto">

62 + Bring your code over with its full history — or mirror it read-only and keep

63 + GitHub as upstream to try siGit risk-free.

64 + </p>

65 + <div class="flex items-center justify-center gap-3">

66 + <%= link_to new_import_path, class: "btn-primary" do %>Import from GitHub<% end %>

67 + <%= link_to new_repository_path, class: "btn-ghost" do %>Start from scratch<% end %>

68 + </div>

69 </div>

70 <% end %>

71 </main>

app/views/repositories/show.html.erb

+25

@@ -17,8 +17,33 @@

17 <% if @repository.is_private %>

18 <span class="badge-gray ml-1">Private</span>

19 <% end %>

20 + <% if @repository.mirror? %>

21 + <span class="badge-gray ml-1">Mirror · read-only</span>

22 + <% end %>

23 </div>

25 + <% if @repository.mirror? %>

26 + <div class="border border-surface-500 bg-surface-700/40 rounded px-4 py-3 mb-4 flex flex-wrap items-center justify-between gap-3">

27 + <div class="min-w-0">

28 + <p class="text-sm text-gray-200">

29 + Mirrored from <span class="text-gray-100 break-all"><%= @repository.upstream_url %></span>

30 + </p>

31 + <p class="text-xs <%= @repository.mirror_status == 'failed' ? 'text-red-400' : 'text-gray-500' %>">

32 + <%= @repository.mirror_state_label %><%= " — #{@repository.mirror_error}" if @repository.mirror_status == 'failed' && @repository.mirror_error.present? %>

33 + </p>

34 + </div>

35 + <% if signed_in? && current_user == @owner %>

36 + <div class="flex items-center gap-2 shrink-0">

37 + <%= button_to "Sync now", repository_mirror_sync_path(@owner.username, @repository.name),

38 + method: :post, class: "btn-ghost text-xs py-1 px-2 cursor-pointer" %>

39 + <%= button_to "Detach", repository_mirror_detach_path(@owner.username, @repository.name),

40 + method: :post, class: "btn-secondary text-xs py-1 px-2 cursor-pointer",

41 + form: { data: { turbo_confirm: "Detach this mirror? It becomes a normal writable repo and stops syncing from upstream." } } %>

42 + </div>

43 + <% end %>

44 + </div>

45 + <% end %>

46 +

47 <% if @repository.description.present? %>

48 <p class="text-sm text-gray-400 mb-4"><%= @repository.description %></p>

49 <% end %>

app/views/shared/_navbar.html.erb

@@ -45,6 +45,8 @@

45 </div>

46 <%= link_to user_profile_path(current_user.username),

47 class: "block px-3 py-1.5 text-sm text-gray-300 hover:bg-surface-600" do %>Your repositories<% end %>

48 + <%= link_to new_import_path,

49 + class: "block px-3 py-1.5 text-sm text-gray-300 hover:bg-surface-600" do %>Import from GitHub<% end %>

50 <%= link_to settings_path,

51 class: "block px-3 py-1.5 text-sm text-gray-300 hover:bg-surface-600" do %>Settings<% end %>

52 <div class="border-t border-surface-600 mt-1">

config/environments/production.rb

+5 -1

@@ -50,7 +50,11 @@ Rails.application.configure do

50 # config.cache_store = :mem_cache_store

52 # Replace the default in-process and non-durable queuing backend for Active Job.

53 - # config.active_job.queue_adapter = :resque

53 + # Solid Queue (database-backed) runs the long-running repository imports and

54 + # mirror syncs off the web tier. Recurring tasks are declared in

55 + # config/recurring.yml.

56 + config.active_job.queue_adapter = :solid_queue

57 + config.solid_queue.connects_to = { database: { writing: :queue } }

59 # Ignore bad email addresses and do not raise email delivery errors.

60 # Set this to true and configure the email server for immediate delivery to raise delivery errors.

config/initializers/active_record_encryption.rb new

+48

@@ -0,0 +1,48 @@

1 +# frozen_string_literal: true

2 +

3 +# Active Record Encryption keys for data encrypted at rest — currently the

4 +# per-user GitHub OAuth token (GithubConnection#access_token) and the upstream

5 +# mirror credential (Repository#upstream_token).

6 +#

7 +# Keys come from the environment so real secrets never live in the repo:

8 +#

9 +# ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY

10 +# ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY

11 +# ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT

12 +#

13 +# In production these MUST be set (generate with `bin/rails db:encryption:init`);

14 +# we fail loudly at boot if they're missing rather than silently storing tokens

15 +# under a predictable key. In development and test we derive deterministic keys

16 +# from secret_key_base so the app and specs run without extra setup — never used

17 +# for real user tokens.

18 +Rails.application.configure do

19 + enc = config.active_record.encryption

20 +

21 + primary = ENV["ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY"]

22 + deterministic = ENV["ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY"]

23 + salt = ENV["ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT"]

24 +

25 + if primary.blank? || deterministic.blank? || salt.blank?

26 + if Rails.env.production?

27 + raise "Active Record Encryption keys are not configured. Set " \

28 + "ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY, " \

29 + "ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY and " \

30 + "ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT " \

31 + "(generate with `bin/rails db:encryption:init`)."

32 + end

33 +

34 + base = Rails.application.secret_key_base.to_s

35 + base = "sigitsi-dev-encryption" if base.blank?

36 + derive = ->(label) { Digest::SHA256.hexdigest("#{label}:#{base}")[0, 32] }

37 + primary ||= derive.call("ar-enc-primary")

38 + deterministic ||= derive.call("ar-enc-deterministic")

39 + salt ||= derive.call("ar-enc-salt")

40 + end

41 +

42 + enc.primary_key = primary

43 + enc.deterministic_key = deterministic

44 + enc.key_derivation_salt = salt

45 + # Tokens are opaque and never queried by ciphertext, so non-deterministic

46 + # (randomized IV) encryption is the safer default.

47 + enc.support_unencrypted_data = false

48 +end

config/recurring.yml new

+15

@@ -0,0 +1,15 @@

1 +# Solid Queue recurring tasks.

2 +# https://github.com/rails/solid_queue#recurring-tasks

3 +#

4 +# The scheduler enqueues a MirrorSyncJob for every mirror repo that is due for a

5 +# refresh (see MirrorSyncSchedulerJob). Webhooks, when configured, sync sooner.

6 +

7 +production:

8 + mirror_sync:

9 + class: MirrorSyncSchedulerJob

10 + schedule: every 15 minutes

11 +

12 +development:

13 + mirror_sync:

14 + class: MirrorSyncSchedulerJob

15 + schedule: every 15 minutes

config/routes.rb

+23

@@ -67,6 +67,25 @@ Rails.application.routes.draw do

67 get "/new", to: "repositories#new", as: :new_repository

68 post "/new", to: "repositories#create"

70 + # Import from GitHub (mirror + migrate). Declared before the "/:username"

71 + # matcher so "/import" isn't read as a profile.

72 + #

73 + # The GitHub OAuth connect routes are declared before the resources block so

74 + # "/import/github/*" isn't swallowed by the "/import/:id" show route. This is

75 + # our own GitHub OAuth app (separate from smbCloud sign-in); the token it mints

76 + # can read/clone the user's GitHub repos.

77 + get "/import/github/connect", to: "github_connections#connect", as: :github_import_connect

78 + get "/import/github/callback", to: "github_connections#callback", as: :github_import_callback

79 + delete "/import/github/disconnect", to: "github_connections#disconnect", as: :github_import_disconnect

80 +

81 + resources :imports, path: "import", only: %i[index new create show],

82 + constraints: { id: /\d+/ } do

83 + post :retry, on: :member

84 + end

85 +

86 + # GitHub push webhook → immediate mirror sync (HMAC-authenticated).

87 + post "/webhooks/github", to: "github_webhooks#create"

88 +

89 # Settings

90 get "/settings", to: "users#settings", as: :settings

91 patch "/settings", to: "users#update_settings"

@@ -141,6 +160,10 @@ Rails.application.routes.draw do

160 get "/:username/:repository/tree/:branch", to: "repositories#tree", as: :repository_branch

161 post "/:username/:repository/star", to: "stars#create", as: :repository_star,

162 constraints: { repository: /[^\/.][^\/]*/ }, format: false

163 + post "/:username/:repository/mirror/detach", to: "mirrors#detach", as: :repository_mirror_detach,

164 + constraints: { repository: /[^\/.][^\/]*/ }, format: false

165 + post "/:username/:repository/mirror/sync", to: "mirrors#sync", as: :repository_mirror_sync,

166 + constraints: { repository: /[^\/.][^\/]*/ }, format: false

167 # `format: false` keeps dotted names (e.g. "Qwen2.5-3B-Instruct-GGUF") intact

168 # rather than treating the dot as a response-format separator.

169 get "/:username/:repository", to: "repositories#show", as: :repository,

db/migrate/20250101000011_create_github_connections.rb new

+25

@@ -0,0 +1,25 @@

1 +# frozen_string_literal: true

2 +

3 +# Per-user GitHub OAuth connection used by the "Import from GitHub" flow.

4 +#

5 +# This is deliberately separate from the smbCloud-brokered "Continue with

6 +# GitHub" sign-in: that gives us a smbCloud session, not a GitHub API token.

7 +# Importing needs a real GitHub token (scoped `repo` or `public_repo`) to list

8 +# private repos and clone them, so we run our own minimal OAuth app and store

9 +# the resulting token here — encrypted at rest, one row per user, revocable.

10 +class CreateGithubConnections < ActiveRecord::Migration[8.1]

11 + def change

12 + create_table :github_connections do |t|

13 + t.references :user, null: false, foreign_key: true, index: { unique: true }

14 + t.string :github_login

15 + # Encrypted at rest via ActiveRecord::Encryption (`encrypts :access_token`).

16 + # text, not string: encrypted + base64 ciphertext is longer than the token.

17 + t.text :access_token, null: false

18 + t.string :scope

19 + t.string :token_type, default: "bearer", null: false

20 + t.datetime :connected_at

21 +

22 + t.timestamps

23 + end

24 + end

25 +end

db/migrate/20250101000012_create_repository_imports.rb new

+41

@@ -0,0 +1,41 @@

1 +# frozen_string_literal: true

2 +

3 +# Tracks one "import a repo into siGit" operation end to end so the UI can show

4 +# live progress and a failed import can be retried. The heavy lifting (clone,

5 +# push, LFS) runs off the request thread in RepositoryImportJob; this row is the

6 +# durable state machine it drives.

7 +class CreateRepositoryImports < ActiveRecord::Migration[8.1]

8 + def change

9 + create_table :repository_imports do |t|

10 + t.references :user, null: false, foreign_key: true

11 + # Set once the target repo exists on disk; nil while queued/cloning and

12 + # after a cleaned-up failure. Nullified if the repo is later destroyed.

13 + t.references :repository, null: true, foreign_key: { on_delete: :nullify }

14 +

15 + t.string :source_url, null: false # normalized https git URL

16 + t.string :source_host # e.g. "github.com"

17 + t.string :mode, null: false, default: "migrate" # migrate | mirror

18 + t.string :status, null: false, default: "queued" # see RepositoryImport::STATUSES

19 + t.text :error_message

20 +

21 + # Target repo attributes captured up front so a retry is deterministic and

22 + # doesn't depend on re-reading GitHub.

23 + t.string :target_name, null: false

24 + t.boolean :target_private, null: false, default: false

25 + t.string :default_branch

26 + t.text :description

27 +

28 + # Best-effort LFS accounting surfaced in the UI.

29 + t.boolean :lfs_detected, null: false, default: false

30 + t.boolean :lfs_skipped, null: false, default: false

31 +

32 + t.bigint :source_size_kb # reported by GitHub, for size caps

33 + t.datetime :started_at

34 + t.datetime :finished_at

35 +

36 + t.timestamps

37 + end

38 +

39 + add_index :repository_imports, [ :user_id, :status ]

40 + end

41 +end

db/migrate/20250101000013_add_mirror_to_repositories.rb new

+25

@@ -0,0 +1,25 @@

1 +# frozen_string_literal: true

2 +

3 +# Mirror mode: a repository that keeps an upstream (e.g. GitHub) as the source

4 +# of truth and auto-syncs from it. Mirrors are read-only on siGit (pushes are

5 +# rejected) to avoid divergence, until the owner "detaches" them into a normal

6 +# writable repo.

7 +class AddMirrorToRepositories < ActiveRecord::Migration[8.1]

8 + def change

9 + change_table :repositories, bulk: true do |t|

10 + t.boolean :mirror, null: false, default: false

11 + t.string :upstream_url # the remote we fetch from

12 + # Encrypted upstream credential for private mirrors (ActiveRecord::

13 + # Encryption). Nil for public upstreams. Never returned to the client.

14 + t.text :upstream_token

15 + t.string :mirror_status # ok | syncing | failed

16 + t.datetime :mirror_synced_at

17 + t.text :mirror_error

18 + end

19 +

20 + add_index :repositories, :mirror

21 + # Due mirrors are polled by the recurring MirrorSyncScheduler; index the

22 + # columns it filters/orders on.

23 + add_index :repositories, [ :mirror, :mirror_synced_at ]

24 + end

25 +end

db/schema.rb

+48 -1

@@ -10,7 +10,7 @@

10 #

11 # It's strongly recommended that you check this file into your version control system.

13 -ActiveRecord::Schema[8.1].define(version: 2025_01_01_000010) do

13 +ActiveRecord::Schema[8.1].define(version: 2025_01_01_000013) do

14 # These are extensions that must be enabled in order to support this database

15 enable_extension "pg_catalog.plpgsql"

@@ -56,6 +56,18 @@ ActiveRecord::Schema[8.1].define(version: 2025_01_01_000010) do

56 t.index ["user_id"], name: "index_comments_on_user_id"

57 end

59 + create_table "github_connections", force: :cascade do |t|

60 + t.text "access_token", null: false

61 + t.datetime "connected_at"

62 + t.datetime "created_at", null: false

63 + t.string "github_login"

64 + t.string "scope"

65 + t.string "token_type", default: "bearer", null: false

66 + t.datetime "updated_at", null: false

67 + t.bigint "user_id", null: false

68 + t.index ["user_id"], name: "index_github_connections_on_user_id", unique: true

69 + end

70 +

71 create_table "issues", force: :cascade do |t|

72 t.text "body"

73 t.datetime "closed_at"

@@ -96,15 +108,47 @@ ActiveRecord::Schema[8.1].define(version: 2025_01_01_000010) do

108 t.integer "downloads_count", default: 0, null: false

109 t.boolean "is_private", default: false, null: false

110 t.string "kind", default: "code", null: false

111 + t.boolean "mirror", default: false, null: false

112 + t.text "mirror_error"

113 + t.string "mirror_status"

114 + t.datetime "mirror_synced_at"

115 t.string "name", null: false

116 t.integer "stars_count", default: 0, null: false

117 t.datetime "updated_at", null: false

118 + t.text "upstream_token"

119 + t.string "upstream_url"

120 t.bigint "user_id", null: false

121 t.index ["kind"], name: "index_repositories_on_kind"

122 + t.index ["mirror", "mirror_synced_at"], name: "index_repositories_on_mirror_and_mirror_synced_at"

123 + t.index ["mirror"], name: "index_repositories_on_mirror"

124 t.index ["user_id", "name"], name: "index_repositories_on_user_id_and_name", unique: true

125 t.index ["user_id"], name: "index_repositories_on_user_id"

126 end

127

128 + create_table "repository_imports", force: :cascade do |t|

129 + t.datetime "created_at", null: false

130 + t.string "default_branch"

131 + t.text "description"

132 + t.text "error_message"

133 + t.datetime "finished_at"

134 + t.boolean "lfs_detected", default: false, null: false

135 + t.boolean "lfs_skipped", default: false, null: false

136 + t.string "mode", default: "migrate", null: false

137 + t.bigint "repository_id"

138 + t.string "source_host"

139 + t.bigint "source_size_kb"

140 + t.string "source_url", null: false

141 + t.datetime "started_at"

142 + t.string "status", default: "queued", null: false

143 + t.string "target_name", null: false

144 + t.boolean "target_private", default: false, null: false

145 + t.datetime "updated_at", null: false

146 + t.bigint "user_id", null: false

147 + t.index ["repository_id"], name: "index_repository_imports_on_repository_id"

148 + t.index ["user_id", "status"], name: "index_repository_imports_on_user_id_and_status"

149 + t.index ["user_id"], name: "index_repository_imports_on_user_id"

150 + end

151 +

152 create_table "ssh_keys", force: :cascade do |t|

153 t.datetime "created_at", null: false

154 t.string "fingerprint", null: false

@@ -157,11 +201,14 @@ ActiveRecord::Schema[8.1].define(version: 2025_01_01_000010) do

201 add_foreign_key "cloud_sessions", "users"

202 add_foreign_key "cloud_usages", "users"

203 add_foreign_key "comments", "users"

204 + add_foreign_key "github_connections", "users"

205 add_foreign_key "issues", "repositories"

206 add_foreign_key "issues", "users"

207 add_foreign_key "pull_requests", "repositories"

208 add_foreign_key "pull_requests", "users"

209 add_foreign_key "repositories", "users"

210 + add_foreign_key "repository_imports", "repositories", on_delete: :nullify

211 + add_foreign_key "repository_imports", "users"

212 add_foreign_key "ssh_keys", "users"

213 add_foreign_key "stars", "repositories"

214 add_foreign_key "stars", "users"

docs/import-from-github.md new

+123

@@ -0,0 +1,123 @@

1 +# Import from GitHub (mirror + migrate)

2 +

3 +One-click import of a Git repository into siGit, in two modes:

4 +

5 +- **Migrate** — copy the repo once; siGit becomes the source of truth and you can

6 + push to it.

7 +- **Mirror** — keep GitHub (or any upstream) as the source of truth; siGit

8 + auto-syncs on a schedule (and on webhook, if configured) and stays **read-only**

9 + until you *detach* it.

10 +

11 +There is also **import by URL** for any public `http(s)` git URL, with no OAuth.

12 +

13 +## How it fits the existing app

14 +

15 +- Repos are still bare repos on disk under `SIGITSI_REPOS_PATH/<user>/<name>.git`,

16 + created via `GitRepositoryService` — the importer reuses that path, it does not

17 + invent new storage.

18 +- The long-running clone/push runs off the request thread in

19 + `RepositoryImportJob` (Solid Queue). Mirror syncs run in `MirrorSyncJob`,

20 + swept by the recurring `MirrorSyncSchedulerJob` (`config/recurring.yml`).

21 +- GitHub API access uses a **dedicated GitHub OAuth app** (env

22 + `GITHUB_IMPORT_CLIENT_ID` / `GITHUB_IMPORT_CLIENT_SECRET`), separate from the

23 + smbCloud "Continue with GitHub" *sign-in* (which yields no GitHub API token).

24 + The token is stored encrypted (`GithubConnection#access_token`, Active Record

25 + Encryption) and never returned to the client.

26 +

27 +## Key files

28 +

29 +| Area | File |

30 +|------|------|

31 +| SSRF URL guard | `app/services/import_url_validator.rb` |

32 +| GitHub OAuth | `app/services/github_oauth_service.rb`, `app/controllers/github_connections_controller.rb` |

33 +| GitHub REST | `app/services/github_api_client.rb` |

34 +| Clone + push | `app/services/repository_import_service.rb` |

35 +| Import orchestration | `app/jobs/repository_import_job.rb`, `app/controllers/imports_controller.rb` |

36 +| Mirror sync | `app/services/mirror_sync_service.rb`, `app/jobs/mirror_sync_job.rb`, `app/jobs/mirror_sync_scheduler_job.rb` |

37 +| Mirror read-only | `app/controllers/git_http_controller.rb` (`reject_mirror_push!`) |

38 +| Detach / manual sync | `app/controllers/mirrors_controller.rb` |

39 +| Webhook | `app/controllers/github_webhooks_controller.rb` |

40 +

41 +## Guardrails

42 +

43 +- **SSRF:** import-by-URL allows only `http(s)`; loopback / RFC1918 / link-local

44 + (incl. `169.254.169.254`) / IPv6 ULA + mapped-v4 are rejected, for literal IPs

45 + and every resolved address. Mirrors re-validate the upstream on every sync

46 + (DNS-rebinding window).

47 +- **Isolation:** clones run in a temp dir that's always removed; every git call

48 + has a hard timeout and its process group is killed on overrun;

49 + `GIT_TERMINAL_PROMPT=0` makes auth failures fail fast.

50 +- **Size/abuse:** per-repo size cap, per-user in-flight volume cap, and a

51 + concurrent-import cap (`IMPORT_MAX_*` env). Oversized repos are skipped with a

52 + message.

53 +- **Secrets:** OAuth + upstream tokens are encrypted at rest, scrubbed from

54 + captured git output, never logged, never sent to the client, dropped on

55 + disconnect/detach.

56 +- **Idempotency:** a failed import leaves no half-created repo (rollback) and is

57 + retryable; retries clear any partial repo; name collisions with a *different*

58 + existing repo fail explicitly; bulk imports auto-suffix colliding names.

59 +- **Rate limits:** `GithubApiClient` raises `RateLimited` with a reset time so

60 + callers back off; the repo list is cached briefly and never blocks the page.

61 +

62 +Not imported in v1 (surfaced in the UI, not silently dropped): issues, pull

63 +requests, wikis, releases, Actions.

64 +

65 +## Configuration

66 +

67 +See `.env.example` (the "Import from GitHub" and "Active Record Encryption"

68 +sections). Minimum for full functionality:

69 +

70 +```

71 +GITHUB_IMPORT_CLIENT_ID=...

72 +GITHUB_IMPORT_CLIENT_SECRET=...

73 +# production only:

74 +ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=...

75 +ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=...

76 +ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=... # bin/rails db:encryption:init

77 +# optional: GITHUB_WEBHOOK_SECRET, IMPORT_MAX_*, MIRROR_SYNC_*

78 +```

79 +

80 +Register the GitHub OAuth app with callback `<SIGITSI_URL>/import/github/callback`.

81 +

82 +## Verify locally

83 +

84 +```sh

85 +bin/rails db:prepare

86 +bin/rails tailwindcss:build # so views render in specs

87 +

88 +# Full suite (all green):

89 +bundle exec rspec

90 +

91 +# Just this feature:

92 +bundle exec rspec \

93 + spec/services/import_url_validator_spec.rb \

94 + spec/models/github_connection_spec.rb \

95 + spec/services/repository_import_service_spec.rb \

96 + spec/jobs/repository_import_job_spec.rb \

97 + spec/requests/github_connections_spec.rb \

98 + spec/requests/git_http_mirror_spec.rb

99 +```

100 +

101 +Manual smoke test (no GitHub app needed — uses import-by-URL against a local

102 +fixture, so nothing hits the network):

103 +

104 +```sh

105 +# 1. Build a tiny bare fixture repo with a branch + tag.

106 +tmp=$(mktemp -d); git init -q -b main "$tmp/w"

107 +git -C "$tmp/w" -c user.email=a@b.c -c user.name=t commit -q --allow-empty -m init

108 +git -C "$tmp/w" tag v1

109 +git clone -q --bare "$tmp/w" "$tmp/src.git"

110 +

111 +# 2. In `bin/rails console`, queue and run an import synchronously:

112 +# u = User.first

113 +# imp = u.repository_imports.create!(source_url: "<tmp>/src.git", mode: "migrate",

114 +# target_name: "fixture", status: "queued")

115 +# RepositoryImportJob.perform_now(imp.id)

116 +# imp.reload.status # => "done"

117 +# imp.repository.default_branch # => "main"; branches/tags preserved

118 +```

119 +

120 +For mirror mode, set `mode: "mirror"`; the resulting repo is read-only

121 +(`repo.writable_by?(u) == false`) and a `git push` to it is rejected with a

122 +message. Use the repo page's **Detach** to convert it to a normal writable repo

123 +and stop syncing.

spec/jobs/repository_import_job_spec.rb new

+147

@@ -0,0 +1,147 @@

1 +# frozen_string_literal: true

2 +

3 +require "rails_helper"

4 +require "tmpdir"

5 +require "fileutils"

6 +

7 +RSpec.describe RepositoryImportJob, type: :job do

8 + let(:user) { User.create!(smbcloud_id: 7, email: "imp@example.com", username: "importer") }

9 +

10 + around do |example|

11 + Dir.mktmpdir("import-job-spec") do |tmp|

12 + @tmp = tmp

13 + @repos = File.join(tmp, "repos")

14 + @source = build_source(File.join(tmp, "src"))

15 + example.run

16 + end

17 + end

18 +

19 + before do

20 + # Point repo storage at the sandbox for both the service and the job.

21 + allow(GitRepositoryService).to receive(:repo_path) do |u, r|

22 + File.join(@repos, u, "#{r}.git")

23 + end

24 + end

25 +

26 + def build_source(dir)

27 + work = File.join(dir, "work")

28 + FileUtils.mkdir_p(work)

29 + system("git", "init", "-q", "-b", "main", work, exception: true)

30 + system("git", "-C", work, "config", "user.email", "t@e.st", exception: true)

31 + system("git", "-C", work, "config", "user.name", "T", exception: true)

32 + File.write(File.join(work, "README.md"), "# Imported\n\nHello.\n")

33 + system("git", "-C", work, "add", ".", exception: true)

34 + system("git", "-C", work, "commit", "-qm", "init", exception: true)

35 + system("git", "-C", work, "tag", "v1", exception: true)

36 + system("git", "-C", work, "checkout", "-q", "-b", "dev", exception: true)

37 + File.write(File.join(work, "d.txt"), "d\n")

38 + system("git", "-C", work, "add", ".", exception: true)

39 + system("git", "-C", work, "commit", "-qm", "dev", exception: true)

40 + system("git", "-C", work, "checkout", "-q", "main", exception: true)

41 + bare = File.join(dir, "source.git")

42 + system("git", "clone", "-q", "--bare", work, bare, exception: true)

43 + bare

44 + end

45 +

46 + def new_import(attrs = {})

47 + user.repository_imports.create!({

48 + source_url: @source, source_host: nil, mode: "migrate",

49 + target_name: "imported", target_private: false, default_branch: "main",

50 + status: "queued"

51 + }.merge(attrs))

52 + end

53 +

54 + it "migrates a repo with all branches, tags, history and a working default branch" do

55 + import = new_import

56 +

57 + described_class.perform_now(import.id)

58 + import.reload

59 +

60 + expect(import.status).to eq("done")

61 + repo = import.repository

62 + expect(repo).to be_present

63 + expect(repo).to be_initialized

64 +

65 + branches = GitRepositoryService.branches(repo.disk_path).sort

66 + expect(branches).to eq(%w[dev main])

67 + expect(`git --git-dir #{repo.disk_path} tag`.split).to eq(%w[v1])

68 + expect(GitRepositoryService.default_branch(repo.disk_path)).to eq("main")

69 + # README is reachable on the default branch → landing page will render it.

70 + expect(GitRepositoryService.readme_content(repo.disk_path, "main")).to be_present

71 + end

72 +

73 + it "sets up a read-only mirror in mirror mode" do

74 + import = new_import(mode: "mirror")

75 +

76 + described_class.perform_now(import.id)

77 + import.reload

78 + repo = import.repository

79 +

80 + expect(import.status).to eq("done")

81 + expect(repo.mirror?).to be(true)

82 + expect(repo.upstream_url).to eq(@source)

83 + expect(repo.mirror_status).to eq("ok")

84 + expect(repo.mirror_synced_at).to be_present

85 + expect(repo.writable_by?(user)).to be(false)

86 + end

87 +

88 + it "leaves no half-created repo behind when the import fails, and stays retryable" do

89 + import = new_import

90 + allow(RepositoryImportService).to receive(:clone_and_push)

91 + .and_raise(RepositoryImportService::ImportError, "boom")

92 +

93 + described_class.perform_now(import.id)

94 + import.reload

95 +

96 + expect(import.status).to eq("failed")

97 + expect(import.error_message).to eq("boom")

98 + expect(import.repository).to be_nil

99 + expect(user.repositories.count).to eq(0)

100 + expect(Dir.exist?(File.join(@repos, user.username, "imported.git"))).to be(false)

101 + expect(import.retryable?).to be(true)

102 + end

103 +

104 + it "rejects a name collision with a different existing repo, without touching it" do

105 + existing = user.repositories.create!(

106 + name: "imported", disk_path: GitRepositoryService.repo_path(user.username, "imported"),

107 + default_branch: "main", description: "the original"

108 + )

109 + import = new_import

110 +

111 + described_class.perform_now(import.id)

112 + import.reload

113 +

114 + expect(import.status).to eq("failed")

115 + expect(import.error_message).to match(/already exists/)

116 + expect(existing.reload.description).to eq("the original")

117 + expect(user.repositories.count).to eq(1)

118 + end

119 +

120 + it "is idempotent against double delivery (already-done import is a no-op)" do

121 + import = new_import

122 + described_class.perform_now(import.id)

123 + described_class.perform_now(import.id) # second delivery

124 +

125 + expect(user.repositories.where(name: "imported").count).to eq(1)

126 + expect(import.reload.status).to eq("done")

127 + end

128 +

129 + it "clears a partial repo from a previous attempt when retrying" do

130 + # Simulate a prior failed attempt that left a repo attached.

131 + partial = user.repositories.create!(

132 + name: "imported", disk_path: GitRepositoryService.repo_path(user.username, "imported"),

133 + default_branch: "main"

134 + )

135 + FileUtils.mkdir_p(partial.disk_path)

136 + import = new_import(status: "failed", repository: partial)

137 +

138 + described_class.perform_now(import.id)

139 + import.reload

140 +

141 + expect(import.status).to eq("done")

142 + expect(Repository.where(id: partial.id)).to be_empty # old partial gone

143 + expect(import.repository).to be_present

144 + expect(import.repository).to be_initialized # fresh, clean import

145 + expect(user.repositories.where(name: "imported").count).to eq(1)

146 + end

147 +end

spec/models/github_connection_spec.rb new

+39

@@ -0,0 +1,39 @@

1 +# frozen_string_literal: true

2 +

3 +require "rails_helper"

4 +

5 +RSpec.describe GithubConnection, type: :model do

6 + let(:user) { User.create!(smbcloud_id: 42, email: "gh@example.com", username: "ghuser") }

7 +

8 + it "stores the access token encrypted at rest, not in plaintext" do

9 + conn = user.create_github_connection!(access_token: "gho_secrettoken123", scope: "repo")

10 +

11 + # The model round-trips the real value...

12 + expect(conn.reload.access_token).to eq("gho_secrettoken123")

13 +

14 + # ...but the raw column holds ciphertext, never the plaintext token.

15 + raw = ActiveRecord::Base.connection.select_value(

16 + "SELECT access_token FROM github_connections WHERE id = #{conn.id}"

17 + )

18 + expect(raw).not_to include("gho_secrettoken123")

19 + expect(raw).to be_present

20 + end

21 +

22 + it "requires an access token" do

23 + conn = user.build_github_connection(access_token: nil)

24 + expect(conn).not_to be_valid

25 + expect(conn.errors[:access_token]).to be_present

26 + end

27 +

28 + it "is unique per user" do

29 + user.create_github_connection!(access_token: "a")

30 + dup = GithubConnection.new(user_id: user.id, access_token: "b")

31 + expect(dup).not_to be_valid

32 + expect(dup.errors[:user_id]).to be_present

33 + end

34 +

35 + it "detects private scope" do

36 + expect(user.build_github_connection(access_token: "a", scope: "repo").private_scope?).to be(true)

37 + expect(user.build_github_connection(access_token: "a", scope: "public_repo").private_scope?).to be(false)

38 + end

39 +end

spec/requests/git_http_mirror_spec.rb new

+61

@@ -0,0 +1,61 @@

1 +# frozen_string_literal: true

2 +

3 +require "rails_helper"

4 +require "tmpdir"

5 +require "fileutils"

6 +

7 +# Mirror repos are read-only over git: a push (receive-pack) must be rejected

8 +# with a message the user actually sees, while clone/fetch and normal repos are

9 +# unaffected.

10 +RSpec.describe "Git HTTP mirror read-only enforcement", type: :request do

11 + let(:user) { User.create!(smbcloud_id: 55, email: "m@example.com", username: "mirroruser") }

12 +

13 + around do |example|

14 + Dir.mktmpdir("git-http-mirror") do |tmp|

15 + @bare = File.join(tmp, "mirrored.git")

16 + system("git", "init", "--bare", "-q", @bare, exception: true)

17 + example.run

18 + end

19 + end

20 +

21 + let!(:mirror) do

22 + user.repositories.create!(

23 + name: "mirrored", disk_path: @bare, default_branch: "main",

24 + mirror: true, upstream_url: "https://github.com/owner/mirrored.git", mirror_status: "ok"

25 + )

26 + end

27 +

28 + it "rejects a push to a mirror with a clear git error message" do

29 + get "/mirroruser/mirrored.git/info/refs", params: { service: "git-receive-pack" }

30 +

31 + expect(response).to have_http_status(:ok)

32 + expect(response.body).to include("ERR")

33 + expect(response.body).to include("read-only mirror")

34 + expect(response.body).to include("Detach")

35 + end

36 +

37 + it "also blocks the receive-pack RPC endpoint directly" do

38 + post "/mirroruser/mirrored.git/git-receive-pack"

39 + # The advertisement carries the ERR packet rather than proxying to git.

40 + expect(response.body).to include("read-only mirror")

41 + end

42 +

43 + it "still allows fetch/clone (upload-pack) from a public mirror" do

44 + get "/mirroruser/mirrored.git/info/refs", params: { service: "git-upload-pack" }

45 + expect(response).to have_http_status(:ok)

46 + expect(response.body).not_to include("read-only mirror")

47 + end

48 +

49 + it "does not block pushes on a normal (non-mirror) repo" do

50 + normal_dir = File.join(Dir.tmpdir, "normal-#{SecureRandom.hex(4)}.git")

51 + system("git", "init", "--bare", "-q", normal_dir, exception: true)

52 + user.repositories.create!(name: "normal", disk_path: normal_dir, default_branch: "main")

53 +

54 + get "/mirroruser/normal.git/info/refs", params: { service: "git-receive-pack" }

55 + # No mirror rejection; instead it falls through to auth (401 challenge).

56 + expect(response.body).not_to include("read-only mirror")

57 + expect(response).to have_http_status(:unauthorized)

58 + ensure

59 + FileUtils.rm_rf(normal_dir) if normal_dir

60 + end

61 +end

spec/requests/github_connections_spec.rb new

+72

@@ -0,0 +1,72 @@

1 +# frozen_string_literal: true

2 +

3 +require "rails_helper"

4 +

5 +# The GitHub OAuth connect flow for imports: the callback must verify state,

6 +# exchange the code, store the token encrypted, and never leak it to the client.

7 +RSpec.describe "GitHub import connection", type: :request do

8 + let(:user) { User.create!(smbcloud_id: 99, email: "conn@example.com", username: "connuser") }

9 +

10 + before do

11 + # Sign the user in (session-based auth) without going through smbCloud.

12 + allow_any_instance_of(ApplicationController).to receive(:current_user).and_return(user)

13 + allow(GithubOauthService).to receive(:configured?).and_return(true)

14 + allow(GithubOauthService).to receive(:generate_state).and_return("state-xyz")

15 + allow(GithubOauthService).to receive(:authorize_url).and_return("https://github.com/login/oauth/authorize?x=1")

16 + end

17 +

18 + # Establishes the CSRF state in the session, the way the real flow does.

19 + def start_connect

20 + get github_import_connect_path

21 + expect(response).to redirect_to("https://github.com/login/oauth/authorize?x=1")

22 + end

23 +

24 + it "exchanges the code and stores the token encrypted on callback" do

25 + start_connect

26 + allow(GithubOauthService).to receive(:exchange_code)

27 + .and_return(access_token: "gho_thetoken", scope: "repo", token_type: "bearer")

28 + fake_client = instance_double(GithubApiClient, login: "octocat", list_repositories: [])

29 + allow(GithubApiClient).to receive(:new).with("gho_thetoken").and_return(fake_client)

30 +

31 + get github_import_callback_path, params: { code: "abc", state: "state-xyz" }

32 +

33 + expect(response).to redirect_to(new_import_path)

34 + conn = user.reload.github_connection

35 + expect(conn).to be_present

36 + expect(conn.access_token).to eq("gho_thetoken")

37 + expect(conn.github_login).to eq("octocat")

38 +

39 + # Ciphertext at rest, not the plaintext token.

40 + raw = ActiveRecord::Base.connection.select_value(

41 + "SELECT access_token FROM github_connections WHERE id = #{conn.id}"

42 + )

43 + expect(raw).not_to include("gho_thetoken")

44 +

45 + # The token is never echoed back to the client.

46 + expect(response.body).not_to include("gho_thetoken")

47 + follow_redirect!

48 + expect(response.body).not_to include("gho_thetoken")

49 + end

50 +

51 + it "rejects a callback whose state doesn't match (CSRF)" do

52 + start_connect

53 + expect(GithubOauthService).not_to receive(:exchange_code)

54 +

55 + get github_import_callback_path, params: { code: "abc", state: "wrong-state" }

56 +

57 + expect(response).to redirect_to(new_import_path)

58 + expect(user.reload.github_connection).to be_nil

59 + end

60 +

61 + it "handles a user cancelling the GitHub authorization" do

62 + get github_import_callback_path, params: { error: "access_denied" }

63 + expect(response).to redirect_to(new_import_path)

64 + expect(user.reload.github_connection).to be_nil

65 + end

66 +

67 + it "disconnects and removes the stored token" do

68 + user.create_github_connection!(access_token: "gho_x", scope: "repo")

69 + delete github_import_disconnect_path

70 + expect(user.reload.github_connection).to be_nil

71 + end

72 +end

spec/services/import_url_validator_spec.rb new

+80

@@ -0,0 +1,80 @@

1 +# frozen_string_literal: true

2 +

3 +require "rails_helper"

4 +

5 +# The SSRF guard for import-by-URL. These are the checks that stop a user from

6 +# pointing the importer at cloud metadata or an internal service.

7 +RSpec.describe ImportUrlValidator do

8 + describe ".validate!" do

9 + it "accepts a public https git URL" do

10 + expect(described_class.validate!("https://192.0.2.10/owner/repo.git"))

11 + .to eq("https://192.0.2.10/owner/repo.git")

12 + end

13 +

14 + it "rejects non-http(s) schemes" do

15 + %w[

16 + ssh://git@github.com/x.git

17 + git://github.com/x.git

18 + file:///etc/passwd

19 + ftp://example.com/x.git

20 + ].each do |url|

21 + expect { described_class.validate!(url) }

22 + .to raise_error(ImportUrlValidator::InvalidUrl), "expected #{url} to be rejected"

23 + end

24 + end

25 +

26 + it "rejects the cloud metadata address" do

27 + expect { described_class.validate!("http://169.254.169.254/latest/meta-data/") }

28 + .to raise_error(ImportUrlValidator::InvalidUrl, /private or internal/)

29 + end

30 +

31 + it "rejects loopback and RFC1918 literal IPs" do

32 + %w[

33 + http://127.0.0.1/x.git

34 + http://10.0.0.5/x.git

35 + http://192.168.1.1/x.git

36 + http://172.16.9.9/x.git

37 + ].each do |url|

38 + expect { described_class.validate!(url) }

39 + .to raise_error(ImportUrlValidator::InvalidUrl), "expected #{url} to be rejected"

40 + end

41 + end

42 +

43 + it "rejects IPv6 loopback and IPv4-mapped private addresses" do

44 + expect { described_class.validate!("http://[::1]/x.git") }

45 + .to raise_error(ImportUrlValidator::InvalidUrl)

46 + expect { described_class.validate!("http://[::ffff:10.0.0.1]/x.git") }

47 + .to raise_error(ImportUrlValidator::InvalidUrl)

48 + end

49 +

50 + it "rejects credentials embedded in the URL" do

51 + expect { described_class.validate!("https://user:pass@github.com/x.git") }

52 + .to raise_error(ImportUrlValidator::InvalidUrl, /credentials/)

53 + end

54 +

55 + it "rejects a host that resolves to a blocked address" do

56 + allow(described_class).to receive(:resolve).and_return([ "127.0.0.1" ])

57 + expect { described_class.validate!("https://internal.example.com/x.git") }

58 + .to raise_error(ImportUrlValidator::InvalidUrl, /private or internal/)

59 + end

60 +

61 + it "accepts a host that resolves to a public address" do

62 + allow(described_class).to receive(:resolve).and_return([ "140.82.112.3" ])

63 + expect(described_class.validate!("https://github.com/rails/rails.git"))

64 + .to eq("https://github.com/rails/rails.git")

65 + end

66 +

67 + it "rejects blank input" do

68 + expect { described_class.validate!("") }.to raise_error(ImportUrlValidator::InvalidUrl)

69 + expect { described_class.validate!(nil) }.to raise_error(ImportUrlValidator::InvalidUrl)

70 + end

71 + end

72 +

73 + describe ".safe?" do

74 + it "is true/false without raising" do

75 + allow(described_class).to receive(:resolve).and_return([ "140.82.112.3" ])

76 + expect(described_class.safe?("https://github.com/x.git")).to be(true)

77 + expect(described_class.safe?("http://127.0.0.1/x.git")).to be(false)

78 + end

79 + end

80 +end

spec/services/repository_import_service_spec.rb new

+95

@@ -0,0 +1,95 @@

1 +# frozen_string_literal: true

2 +

3 +require "rails_helper"

4 +require "tmpdir"

5 +require "fileutils"

6 +require "open3"

7 +

8 +# Exercises the git mechanics against a small local fixture repo (no network):

9 +# a mirror clone + push must carry every branch, every tag, and the full

10 +# history, and the guardrails (size cap, timeout) must fire.

11 +RSpec.describe RepositoryImportService do

12 + # Builds a bare source repo with two branches (main, feature) and a tag (v1),

13 + # returning its path. Yields nothing; caller cleans up the tmpdir.

14 + def build_source(dir)

15 + work = File.join(dir, "work")

16 + FileUtils.mkdir_p(work)

17 + run = ->(*a) { system(*a, exception: true) }

18 + run.call("git", "init", "-q", "-b", "main", work)

19 + run.call("git", "-C", work, "config", "user.email", "t@e.st")

20 + run.call("git", "-C", work, "config", "user.name", "Test")

21 + File.write(File.join(work, "README.md"), "# Fixture\n")

22 + run.call("git", "-C", work, "add", ".")

23 + run.call("git", "-C", work, "commit", "-qm", "initial")

24 + run.call("git", "-C", work, "tag", "v1")

25 + run.call("git", "-C", work, "checkout", "-q", "-b", "feature")

26 + File.write(File.join(work, "f.txt"), "x\n")

27 + run.call("git", "-C", work, "add", ".")

28 + run.call("git", "-C", work, "commit", "-qm", "feature work")

29 + run.call("git", "-C", work, "checkout", "-q", "main")

30 +

31 + bare = File.join(dir, "source.git")

32 + run.call("git", "clone", "-q", "--bare", work, bare)

33 + bare

34 + end

35 +

36 + def empty_bare(dir)

37 + dest = File.join(dir, "dest.git")

38 + system("git", "init", "--bare", "-q", dest, exception: true)

39 + dest

40 + end

41 +

42 + around do |example|

43 + Dir.mktmpdir("import-svc-spec") do |tmp|

44 + @tmp = tmp

45 + example.run

46 + end

47 + end

48 +

49 + it "imports every branch, tag, and the full history, reporting progress" do

50 + source = build_source(@tmp)

51 + dest = empty_bare(@tmp)

52 + phases = []

53 +

54 + result = described_class.clone_and_push(

55 + source_url: source, dest_path: dest,

56 + progress: ->(p) { phases << p }

57 + )

58 +

59 + branches = GitRepositoryService.branches(dest).sort

60 + tags = Open3.capture3("git", "--git-dir", dest, "tag").first.split

61 +

62 + expect(branches).to eq(%w[feature main])

63 + expect(tags).to eq(%w[v1])

64 + expect(GitRepositoryService.commit_count(dest, "main")).to eq(1)

65 + expect(GitRepositoryService.commit_count(dest, "feature")).to eq(2)

66 + expect(phases).to eq(%i[cloning pushing fetching_lfs])

67 + expect(result.lfs_detected).to be(false)

68 + expect(result.size_kb).to be_a(Integer)

69 + end

70 +

71 + it "rejects a repo larger than the size cap" do

72 + source = build_source(@tmp)

73 + dest = empty_bare(@tmp)

74 +

75 + expect {

76 + described_class.clone_and_push(source_url: source, dest_path: dest, max_size_kb: 0)

77 + }.to raise_error(RepositoryImportService::SizeExceeded)

78 + end

79 +

80 + it "fails cleanly on an unreachable source instead of hanging" do

81 + dest = empty_bare(@tmp)

82 + expect {

83 + described_class.clone_and_push(source_url: "https://192.0.2.1/nope.git",

84 + dest_path: dest, timeout: 8)

85 + }.to raise_error(RepositoryImportService::ImportError)

86 + end

87 +

88 + it "terminates a git call that overruns its timeout" do

89 + source = build_source(@tmp)

90 + dest = empty_bare(@tmp)

91 + expect {

92 + described_class.clone_and_push(source_url: source, dest_path: dest, timeout: 0.001)

93 + }.to raise_error(RepositoryImportService::CloneTimeout)

94 + end

95 +end