Skip to content

Migrating GitLab by using direct transfer

DETAILS: Tier: Free, Premium, Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated

  • Enabled on GitLab.com in GitLab 15.6.
  • New application setting bulk_import_enabled introduced in GitLab 15.8. bulk_import feature flag removed.
  • bulk_import_projects feature flag removed in GitLab 15.10.

You can migrate GitLab groups:

  • From self-managed GitLab to GitLab.com.
  • From GitLab.com to self-managed GitLab.
  • From one self-managed GitLab instance to another.
  • Between groups in the same GitLab instance.

WARNING: Migrating GitLab groups and projects by using direct transfer is currently unavailable. We don't have an estimated time for resolution. For more information, please contact support.

Migration by direct transfer creates a new copy of the group. If you want to move groups instead of copying groups, you can transfer groups if the groups are in the same GitLab instance. Transferring groups instead of migrating them is a faster and more complete option.

You can migrate groups in two ways:

If you migrate from GitLab.com to self-managed GitLab, an administrator can create users on the self-managed GitLab instance.

On self-managed GitLab, by default migrating group items is not available. To show the feature, an administrator can enable it in application settings.

Migrating groups by direct transfer copies the groups from one place to another. You can:

  • Copy many groups at once.
  • In the GitLab UI, copy top-level groups to:
    • Another top-level group.
    • The subgroup of any existing top-level group.
    • Another GitLab instance, including GitLab.com.
  • In the API, copy top-level groups and subgroups to these locations.
  • Copy groups with projects (in beta and not ready for production use) or without projects. Copying projects with groups is available:
    • On GitLab.com by default.

Not all group and project resources are copied. See list of copied resources below:

WARNING: Importing groups with projects is in beta. This feature is not ready for production use.

We invite you to leave your feedback about migrating by direct transfer in the feedback issue.

Migrating specific projects

Migrating groups by using direct transfer in the GitLab UI migrates all projects in the group. If you want to migrate only specific projects in the group by using direct transfer, you must use the API.

Known issues

  • Because of issue 406685, files with a filename longer than 255 characters are not migrated.
  • In GitLab 16.1 and earlier, you should not use direct transfer with scheduled scan execution policies.
  • For a list of other known issues, see epic 6629.
  • In GitLab 16.9 and earlier, because of issue 438422, you might see the DiffNote::NoteDiffFileCreationError error. When this error occurs, the diff of a note on a merge request's diff is missing, but the note and the merge request are still imported.
  • When mapped from the source instance, shared members are mapped as direct members on the destination unless those memberships already exist on the destination. This means that importing a top-level group on the source instance to a top-level group on the destination instance always maps to direct members in projects, even though the source top-level group contains the necessary shared membership hierarchy details. Support for full mapping of shared memberships is proposed in issue 458345.
  • In GitLab 17.0, 17.1, and 17.2, imported epics and work items are mapped to the importing user rather than the original author.

Estimating migration duration

Estimating the duration of migration by direct transfer is difficult. The following factors affect migration duration:

  • Hardware and database resources available on the source and destination GitLab instances. More resources on the source and destination instances can result in shorter migration duration because:
    • The source instance receives API requests, and extracts and serializes the entities to export.
    • The destination instance runs the jobs and creates the entities in its database.
  • Complexity and size of data to be exported. For example, imagine you want to migrate two different projects with 1000 merge requests each. The two projects can take very different amounts of time to migrate if one of the projects has a lot more attachments, comments, and other items on the merge requests. Therefore, the number of merge requests on a project is a poor predictor of how long a project will take to migrate.

There's no exact formula to reliably estimate a migration. However, the average durations of each pipeline worker importing a project relation can help you to get an idea of how long importing your projects might take:

Project resource type Average time (in seconds) to import a record
Empty Project 2.4
Repository 20
Project Attributes 1.5
Members 0.2
Labels 0.1
Milestones 0.07
Badges 0.1
Issues 0.1
Snippets 0.05
Snippet Repositories 0.5
Boards 0.1
Merge Requests 1
External Pull Requests 0.5
Protected Branches 0.1
Project Feature 0.3
Container Expiration Policy 0.3
Service Desk Setting 0.3
Releases 0.1
CI Pipelines 0.2
Commit Notes 0.05
Wiki 10
Uploads 0.5
LFS Objects 0.5
Design 0.1
Auto DevOps 0.1
Pipeline Schedules 0.5
References 5
Push rule 0.1

Though it's difficult to predict migration duration, we've seen:

  • 100 projects (19.9k issues, 83k merge requests, 100k+ pipelines) migrated in 8 hours.
  • 1926 projects (22k issues, 160k merge requests, 1.1 million pipelines) migrated in 34 hours.

If you are migrating large projects and encounter problems with timeouts or duration of the migration, see Reducing migration duration.

Reducing migration duration

These are some strategies for reducing the duration of migrations that use direct transfer.

Add Sidekiq workers to the destination instance

A single direct transfer migration runs 5 entities (groups or projects) per import at a time, independent of the number of workers available on the destination instance. That said, adding more Sidekiq worker processes on the destination instance speeds up migration by decreasing the time it takes to import each entity.

To add more Sidekiq workers on the destination instance, you can either:

  1. Use routing rules. This approach creates additional Sidekiq workers that are dedicated to direct transfer operations.
  2. Start multiple Sidekiq processes. This approach allows all queues in GitLab to make use of the additional Sidekiq worker processes.

An example of how to use the routing rules is below. This example:

  • Can be added to the /etc/gitlab/gitlab.rb file on a destination instance, with a subsequent reconfigure to apply the changes.
  • Creates four Sidekiq worker processes. Three of the processes are used exclusively for the direct transfer importer queues, and the last process is used for all other queues.
  • Should have a number of processes set that, at most, equal (and not exceed) the number of CPU cores you want to dedicate to Sidekiq. The Sidekiq worker process uses no more than one CPU core.
sidekiq['routing_rules'] = [
  ['feature_category=importers', 'importers'],
  ['*', 'default']
]

sidekiq['queue_groups'] = [
  # Run two processes just for importers
  'importers',
  'importers',
  'importers',

  # Run one 'catchall' process on the default and mailers queues
  'default,mailers'
]

If you are using GitLab 16.11 and earlier, explicitly disable any queue selectors:

sidekiq['queue_selector'] = false

Increasing the number of workers on the destination instance helps reduce the migration duration until the source instance hardware resources are saturated. Exporting and importing relations in batches, available by default from GitLab 16.8, makes having enough available workers on the destination instance even more useful.

Redistribute large projects or start separate migrations

The number of workers on the source instance should be enough to export the 5 concurrent entities in parallel (for each running import). Otherwise, there can be delays and potential timeouts as the destination is waiting for exported data to become available.

Distributing projects in different groups helps to avoid timeouts. If several large projects are in the same group, you can:

  1. Move large projects to different groups or subgroups.
  2. Start separate migrations each group and subgroup.

The GitLab UI can only migrate top-level groups. Using the API, you can also migrate subgroups.

Limits

  • Eight hour time limit on migrations removed in GitLab 16.7.

Hardcoded limits apply on migration by direct transfer.

Limit Description
6 Maximum number of migrations permitted by a destination GitLab instance per minute per user. Introduced in GitLab 15.9.
210 seconds Maximum number of seconds to wait for decompressing an archive file.
50 MB Maximum length an NDJSON row can have.
5 minutes Maximum number of seconds until an empty export status on source instance is raised.

Configurable limits are also available.

In GitLab 16.3 and later, the following previously hard-coded settings are configurable:

  • Maximum relation size that can be downloaded from the source instance (set to 5 GiB).
  • Maximum size of a decompressed archive (set to 10 GiB).

You can test the maximum relation size limit using these APIs:

If either API produces files larger than the maximum relation size limit, group migration by direct transfer fails.

Visibility rules

After migration:

  • Private groups and projects stay private.
  • Internal groups and projects:
    • Stay internal when copied into an internal group unless internal visibility is restricted. In that case, the groups and projects become private.
    • Become private when copied into a private group.
  • Public groups and projects:
    • Stay public when copied into a public group unless public visibility is restricted. In that case, the groups and projects become internal.
    • Become internal when copied into an internal group unless internal visibility is restricted. In that case, the groups and projects become private.
    • Become private when copied into a private group.

If you used a private network on your source instance to hide content from the general public, make sure to have a similar setup on the destination instance, or to import into a private group.

Migration by direct transfer process

See Migrate groups and projects by using direct transfer.