• Zeger-Jan van de Weg's avatar
    Allow public forks to be deduplicated · 896c0bdb
    Zeger-Jan van de Weg authored
    When a project is forked, the new repository used to be a deep copy of everything
    stored on disk by leveraging `git clone`. This works well, and makes isolation
    between repository easy. However, the clone is at the start 100% the same as the
    origin repository. And in the case of the objects in the object directory, this
    is almost always going to be a lot of duplication.
    
    Object Pools are a way to create a third repository that essentially only exists
    for its 'objects' subdirectory. This third repository's object directory will be
    set as alternate location for objects. This means that in the case an object is
    missing in the local repository, git will look in another location. This other
    location is the object pool repository.
    
    When Git performs garbage collection, it's smart enough to check the
    alternate location. When objects are duplicated, it will allow git to
    throw one copy away. This copy is on the local repository, where to pool
    remains as is.
    
    These pools have an origin location, which for now will always be a
    repository that itself is not a fork. When the root of a fork network is
    forked by a user, the fork still clones the full repository. Async, the
    pool repository will be created.
    
    Either one of these processes can be done earlier than the other. To
    handle this race condition, the Join ObjectPool operation is
    idempotent. Given its idempotent, we can schedule it twice, with the
    same effect.
    
    To accommodate the holding of state two migrations have been added.
    1. Added a state column to the pool_repositories column. This column is
    managed by the state machine, allowing for hooks on transitions.
    2. pool_repositories now has a source_project_id. This column in
    convenient to have for multiple reasons: it has a unique index allowing
    the database to handle race conditions when creating a new record. Also,
    it's nice to know who the host is. As that's a short link to the fork
    networks root.
    
    Object pools are only available for public project, which use hashed
    storage and when forking from the root of the fork network. (That is,
    the project being forked from itself isn't a fork)
    
    In this commit message I use both ObjectPool and Pool repositories,
    which are alike, but different from each other. ObjectPool refers to
    whatever is on the disk stored and managed by Gitaly. PoolRepository is
    the record in the database.
    896c0bdb
Name
Last commit
Last update
..
gitlab/github_import Loading commit data...
application_worker.rb Loading commit data...
auto_devops_queue.rb Loading commit data...
cluster_applications.rb Loading commit data...
cluster_queue.rb Loading commit data...
cronjob_queue.rb Loading commit data...
each_shard_worker.rb Loading commit data...
exception_backtrace.rb Loading commit data...
mail_scheduler_queue.rb Loading commit data...
new_issuable.rb Loading commit data...
object_pool_queue.rb Loading commit data...
object_storage_queue.rb Loading commit data...
pipeline_background_queue.rb Loading commit data...
pipeline_queue.rb Loading commit data...
project_import_options.rb Loading commit data...
project_start_import.rb Loading commit data...
repository_check_queue.rb Loading commit data...
todos_destroyer_queue.rb Loading commit data...
waitable_worker.rb Loading commit data...