1. 06 Feb, 2019 2 commits
  2. 11 Jan, 2019 1 commit
  3. 19 Dec, 2018 1 commit
  4. 13 Dec, 2018 1 commit
    • Zeger-Jan van de Weg's avatar
      Restore Object Pools when restoring an object pool · 89a407dc
      Zeger-Jan van de Weg authored
      Pool repositories are persisted in the database, and when the DB is
      restored, the data need to be restored on disk. This is done by
      resetting the state machine and rescheduling the object pool creation.
      
      This is not an exact replica of the state like at the time of the
      creation of the backup. However, the data is consistent again.
      
      Dumping isn't required as internally GitLab uses git bundles which
      bundle all refs and include all objects in the bundle that they require,
      reduplicating as more repositories get backed up. This does require more
      data to be stored.
      
      Fixes https://gitlab.com/gitlab-org/gitaly/issues/1355
      89a407dc
  5. 07 Dec, 2018 1 commit
    • Zeger-Jan van de Weg's avatar
      Allow public forks to be deduplicated · 896c0bdb
      Zeger-Jan van de Weg authored
      When a project is forked, the new repository used to be a deep copy of everything
      stored on disk by leveraging `git clone`. This works well, and makes isolation
      between repository easy. However, the clone is at the start 100% the same as the
      origin repository. And in the case of the objects in the object directory, this
      is almost always going to be a lot of duplication.
      
      Object Pools are a way to create a third repository that essentially only exists
      for its 'objects' subdirectory. This third repository's object directory will be
      set as alternate location for objects. This means that in the case an object is
      missing in the local repository, git will look in another location. This other
      location is the object pool repository.
      
      When Git performs garbage collection, it's smart enough to check the
      alternate location. When objects are duplicated, it will allow git to
      throw one copy away. This copy is on the local repository, where to pool
      remains as is.
      
      These pools have an origin location, which for now will always be a
      repository that itself is not a fork. When the root of a fork network is
      forked by a user, the fork still clones the full repository. Async, the
      pool repository will be created.
      
      Either one of these processes can be done earlier than the other. To
      handle this race condition, the Join ObjectPool operation is
      idempotent. Given its idempotent, we can schedule it twice, with the
      same effect.
      
      To accommodate the holding of state two migrations have been added.
      1. Added a state column to the pool_repositories column. This column is
      managed by the state machine, allowing for hooks on transitions.
      2. pool_repositories now has a source_project_id. This column in
      convenient to have for multiple reasons: it has a unique index allowing
      the database to handle race conditions when creating a new record. Also,
      it's nice to know who the host is. As that's a short link to the fork
      networks root.
      
      Object pools are only available for public project, which use hashed
      storage and when forking from the root of the fork network. (That is,
      the project being forked from itself isn't a fork)
      
      In this commit message I use both ObjectPool and Pool repositories,
      which are alike, but different from each other. ObjectPool refers to
      whatever is on the disk stored and managed by Gitaly. PoolRepository is
      the record in the database.
      896c0bdb
  6. 29 Nov, 2018 1 commit
  7. 28 Nov, 2018 1 commit
  8. 27 Nov, 2018 1 commit
    • Zeger-Jan van de Weg's avatar
      Rename the Repository table to PoolRepository · fff77541
      Zeger-Jan van de Weg authored
      To separate the different kinds of repositories we have at GitLab this
      table will be renamed to pool_repositories. A project can, for now at
      least, be member of none, or one of these. The table will get additional
      columns in a later merge request where more logic is implemented for the
      model.
      
      Further included is a small refactor of logic around hashing ids for the
      disk_path, mainly to ensure a previous implementation is reusable.
      
      The disk_path for the pool_repositories table no longer has a NOT NULL
      constraint, but given the hashing of the ID requires the DB to assign
      the record an ID, an after_create hook is used to update the value.
      
      A related MR is:
      https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23143, adding
      tables for 'normal' repositories and wiki_repositories.
      fff77541
  9. 05 Nov, 2018 1 commit