GitLab

During a project import, it's possible that new branches are created by
the importer to handle pull requests that have been created from forked
projects, which would increment the `pushes_since_gc` value via
`HousekeepingService.increment!` before a full garbage collection gets
to run. This causes HousekeepingService to skip the full `git gc` and
move to the incremental repack mode. To ensure that a garbage collection
is run to pack refs and objects, explicitly execute the task.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/59477

This style change enforces `return if ...` instead of
`return nil if ...` to save maintainers a few minor review points

This is a fix for the Hashed Storage migration and Rollback procedure
to ignore any project-level validation error that can happen in a
long-running instance.

There are many situations where defaults and acceptable values changed
but, because we didn't provide a migration to "valid" attributes, it
can happen that project will not be `valid? => true`.

Because the changes we are making are limited to setting a project as
read_only or changing the storage_level, it's safe to bypass validation.

- Move some specs into contexts
- Let get_slugs method take a parameter and return a specific slug.
- Add rescues when using Addressable::URI.

Exceptions were also simplified from 2 to 1.

`try_to_set_repository_read_only!` is now on the Base class.
Simplified Exception classes from 2 to 1 with a more descriptive name.

New class contains the ExclusiveLease specifics that is shared among
both the Migration and Rollback workers.

This change takes the case where repository doesn't exist on disk
and make it ignore but succeed the step, increasing or decreasing the
version depending on the operation.

Rollback is done similar to Migration for the Hashed Storage.
It also shares the same ExclusiveLease key to prevent both happening
at the same time.

All Hashed Storage related workers now share the same queue namespace
which allows for assigning dedicated workers easily.

We are adding sidekiq workers and service classes to allow to rollback
a hashed storage migration. There are some refactoring involved as well
as part of the code can be reused by both the migration and the rollback
logic.

The API get projects/:id/traffic/fetches allows user with write
access to the repository to get the number of clones for the
last 30 days.

The previous behavior would pass in a list of parameters
to Shell, but we can improve this by using the WikiFormatter
and Project models to give us the same information.

When hashed storage is in use, it's helpful to have the project
name associated with the request.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1394

RubyZip allows us to perform strong validation of
expanded paths where we do extract file.

We introduce the following additional checks
to extract routines:

1. None of path components can be symlinked,
2. We drop privileges support for directories,
3. Symlink source needs to point within the target directory,
   like `public/`,
4. The symlink source needs to exist ahead of time.

We need this new state for the Geo event logic in EE

Specs were reviewed and improved to better cover the current behavior.
There was some standardization done as well to facilitate the
implementation of the rollback functionality.

StorageMigratorWorker was extracted to HashedStorage namespace were
RollbackerWorker will live one as well.

This will serve as new base class for both Migrate and Rollback

This is part of the refactor to include a RollbackService into
HashedStorage module

This includes a set of APIs to manipulate container registry.
This includes also an ability to delete tags based on requested
criteria, like keep-last-n, matching-name, older-than.

RubyZip allows us to perform strong validation of
expanded paths where we do extract file.

We introduce the following additional checks
to extract routines:

1. None of path components can be symlinked,
2. We drop privileges support for directories,
3. Symlink source needs to point within the target directory,
   like `public/`,
4. The symlink source needs to exist ahead of time.

We still rely on the Dirty API for project rename (before/after) values,
but we don't access the dirty api from the service class anymore.

The previous value is now part of the initialization, which makes it
easier to test and the behavior is clearer.

The same was done with the `rename_repo` on the Storage classes, we now
provide before and after values as part of the method signature.

During a previous refactor on project model, code related to the
hashed storage was extracted into AfterRenameService, see
4b9c17f1.

The "path_before" was changed from using `previous_changes['path']` to
`path_was`. They are not equivalent. `path_was` exists reliably only
*before* persisting to the database. After database persistence is
confirmed, the value is moved to `previous_changes[:attribute_name]`.

Because the repository/attachments rename or storage upgrade happens
after it was persisted to the database, we were in fact not informing
the right parameters (and therefore not doing what it was supposed to).

1. When removing projects, we can end-up leaving the +deleted
repo path dirty and not successfully removing the non-deleted
namespace (mv process is not atomic and can be killed without
fully moving the path).

2. In order to solve that, we're adding a clean-up phase on
ensure which will schedule possible staled +deleted path deletion.
Note that we don't check the current state (if there is or not a
repo) in order to schedule the deletion. That's intentional
in order to leverage Gitlab::GitalyClient::NamespaceService#remove
idempotency and ensure consistency.

This refactors some of the logic used for protecting default branches,
in particular Project#after_create_default_branch. The logic for this
method is moved into a separate service class. Ideally we'd get rid of
Project#after_create_default_branch entirely, but unfortunately
Project#after_import depends on it. This means it has to stick around
until we also refactor Project#after_import.

For branch protection levels we introduce
Gitlab::Access::BranchProtection, which provides a small wrapper around
Integer based branch protection levels. Using this class removes the
need for having to constantly refer to Gitlab::Access::PROTECTION_*
constants.

Instead of querying relations into ids we just pass them to the model
scope because the scope supports it now.

Also changes other calls to `Milestone.for_projects_and_groups`

In the previous code, we locked the project during the migration
scheduling step, which works fine for small setups, but can be
problematic in really big installations.

We now moved the logic to inside the worker, so we minimize the time a
project will be read-only. We also make sure we only do that if
reference counter is `0` (no current operation is in progress).

Re-use operations controller which already handles tracing settings.

This commit prepares the structure for the upcoming feature error
tracking.