GitLab wurde erfolgreich aktualisiert. Durch regelmäßige Updates bleibt das THM GitLab sicher. Danke für Ihre Geduld.

Commit cb3b4a15 authored by Paul Charlton's avatar Paul Charlton Committed by Robert Speicher

Support multiple Redis instances based on queue type

parent 4daa6da5
...@@ -3,3 +3,4 @@ lib/gitlab/sanitizers/svg/whitelist.rb ...@@ -3,3 +3,4 @@ lib/gitlab/sanitizers/svg/whitelist.rb
lib/gitlab/diff/position_tracer.rb lib/gitlab/diff/position_tracer.rb
app/policies/project_policy.rb app/policies/project_policy.rb
app/models/concerns/relative_positioning.rb app/models/concerns/relative_positioning.rb
lib/gitlab/redis/*.rb
...@@ -31,6 +31,9 @@ eslint-report.html ...@@ -31,6 +31,9 @@ eslint-report.html
/config/initializers/smtp_settings.rb /config/initializers/smtp_settings.rb
/config/initializers/relative_url.rb /config/initializers/relative_url.rb
/config/resque.yml /config/resque.yml
/config/redis.cache.yml
/config/redis.queues.yml
/config/redis.shared_state.yml
/config/unicorn.rb /config/unicorn.rb
/config/secrets.yml /config/secrets.yml
/config/sidekiq.yml /config/sidekiq.yml
......
...@@ -4,7 +4,10 @@ class HealthController < ActionController::Base ...@@ -4,7 +4,10 @@ class HealthController < ActionController::Base
CHECKS = [ CHECKS = [
Gitlab::HealthChecks::DbCheck, Gitlab::HealthChecks::DbCheck,
Gitlab::HealthChecks::RedisCheck, Gitlab::HealthChecks::Redis::RedisCheck,
Gitlab::HealthChecks::Redis::CacheCheck,
Gitlab::HealthChecks::Redis::QueuesCheck,
Gitlab::HealthChecks::Redis::SharedStateCheck,
Gitlab::HealthChecks::FsShardsCheck Gitlab::HealthChecks::FsShardsCheck
].freeze ].freeze
......
...@@ -149,7 +149,7 @@ def is_runner_queue_value_latest?(value) ...@@ -149,7 +149,7 @@ def is_runner_queue_value_latest?(value)
private private
def cleanup_runner_queue def cleanup_runner_queue
Gitlab::Redis.with do |redis| Gitlab::Redis::Queues.with do |redis|
redis.del(runner_queue_key) redis.del(runner_queue_key)
end end
end end
......
...@@ -1386,15 +1386,15 @@ def append_or_update_attribute(name, value) ...@@ -1386,15 +1386,15 @@ def append_or_update_attribute(name, value)
end end
def pushes_since_gc def pushes_since_gc
Gitlab::Redis.with { |redis| redis.get(pushes_since_gc_redis_key).to_i } Gitlab::Redis::SharedState.with { |redis| redis.get(pushes_since_gc_redis_shared_state_key).to_i }
end end
def increment_pushes_since_gc def increment_pushes_since_gc
Gitlab::Redis.with { |redis| redis.incr(pushes_since_gc_redis_key) } Gitlab::Redis::SharedState.with { |redis| redis.incr(pushes_since_gc_redis_shared_state_key) }
end end
def reset_pushes_since_gc def reset_pushes_since_gc
Gitlab::Redis.with { |redis| redis.del(pushes_since_gc_redis_key) } Gitlab::Redis::SharedState.with { |redis| redis.del(pushes_since_gc_redis_shared_state_key) }
end end
def route_map_for(commit_sha) def route_map_for(commit_sha)
...@@ -1457,7 +1457,7 @@ def cross_project_reference?(from) ...@@ -1457,7 +1457,7 @@ def cross_project_reference?(from)
from && self != from from && self != from
end end
def pushes_since_gc_redis_key def pushes_since_gc_redis_shared_state_key
"projects/#{id}/pushes_since_gc" "projects/#{id}/pushes_since_gc"
end end
......
...@@ -3,7 +3,10 @@ ...@@ -3,7 +3,10 @@
class MetricsService class MetricsService
CHECKS = [ CHECKS = [
Gitlab::HealthChecks::DbCheck, Gitlab::HealthChecks::DbCheck,
Gitlab::HealthChecks::RedisCheck, Gitlab::HealthChecks::Redis::RedisCheck,
Gitlab::HealthChecks::Redis::CacheCheck,
Gitlab::HealthChecks::Redis::QueuesCheck,
Gitlab::HealthChecks::Redis::SharedStateCheck,
Gitlab::HealthChecks::FsShardsCheck Gitlab::HealthChecks::FsShardsCheck
].freeze ].freeze
......
module Milestones module Milestones
class DestroyService < Milestones::BaseService class DestroyService < Milestones::BaseService
def execute(milestone) def execute(milestone)
return unless milestone.is_project_milestone?
Milestone.transaction do Milestone.transaction do
update_params = { milestone: nil } update_params = { milestone: nil }
milestone.issues.each do |issue| milestone.issues.each do |issue|
Issues::UpdateService.new(parent, current_user, update_params).execute(issue) Issues::UpdateService.new(project, current_user, update_params).execute(issue)
end end
milestone.merge_requests.each do |merge_request| milestone.merge_requests.each do |merge_request|
MergeRequests::UpdateService.new(parent, current_user, update_params).execute(merge_request) MergeRequests::UpdateService.new(project, current_user, update_params).execute(merge_request)
end end
event_service.destroy_milestone(milestone, current_user) event_service.destroy_milestone(milestone, current_user)
......
...@@ -19,4 +19,132 @@ an ERB file and then loads the resulting YML as its configuration. ...@@ -19,4 +19,132 @@ an ERB file and then loads the resulting YML as its configuration.
This file is called `resque.yml` for historical reasons. We are **NOT** This file is called `resque.yml` for historical reasons. We are **NOT**
using Resque at the moment. It is used to specify Redis configuration using Resque at the moment. It is used to specify Redis configuration
values instead. values when a single database instance of Redis is desired.
# Advanced Redis configuration files
In more advanced configurations of Redis key-value storage, it is desirable
to separate the keys by lifecycle and intended use to ease provisioning and
management of scalable Redis clusters.
These settings provide routing and other configuration data (such as sentinel,
persistence policies, and other Redis customization) for connections
to Redis single instances, Redis sentinel, and Redis clusters.
If desired, the routing URL provided by these settings can be used with:
1. Unix Socket
1. named socket for each Redis instance desired.
2. `database number` for each Redis instance desired.
2. TCP Socket
1. `host name` or IP for each Redis instance desired
2. TCP port number for each Redis instance desired
3. `database number` for each Redis instance desired
## Example URL attribute formats for GitLab Redis `.yml` configuration files
* Unix Socket, default Redis database (0)
* `url: unix:/path/to/redis.sock`
* `url: unix:/path/to/redis.sock?db=`
* Unix Socket, Redis database 44
* `url: unix:/path/to/redis.sock?db=44`
* `url: unix:/path/to/redis.sock?extra=foo&db=44`
* TCP Socket for Redis on localhost, port 6379, database 33
* `url: redis://:mynewpassword@localhost:6379/33`
* TCP Socket for Redis on remote host `myserver`, port 6379, database 33
* `url: redis://:mynewpassword@myserver:6379/33`
## redis.cache.yml
If configured, `redis.cache.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for `Rails.cache` and other volatile non-persistent data which enhances
the performance of GitLab.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_CACHE_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `cache` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
2. URL from `redis.cache.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6380`
The order of precedence for all other configuration settings for `cache`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
2. the configuration file `redis.cache.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
## redis.queues.yml
If configured, `redis.queues.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::Queues`.
These queues are intended to be the foundation
of reliable inter-process communication between modules, whether on the same
host node, or within a cluster. The primary clients of the queues are
SideKiq, Mailroom, CI Runner, Workhorse, and push services. Settings here can
be overridden by the environment variable
`GITLAB_REDIS_QUEUES_CONFIG_FILE` which provides an alternate location for
configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `queues` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
2. URL from `redis.queues.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6381`
The order of precedence for all other configuration settings for `queues`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
2. the configuration file `redis.queues.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
## redis.shared_state.yml
If configured, `redis.shared_state.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::SharedState` such as session state,
and rate limiting.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `shared_state` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
2. URL from `redis.shared_state.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6382`
The order of precedence for all other configuration settings for `shared_state`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
2. the configuration file `redis.shared_state.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
...@@ -6,7 +6,9 @@ ...@@ -6,7 +6,9 @@
module Gitlab module Gitlab
class Application < Rails::Application class Application < Rails::Application
require_dependency Rails.root.join('lib/gitlab/redis') require_dependency Rails.root.join('lib/gitlab/redis/cache')
require_dependency Rails.root.join('lib/gitlab/redis/queues')
require_dependency Rails.root.join('lib/gitlab/redis/shared_state')
require_dependency Rails.root.join('lib/gitlab/request_context') require_dependency Rails.root.join('lib/gitlab/request_context')
# Settings in config/environments/* take precedence over those specified here. # Settings in config/environments/* take precedence over those specified here.
...@@ -142,15 +144,15 @@ class Application < Rails::Application ...@@ -142,15 +144,15 @@ class Application < Rails::Application
end end
end end
# Use Redis caching across all environments # Use caching across all environments
redis_config_hash = Gitlab::Redis.params caching_config_hash = Gitlab::Redis::Cache.params
redis_config_hash[:namespace] = Gitlab::Redis::CACHE_NAMESPACE caching_config_hash[:namespace] = Gitlab::Redis::Cache::CACHE_NAMESPACE
redis_config_hash[:expires_in] = 2.weeks # Cache should not grow forever caching_config_hash[:expires_in] = 2.weeks # Cache should not grow forever
if Sidekiq.server? # threaded context if Sidekiq.server? # threaded context
redis_config_hash[:pool_size] = Sidekiq.options[:concurrency] + 5 caching_config_hash[:pool_size] = Sidekiq.options[:concurrency] + 5
redis_config_hash[:pool_timeout] = 1 caching_config_hash[:pool_timeout] = 1
end end
config.cache_store = :redis_store, redis_config_hash config.cache_store = :redis_store, caching_config_hash
config.active_record.raise_in_transactional_callbacks = true config.active_record.raise_in_transactional_callbacks = true
......
# Make sure we initialize a Redis connection pool before Sidekiq starts # Make sure we initialize a Redis connection pool before multi-threaded
# multi-threaded execution. # execution starts by
Gitlab::Redis.with { nil } # 1. Sidekiq
# 2. Rails.cache
# 3. HTTP clients
Gitlab::Redis::Cache.with { nil }
Gitlab::Redis::Queues.with { nil }
Gitlab::Redis::SharedState.with { nil }
Rails.application.config.peek.adapter = :redis, { client: ::Redis.new(Gitlab::Redis.params) } Rails.application.config.peek.adapter = :redis, { client: ::Redis.new(Gitlab::Redis::Cache.params) }
Peek.into Peek::Views::Host Peek.into Peek::Views::Host
Peek.into Peek::Views::PerformanceBar Peek.into Peek::Views::PerformanceBar
......
...@@ -19,12 +19,12 @@ ...@@ -19,12 +19,12 @@
if Rails.env.test? if Rails.env.test?
Gitlab::Application.config.session_store :cookie_store, key: "_gitlab_session" Gitlab::Application.config.session_store :cookie_store, key: "_gitlab_session"
else else
redis_config = Gitlab::Redis.params sessions_config = Gitlab::Redis::SharedState.params
redis_config[:namespace] = Gitlab::Redis::SESSION_NAMESPACE sessions_config[:namespace] = Gitlab::Redis::SharedState::SESSION_NAMESPACE
Gitlab::Application.config.session_store( Gitlab::Application.config.session_store(
:redis_store, # Using the cookie_store would enable session replay attacks. :redis_store, # Using the cookie_store would enable session replay attacks.
servers: redis_config, servers: sessions_config,
key: cookie_key, key: cookie_key,
secure: Gitlab.config.gitlab.https, secure: Gitlab.config.gitlab.https,
httponly: true, httponly: true,
......
# Custom Redis configuration # Custom Queues configuration
redis_config_hash = Gitlab::Redis.params queues_config_hash = Gitlab::Redis::Queues.params
redis_config_hash[:namespace] = Gitlab::Redis::SIDEKIQ_NAMESPACE queues_config_hash[:namespace] = Gitlab::Redis::Queues::SIDEKIQ_NAMESPACE
# Default is to retry 25 times with exponential backoff. That's too much. # Default is to retry 25 times with exponential backoff. That's too much.
Sidekiq.default_worker_options = { retry: 3 } Sidekiq.default_worker_options = { retry: 3 }
Sidekiq.configure_server do |config| Sidekiq.configure_server do |config|
config.redis = redis_config_hash config.redis = queues_config_hash
config.server_middleware do |chain| config.server_middleware do |chain|
chain.add Gitlab::SidekiqMiddleware::ArgumentsLogger if ENV['SIDEKIQ_LOG_ARGUMENTS'] chain.add Gitlab::SidekiqMiddleware::ArgumentsLogger if ENV['SIDEKIQ_LOG_ARGUMENTS']
...@@ -54,7 +54,7 @@ ...@@ -54,7 +54,7 @@
end end
Sidekiq.configure_client do |config| Sidekiq.configure_client do |config|
config.redis = redis_config_hash config.redis = queues_config_hash
config.client_middleware do |chain| config.client_middleware do |chain|
chain.add Gitlab::SidekiqStatus::ClientMiddleware chain.add Gitlab::SidekiqStatus::ClientMiddleware
......
...@@ -21,7 +21,7 @@ ...@@ -21,7 +21,7 @@
:delivery_method: sidekiq :delivery_method: sidekiq
:delivery_options: :delivery_options:
:redis_url: <%= config[:redis_url].to_json %> :redis_url: <%= config[:redis_url].to_json %>
:namespace: <%= Gitlab::Redis::SIDEKIQ_NAMESPACE %> :namespace: <%= Gitlab::Redis::Queues::SIDEKIQ_NAMESPACE %>
:queue: email_receiver :queue: email_receiver
:worker: EmailReceiverWorker :worker: EmailReceiverWorker
<% if config[:sentinels] %> <% if config[:sentinels] %>
...@@ -36,7 +36,7 @@ ...@@ -36,7 +36,7 @@
:arbitration_method: redis :arbitration_method: redis
:arbitration_options: :arbitration_options:
:redis_url: <%= config[:redis_url].to_json %> :redis_url: <%= config[:redis_url].to_json %>
:namespace: <%= Gitlab::Redis::MAILROOM_NAMESPACE %> :namespace: <%= Gitlab::Redis::Queues::MAILROOM_NAMESPACE %>
<% if config[:sentinels] %> <% if config[:sentinels] %>
:sentinels: :sentinels:
<% config[:sentinels].each do |sentinel| %> <% config[:sentinels].each do |sentinel| %>
......
# If you change this file in a Merge Request, please also create
# a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/10
#
# url: redis://localhost:6380
# sentinels:
# -
# host: localhost
# port: 26380 # point to sentinel, not to redis port
# -
# host: slave2
# port: 26380 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/10
#
# url: redis://localhost:6380
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.cache.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ce/administration/high_availability/redis.html
##
# url: redis://master:6380
# sentinels:
# -
# host: slave1
# port: 26380 # point to sentinel, not to redis port
# -
# host: slave2
# port: 26380 # point to sentinel, not to redis port
# If you change this file in a Merge Request, please also create
# a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/11
#
# url: redis://localhost:6381
# sentinels:
# -
# host: localhost
# port: 26381 # point to sentinel, not to redis port
# -
# host: slave2
# port: 26381 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/11
#
# url: redis://localhost:6381
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.queues.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ce/administration/high_availability/redis.html
##
# url: redis://master:6381
# sentinels:
# -
# host: slave1
# port: 26381 # point to sentinel, not to redis port
# -
# host: slave2
# port: 26381 # point to sentinel, not to redis port
# If you change this file in a Merge Request, please also create
# a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/12
#
# url: redis://localhost:6382
# sentinels:
# -
# host: localhost
# port: 26382 # point to sentinel, not to redis port
# -
# host: slave2
# port: 26382 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/12
#
# url: redis://localhost:6382
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.shared_state.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ce/administration/high_availability/redis.html
##
# url: redis://master:6382
# sentinels:
# -
# host: slave1
# port: 26382 # point to sentinel, not to redis port
# -
# host: slave2
# port: 26382 # point to sentinel, not to redis port
...@@ -56,7 +56,7 @@ def users_table ...@@ -56,7 +56,7 @@ def users_table
end end
def activities(from, to, page: 1) def activities(from, to, page: 1)
Gitlab::Redis.with do |redis| Gitlab::Redis::SharedState.with do |redis|
redis.zrangebyscore(USER_ACTIVITY_SET_KEY, from.to_i, to.to_i, redis.zrangebyscore(USER_ACTIVITY_SET_KEY, from.to_i, to.to_i,
with_scores: true, with_scores: true,
limit: limit(page)) limit: limit(page))
...@@ -64,7 +64,7 @@ def activities(from, to, page: 1) ...@@ -64,7 +64,7 @@ def activities(from, to, page: 1)
end end
def activities_count(from, to) def activities_count(from, to)
Gitlab::Redis.with do |redis| Gitlab::Redis::SharedState.with do |redis|
redis.zcount(USER_ACTIVITY_SET_KEY, from.to_i, to.to_i) redis.zcount(USER_ACTIVITY_SET_KEY, from.to_i, to.to_i)
end end
end end
......
...@@ -4,6 +4,11 @@ This is the documentation for configuring a Highly Available Redis setup when ...@@ -4,6 +4,11 @@ This is the documentation for configuring a Highly Available Redis setup when
you have installed Redis all by yourself and not using the bundled one that you have installed Redis all by yourself and not using the bundled one that
comes with the Omnibus packages. comes with the Omnibus packages.
Note also that you may elect to override all references to
`/home/git/gitlab/config/resque.yml` in accordance with the advanced Redis
settings outlined in
[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/README.md).
We cannot stress enough the importance of reading the We cannot stress enough the importance of reading the
[Overview section](redis.md#overview) of the Omnibus Redis HA as it provides [Overview section](redis.md#overview) of the Omnibus Redis HA as it provides
some invaluable information to the configuration of Redis. Please proceed to