Commit cd645092 authored by Sean McGivern's avatar Sean McGivern

Merge branch '52767-more-chaos-for-gitlab' into 'master'

Add more chaos to GitLab

Closes #53362 and #52767

See merge request gitlab-org/gitlab-ce!22746
parents 8a55e477 673f0625
# frozen_string_literal: true
class ChaosController < ActionController::Base
before_action :validate_request
def leakmem
memory_mb = (params[:memory_mb]&.to_i || 100)
duration_s = (params[:duration_s]&.to_i || 30).seconds
start =
retainer = []
# Add `n` 1mb chunks of memory to the retainer array
memory_mb.times { retainer << "x" * 1.megabyte }
duration_taken = ( - start).seconds
Kernel.sleep duration_s - duration_taken if duration_s > duration_taken
render text: "OK", content_type: 'text/plain'
def cpuspin
duration_s = (params[:duration_s]&.to_i || 30).seconds
end_time = + duration_s.seconds
rand while < end_time
render text: "OK", content_type: 'text/plain'
def sleep
duration_s = (params[:duration_s]&.to_i || 30).seconds
Kernel.sleep duration_s
render text: "OK", content_type: 'text/plain'
def kill
def validate_request
# GITLAB_CHAOS_SECRET is required unless you're running in Development mode
if !secret && !Rails.env.development?
render text: "chaos misconfigured: please configure GITLAB_CHAOS_SECRET when using GITLAB_ENABLE_CHAOS_ENDPOINTS outside of a development environment", content_type: 'text/plain', status: 500
return unless secret
unless request.headers["HTTP_X_CHAOS_SECRET"] == secret
render text: "To experience chaos, please set X-Chaos-Secret header", content_type: 'text/plain', status: 401
title: Add endpoints for simulating certain failure modes in the application
merge_request: 22746
type: other
......@@ -82,6 +82,13 @@ Rails.application.routes.draw do
draw :operations
draw :instance_statistics
get '/chaos/leakmem' => 'chaos#leakmem'
get '/chaos/cpuspin' => 'chaos#cpuspin'
get '/chaos/sleep' => 'chaos#sleep'
get '/chaos/kill' => 'chaos#kill'
concern :clusterable do
# Generating chaos in a test GitLab instance
As [Werner Vogels](, the CTO at Amazon Web Services, famously put it, **Everything fails, all the time**.
As a developer, it's as important to consider the failure modes in which your software will operate as much as normal operation. Doing so can mean the difference between a minor hiccup leading to a scattering of `500` errors experienced by a tiny fraction of users and a full site outage that affects all users for an extended period.
To paraphrase [Tolstoy](, _all happy servers are alike, but all failing servers are failing in their own way_. Luckily, there are ways we can attempt to simulate these failure modes, and the chaos endpoints are tools for assisting in this process.
Currently, there are four endpoints for simulating the following conditions:
- Slow requests.
- CPU-bound requests.
- Memory leaks.
- Unexpected process crashes.
## Enabling chaos endpoints
For obvious reasons, these endpoints are not enabled by default. They can be enabled by setting the `GITLAB_ENABLE_CHAOS_ENDPOINTS` environment variable to `1`.
For example, if you're using the [GDK]( this can be done with the following command:
## Securing the chaos endpoints
DANGER: **Danger:**
It is highly recommended that you secure access to the chaos endpoints using a secret token. This is recommended when enabling these endpoints locally and essential when running in a staging or other shared environment. You should not enable them in production unless you absolutely know what you're doing.
A secret token can be set through the `GITLAB_CHAOS_SECRET` environment variable. For example, when using the [GDK]( this can be done with the following command:
Replace `secret` with your own secret token.
## Invoking chaos
Once you have enabled the chaos endpoints and restarted the application, you can start testing using the endpoints.
## Memory leaks
To simulate a memory leak in your application, use the `/-/chaos/leakmem` endpoint.
NOTE: **Note:**
The memory is not retained after the request finishes. Once the request has completed, the Ruby garbage collector will attempt to recover the memory.
GET /-/chaos/leakmem
GET /-/chaos/leakmem?memory_mb=1024
GET /-/chaos/leakmem?memory_mb=1024&duration_s=50
| Attribute | Type | Required | Description |
| ------------ | ------- | -------- | ---------------------------------------------------------------------------------- |
| `memory_mb` | integer | no | How much memory, in MB, should be leaked. Defaults to 100MB. |
| `duration_s` | integer | no | Minimum duration, in seconds, that the memory should be retained. Defaults to 30s. |
curl http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10 --header 'X-Chaos-Secret: secret'
## CPU spin
This endpoint attempts to fully utilise a single core, at 100%, for the given period.
Depending on your rack server setup, your request may timeout after a predermined period (normally 60 seconds).
If you're using Unicorn, this is done by killing the worker process.
GET /-/chaos/cpuspin
GET /-/chaos/cpuspin?duration_s=50
| Attribute | Type | Required | Description |
| ------------ | ------- | -------- | --------------------------------------------------------------------- |
| `duration_s` | integer | no | Duration, in seconds, that the core will be utilised. Defaults to 30s |
curl http://localhost:3000/-/chaos/cpuspin?duration_s=60 --header 'X-Chaos-Secret: secret'
## Sleep
This endpoint is similar to the CPU Spin endpoint but simulates off-processor activity, such as network calls to backend services. It will sleep for a given duration.
As with the CPU Spin endpoint, this may lead to your request timing out if duration exceeds the configured limit.
GET /-/chaos/sleep
GET /-/chaos/sleep?duration_s=50
| Attribute | Type | Required | Description |
| ------------ | ------- | -------- | ---------------------------------------------------------------------- |
| `duration_s` | integer | no | Duration, in seconds, that the request will sleep for. Defaults to 30s |
curl http://localhost:3000/-/chaos/sleep?duration_s=60 --header 'X-Chaos-Secret: secret'
## Kill
This endpoint will simulate the unexpected death of a worker process using a `kill` signal.
NOTE: **Note:**
Since this endpoint uses the `KILL` signal, the worker is not given a chance to cleanup or shutdown.
GET /-/chaos/kill
curl http://localhost:3000/-/chaos/kill --header 'X-Chaos-Secret: secret'
......@@ -34,13 +34,14 @@ graphs/dashboards.
## Tooling
GitLab provides built-in tools to aid the process of improving performance:
GitLab provides built-in tools to help improve performance and availability:
* [Profiling](
* [Sherlock](
* [GitLab Performance Monitoring](../administration/monitoring/performance/
* [Request Profiling](../administration/monitoring/performance/
* [QueryRecoder]( for preventing `N+1` regressions
* [Chaos endpoints]( for testing failure scenarios. Intended mainly for testing availability.
GitLab employees can use's performance monitoring systems located at
<>, this requires you to log in using your
Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment