index.md 52.2 KB
Newer Older
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
1 2
# Auto DevOps

3
> [Introduced][ce-37115] in GitLab 10.0. Generally available on GitLab 11.0.
4

5 6 7
Auto DevOps provides pre-defined CI/CD configuration which allows you to automatically detect, build, test,
deploy, and monitor your applications. Leveraging CI/CD best practices and tools, Auto DevOps aims
to simplify the setup and execution of a mature & modern software development lifecycle.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
8 9 10

## Overview

11
NOTE: **Enabled by default:**
12 13
Starting with GitLab 11.3, the Auto DevOps pipeline is enabled by default for all
projects. If it has not been explicitly enabled for the project, Auto DevOps will be automatically
14 15
disabled on the first pipeline failure. Your project will continue to use an alternative
[CI/CD configuration file](../../ci/yaml/README.md) if one is found. A GitLab
16
administrator can [change this setting](../../user/admin_area/settings/continuous_integration.html#auto-devops-core-only)
17 18
in the admin area.

19
With Auto DevOps, the software development process becomes easier to set up
20
as every project can have a complete workflow from verification to monitoring
21
with minimal configuration. Just push your code and GitLab takes
22 23 24
care of everything else. This makes it easier to start new projects and brings
consistency to how applications are set up throughout a company.

25 26 27
## Quick start

If you are using GitLab.com, see the [quick start guide](quick_start_guide.md)
28 29 30 31 32 33 34
for how to use Auto DevOps with GitLab.com and a Kubernetes cluster on Google Kubernetes
Engine (GKE).

If you are using a self-hosted instance of GitLab, you will need to configure the
[Google OAuth2 OmniAuth Provider](../../integration/google.md) before
you can configure a cluster on GKE. Once this is set up, you can follow the steps on the
[quick start guide](quick_start_guide.md) to get started.
35

36 37
## Comparison to application platforms and PaaS

38 39
Auto DevOps provides functionality that is often included in an application
platform or a Platform as a Service (PaaS). It takes inspiration from the
40
innovative work done by [Heroku](https://www.heroku.com/) and goes beyond it
41
in multiple ways:
42

43 44
1. Auto DevOps works with any Kubernetes cluster; you're not limited to running
   on GitLab's infrastructure. (Note that many features also work without Kubernetes.)
45 46
1. There is no additional cost (no markup on the infrastructure costs), and you
   can use a self-hosted Kubernetes cluster or Containers as a Service on any
47
   public cloud (for example, [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/)).
48 49
1. Auto DevOps has more features including security testing, performance testing,
   and code quality testing.
50
1. Auto DevOps offers an incremental graduation path. If you need advanced customizations,
51
   you can start modifying the templates without having to start over on a
52
   completely different platform. Review the [customizing](#customizing) section for more information.
53 54

## Features
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
55

56
Comprised of a set of stages, Auto DevOps brings these best practices to your
57
project in a simple and automatic way:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
58

59 60
1. [Auto Build](#auto-build)
1. [Auto Test](#auto-test)
61 62 63 64
1. [Auto Code Quality](#auto-code-quality-starter) **[STARTER]**
1. [Auto SAST (Static Application Security Testing)](#auto-sast-ultimate) **[ULTIMATE]**
1. [Auto Dependency Scanning](#auto-dependency-scanning-ultimate) **[ULTIMATE]**
1. [Auto License Management](#auto-license-management-ultimate) **[ULTIMATE]**
65
1. [Auto Container Scanning](#auto-container-scanning)
66
1. [Auto Review Apps](#auto-review-apps)
67
1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast-ultimate) **[ULTIMATE]**
68
1. [Auto Deploy](#auto-deploy)
69
1. [Auto Browser Performance Testing](#auto-browser-performance-testing-premium) **[PREMIUM]**
70
1. [Auto Monitoring](#auto-monitoring)
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
71

72 73
As Auto DevOps relies on many different components, it's good to have a basic
knowledge of the following:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
74

75 76 77 78 79
- [Kubernetes](https://kubernetes.io/docs/home/)
- [Helm](https://docs.helm.sh/)
- [Docker](https://docs.docker.com)
- [GitLab Runner](https://docs.gitlab.com/runner/)
- [Prometheus](https://prometheus.io/docs/introduction/overview/)
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
80

81 82
Auto DevOps provides great defaults for all the stages; you can, however,
[customize](#customizing) almost everything to your needs.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
83

Marcia Ramos's avatar
Marcia Ramos committed
84 85
For an overview on the creation of Auto DevOps, read the blog post [From 2/3 of the Self-Hosted Git Market, to the Next-Generation CI System, to Auto DevOps](https://about.gitlab.com/2017/06/29/whats-next-for-gitlab-ci/).

86
## Requirements
87 88

To make full use of Auto DevOps, you will need:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
89

90 91
1. **GitLab Runner** (needed for all stages) - Your Runner needs to be
   configured to be able to run Docker. Generally this means using the
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
92 93 94 95 96 97 98 99 100 101
   [Docker](https://docs.gitlab.com/runner/executors/docker.html) or [Kubernetes
   executor](https://docs.gitlab.com/runner/executors/kubernetes.html), with
   [privileged mode enabled](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode).
   The Runners do not need to be installed in the Kubernetes cluster, but the
   Kubernetes executor is easy to use and is automatically autoscaling.
   Docker-based Runners can be configured to autoscale as well, using [Docker
   Machine](https://docs.gitlab.com/runner/install/autoscaling.html). Runners
   should be registered as [shared Runners](../../ci/runners/README.md#registering-a-shared-runner)
   for the entire GitLab instance, or [specific Runners](../../ci/runners/README.md#registering-a-specific-runner)
   that are assigned to specific projects.
102
1. **Base domain** (needed for Auto Review Apps and Auto Deploy) - You will need
103
   a domain configured with wildcard DNS which is going to be used by all of your
104 105
   Auto DevOps applications. [Read the specifics](#auto-devops-base-domain).
1. **Kubernetes** (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring) -
106 107
   To enable deployments, you will need Kubernetes 1.5+. You need a [Kubernetes cluster][kubernetes-clusters]
   for the project, or a Kubernetes [default service template](../../user/project/integrations/services_templates.md)
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
108
   for the entire GitLab installation.
109 110 111 112 113 114 115 116
    1. **A load balancer** - You can use NGINX ingress by deploying it to your
       Kubernetes cluster using the
       [`nginx-ingress`](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress)
       Helm chart.
1. **Prometheus** (needed for Auto Monitoring) - To enable Auto Monitoring, you
   will need Prometheus installed somewhere (inside or outside your cluster) and
   configured to scrape your Kubernetes cluster. To get response metrics
   (in addition to system metrics), you need to
117
   [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
118 119 120 121 122
   The [Prometheus service](../../user/project/integrations/prometheus.md)
   integration needs to be enabled for the project, or enabled as a
   [default service template](../../user/project/integrations/services_templates.md)
   for the entire GitLab installation.

123 124 125 126
NOTE: **Note:**
If you do not have Kubernetes or Prometheus installed, then Auto Review Apps,
Auto Deploy, and Auto Monitoring will be silently skipped.

127
## Auto DevOps base domain
128

129 130
NOTE: **Note**
`AUTO_DEVOPS_DOMAIN` environment variable is deprecated and
131
[is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959).
132

133
The Auto DevOps base domain is required if you want to make use of [Auto
134
Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It can be defined
135
in any of the following places:
136

137
- either under the cluster's settings, whether for [projects](../../user/project/clusters/index.md#base-domain) or [groups](../../user/group/clusters/index.md#base-domain)
138
- or in instance-wide settings in the **admin area > Settings** under the "Continuous Integration and Delivery" section
139 140 141 142
- or at the project level as a variable: `KUBE_INGRESS_BASE_DOMAIN`
- or at the group level as a variable: `KUBE_INGRESS_BASE_DOMAIN`.

NOTE: **Note**
143 144
The Auto DevOps base domain variable (`KUBE_INGRESS_BASE_DOMAIN`) follows the same order of precedence
as other environment [variables](../../ci/variables/README.md#priority-of-variables).
145 146

A wildcard DNS A record matching the base domain(s) is required, for example,
147 148 149 150 151 152
given a base domain of `example.com`, you'd need a DNS entry like:

```
*.example.com   3600     A     1.2.3.4
```

153
In this case, `example.com` is the domain name under which the deployed apps will be served,
154
and `1.2.3.4` is the IP address of your load balancer; generally NGINX
155
([see requirements](#requirements)). How to set up the DNS record is beyond
156 157
the scope of this document; you should check with your DNS provider.

158 159 160
Alternatively you can use free public services like [nip.io](http://nip.io)
which provide automatic wildcard DNS without any configuration. Just set the
Auto DevOps base domain to `1.2.3.4.nip.io`.
161

162 163
Once set up, all requests will hit the load balancer, which in turn will route
them to the Kubernetes pods that run your application(s).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
164

165 166 167 168
## Using multiple Kubernetes clusters **[PREMIUM]**

When using Auto DevOps, you may want to deploy different environments to
different Kubernetes clusters. This is possible due to the 1:1 connection that
169
[exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters-premium).
170

171
In the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml)
172 173 174 175 176 177 178 179 180
(used behind the scenes by Auto DevOps), there are currently 3 defined environment names that you need to know:

- `review/` (every environment starting with `review/`)
- `staging`
- `production`

Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so
except for the environment scope, they would also need to have a different
domain they would be deployed to. This is why you need to define a separate
181
`KUBE_INGRESS_BASE_DOMAIN` variable for all the above
182
[based on the environment](https://docs.gitlab.com/ee/ci/variables/index.html#limiting-environment-scopes-of-variables-premium).
183 184 185 186

The following table is an example of how the three different clusters would
be configured.

187
| Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes |
188 189 190
| ------------ | -------------- | ----------------------------- | ------------- | ------ |
| review       |  `review/*`    | `review.example.com`  | `review/*`      | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. |
| staging      |  `staging`     | `staging.example.com` | `staging`       | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). |
191
| production   |  `production`  | `example.com`         | `production`    | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production-premium). |
192 193 194 195 196 197 198 199 200

To add a different cluster for each environment:

1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters
   with their respective environment scope as described from the table above.

    ![Auto DevOps multiple clusters](img/autodevops_multiple_clusters.png)

1. After the clusters are created, navigate to each one and install Helm Tiller
201
   and Ingress. Wait for the Ingress IP address to be assigned.
202 203
1. Make sure you have [configured your DNS](#auto-devops-base-domain) with the
   specified Auto DevOps domains.
204 205
1. Navigate to each cluster's page, through **Operations > Kubernetes**,
   and add the domain based on its Ingress IP address.
206 207 208 209 210 211

Now that all is configured, you can test your setup by creating a merge request
and verifying that your app is deployed as a review app in the Kubernetes
cluster with the `review/*` environment scope. Similarly, you can check the
other environments.

212
NOTE: **Note:**
213
From GitLab 11.8, `KUBE_INGRESS_BASE_DOMAIN` replaces `AUTO_DEVOPS_DOMAIN`.
214
`AUTO_DEVOPS_DOMAIN` [is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959).
215

216
## Enabling/Disabling Auto DevOps
217

218 219
When first using Auto Devops, review the [requirements](#requirements) to ensure all necessary components to make
full use of Auto DevOps are available. If this is your fist time, we recommend you follow the
220
[quick start guide](quick_start_guide.md).
221

222 223 224 225 226 227 228 229 230 231 232 233 234 235 236
GitLab.com users can enable/disable Auto DevOps at the project-level only. Self-managed users
can enable/disable Auto DevOps at either the project-level or instance-level.

### Enabling/disabling Auto DevOps at the instance-level (Administrators only)

1. Go to **Admin area > Settings > Continuous Integration and Deployment**.
1. Toggle the checkbox labeled **Default to Auto DevOps pipeline for all projects**.
1. If enabling, optionally set up the Auto DevOps [base domain](#auto-devops-base-domain) which will be used for Auto Deploy and Auto Review Apps.
1. Click **Save changes** for the changes to take effect.

NOTE: **Note:**
Even when disabled at the instance level, project maintainers are still able to enable
Auto DevOps at the project level.

### Enabling/disabling Auto DevOps at the project-level
Fabio Busatto's avatar
Fabio Busatto committed
237

238 239
If enabling, check that your project doesn't have a `.gitlab-ci.yml`, or if one exists, remove it.

240
1. Go to your project's **Settings > CI/CD > Auto DevOps**.
241 242
1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable)
1. When enabling, it's optional but recommended to add in the [base domain](#auto-devops-base-domain)
243 244 245
   that will be used by Auto DevOps to [deploy your application](#auto-deploy)
   and choose the [deployment strategy](#deployment-strategy).
1. Click **Save changes** for the changes to take effect.
246

247
When the feature has been enabled, an Auto DevOps pipeline is triggered on the default branch.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
248

249
NOTE: **Note:**
Fabio Busatto's avatar
Fabio Busatto committed
250 251 252 253
For GitLab versions 10.0 - 10.2, when enabling Auto DevOps, a pipeline needs to be
manually triggered either by pushing a new commit to the repository or by visiting
`https://example.gitlab.com/<username>/<project>/pipelines/new` and creating
a new pipeline for your default branch, generally `master`.
254

255 256 257 258 259
NOTE: **Note:**
There is also a feature flag to enable Auto DevOps to a percentage of projects
which can be enabled from the console with
`Feature.get(:force_autodevops_on_by_default).enable_percentage_of_actors(10)`.

260
### Deployment strategy
Matija Čupić's avatar
Matija Čupić committed
261

262 263 264 265
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/38542) in GitLab 11.0.

You can change the deployment strategy used by Auto DevOps by going to your
project's **Settings > CI/CD > Auto DevOps**.
Matija Čupić's avatar
Matija Čupić committed
266 267 268

The available options are:

269 270 271
- **Continuous deployment to production**: Enables [Auto Deploy](#auto-deploy)
  with `master` branch directly deployed to production.
- **Continuous deployment to production using timed incremental rollout**: Sets the
272
  [`INCREMENTAL_ROLLOUT_MODE`](#timed-incremental-rollout-to-production-premium) variable
273 274 275
  to `timed`, and production deployment will be executed with a 5 minute delay between
  each increment in rollout.
- **Automatic deployment to staging, manual deployment to production**: Sets the
276
  [`STAGING_ENABLED`](#deploy-policy-for-staging-and-production-environments) and
277
  [`INCREMENTAL_ROLLOUT_MODE`](#incremental-rollout-to-production-premium) variables
278 279 280 281
  to `1` and `manual`. This means:

  - `master` branch is directly deployed to staging.
  - Manual actions are provided for incremental rollout to production.
Matija Čupić's avatar
Matija Čupić committed
282

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
283 284
## Stages of Auto DevOps

285 286
The following sections describe the stages of Auto DevOps. Read them carefully
to understand how each one works.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
287 288 289

### Auto Build

290 291
Auto Build creates a build of the application using an existing `Dockerfile` or
Heroku buildpacks.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
292 293

Either way, the resulting Docker image is automatically pushed to the
294 295
[Container Registry][container-registry] and tagged with the commit SHA.

296 297 298 299 300
#### Auto Build using a Dockerfile

If a project's repository contains a `Dockerfile`, Auto Build will use
`docker build` to create a Docker image.

301 302 303
If you are also using Auto Review Apps and Auto Deploy and choose to provide
your own `Dockerfile`, make sure you expose your application to port
`5000` as this is the port assumed by the default Helm chart.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
304

305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327
#### Auto Build using Heroku buildpacks

Auto Build builds an application using a project's `Dockerfile` if present, or
otherwise it will use [Herokuish](https://github.com/gliderlabs/herokuish)
and [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks)
to automatically detect and build the application into a Docker image.

Each buildpack requires certain files to be in your project's repository for
Auto Build to successfully build your application. For example, the following
files are required at the root of your application's repository, depending on
the language:

- A `Pipfile` or `requirements.txt` file for Python projects.
- A `Gemfile` or `Gemfile.lock` file for Ruby projects.

For the requirements of other languages and frameworks, read the
[buildpacks docs](https://devcenter.heroku.com/articles/buildpacks#officially-supported-buildpacks).

TIP: **Tip:**
If Auto Build fails despite the project meeting the buildpack requirements, set
a project variable `TRACE=true` to enable verbose logging, which may help to
troubleshoot.

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
328 329
### Auto Test

330
Auto Test automatically runs the appropriate tests for your application using
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
331
[Herokuish](https://github.com/gliderlabs/herokuish) and [Heroku
332 333 334 335 336
buildpacks](https://devcenter.heroku.com/articles/buildpacks) by analyzing
your project to detect the language and framework. Several languages and
frameworks are detected automatically, but if your language is not detected,
you may succeed with a [custom buildpack](#custom-buildpacks). Check the
[currently supported languages](#currently-supported-languages).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
337

338
NOTE: **Note:**
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
339 340 341
Auto Test uses tests you already have in your application. If there are no
tests, it's up to you to add them.

342
### Auto Code Quality **[STARTER]**
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
343

344 345
Auto Code Quality uses the
[Code Quality image](https://gitlab.com/gitlab-org/security-products/codequality) to run
346 347
static analysis and other code checks on the current code. The report is
created, and is uploaded as an artifact which you can later download and check
348 349
out.

350
Any differences between the source and target branches are also
351
[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
352

353
### Auto SAST **[ULTIMATE]**
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
354

355
> Introduced in [GitLab Ultimate][ee] 10.3.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
356 357

Static Application Security Testing (SAST) uses the
358
[SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) to run static
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
359 360 361 362
analysis on the current code and checks for potential security issues. Once the
report is created, it's uploaded as an artifact which you can later download and
check out.

363
Any security warnings are also
364
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/sast.html).
365

366 367 368
NOTE: **Note:**
The Auto SAST stage will be skipped on licenses other than Ultimate.

Fabio Busatto's avatar
Fabio Busatto committed
369 370 371
NOTE: **Note:**
The Auto SAST job requires GitLab Runner 11.5 or above.

372
### Auto Dependency Scanning **[ULTIMATE]**
373 374 375 376 377 378 379 380 381

> Introduced in [GitLab Ultimate][ee] 10.7.

Dependency Scanning uses the
[Dependency Scanning Docker image](https://gitlab.com/gitlab-org/security-products/dependency-scanning)
to run analysis on the project dependencies and checks for potential security issues. Once the
report is created, it's uploaded as an artifact which you can later download and
check out.

382
Any security warnings are also
383 384
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dependency_scanning.html).

385 386 387
NOTE: **Note:**
The Auto Dependency Scanning stage will be skipped on licenses other than Ultimate.

388 389 390
NOTE: **Note:**
The Auto Dependency Scanning job requires GitLab Runner 11.5 or above.

391 392 393 394 395
### Auto License Management **[ULTIMATE]**

> Introduced in [GitLab Ultimate][ee] 11.0.

License Management uses the
396
[License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management)
397 398 399 400
to search the project dependencies for their license. Once the
report is created, it's uploaded as an artifact which you can later download and
check out.

401
Any licenses are also
402
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/license_management.html).
403

404 405 406
NOTE: **Note:**
The Auto License Management stage will be skipped on licenses other than Ultimate.

407
### Auto Container Scanning
408 409 410 411 412 413 414 415 416

> Introduced in GitLab 10.4.

Vulnerability Static Analysis for containers uses
[Clair](https://github.com/coreos/clair) to run static analysis on a
Docker image and checks for potential security issues. Once the report is
created, it's uploaded as an artifact which you can later download and
check out.

417
Any security warnings are also
418
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/container_scanning.html).
419

420 421 422
NOTE: **Note:**
The Auto Container Scanning stage will be skipped on licenses other than Ultimate.

423 424 425 426
### Auto Review Apps

NOTE: **Note:**
This is an optional step, since many projects do not have a Kubernetes cluster
427
available. If the [requirements](#requirements) are not met, the job will
428 429 430 431 432 433 434 435 436 437 438 439 440 441
silently be skipped.

CAUTION: **Caution:**
Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.)
This can cause confusion with Helm not detecting the change, and subsequent
deploys with Auto DevOps can undo your changes. Also, if you change something
and want to undo it by deploying again, Helm may not detect that anything changed
in the first place, and thus not realize that it needs to re-apply the old config.

[Review Apps][review-app] are temporary application environments based on the
branch's code so developers, designers, QA, product managers, and other
reviewers can actually see and interact with code changes as part of the review
process. Auto Review Apps create a Review App for each branch.

442 443 444
Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster
is available, no deployment will occur.

445 446 447 448 449 450 451
The Review App will have a unique URL based on the project name, the branch
name, and a unique number, combined with the Auto DevOps base domain. For
example, `user-project-branch-1234.example.com`. A link to the Review App shows
up in the merge request widget for easy discovery. When the branch is deleted,
for example after the merge request is merged, the Review App will automatically
be deleted.

452
### Auto DAST **[ULTIMATE]**
453

454
> Introduced in [GitLab Ultimate][ee] 10.4.
455 456 457 458 459 460 461

Dynamic Application Security Testing (DAST) uses the
popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy)
to perform an analysis on the current code and checks for potential security
issues. Once the report is created, it's uploaded as an artifact which you can
later download and check out.

462
Any security warnings are also
463
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dast.html).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
464

465 466 467
NOTE: **Note:**
The Auto DAST stage will be skipped on licenses other than Ultimate.

468
### Auto Browser Performance Testing **[PREMIUM]**
Joshua Lambert's avatar
Joshua Lambert committed
469

470
> Introduced in [GitLab Premium][ee] 10.4.
Joshua Lambert's avatar
Joshua Lambert committed
471 472 473 474 475 476 477 478 479

Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) to measure the performance of a web page. A JSON report is created and uploaded as an artifact, which includes the overall performance score for each page. By default, the root page of Review and Production environments will be tested. If you would like to add additional URL's to test, simply add the paths to a file named `.gitlab-urls.txt` in the root directory, one per line. For example:

```
/
/features
/direction
```

480 481
Any performance differences between the source and target branches are also
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html).
Joshua Lambert's avatar
Joshua Lambert committed
482

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
483 484
### Auto Deploy

485
NOTE: **Note:**
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
486
This is an optional step, since many projects do not have a Kubernetes cluster
487
available. If the [requirements](#requirements) are not met, the job will
488 489 490 491 492 493 494 495 496 497 498 499 500
silently be skipped.

CAUTION: **Caution:**
Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.)
This can cause confusion with Helm not detecting the change, and subsequent
deploys with Auto DevOps can undo your changes. Also, if you change something
and want to undo it by deploying again, Helm may not detect that anything changed
in the first place, and thus not realize that it needs to re-apply the old config.

After a branch or merge request is merged into the project's default branch (usually
`master`), Auto Deploy deploys the application to a `production` environment in
the Kubernetes cluster, with a namespace based on the project name and unique
project ID, for example `project-4321`.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
501 502

Auto Deploy doesn't include deployments to staging or canary by default, but the
503
[Auto DevOps template] contains job definitions for these tasks if you want to
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
504 505
enable them.

506
You can make use of [environment variables](#environment-variables) to automatically
507 508
scale your pod replicas.

509 510 511 512 513 514 515 516 517 518 519 520
It's important to note that when a project is deployed to a Kubernetes cluster,
it relies on a Docker image that has been pushed to the
[GitLab Container Registry](../../user/project/container_registry.md). Kubernetes
fetches this image and uses it to run the application. If the project is public,
the image can be accessed by Kubernetes without any authentication, allowing us
to have deployments more usable. If the project is private/internal, the
Registry requires credentials to pull the image. Currently, this is addressed
by providing `CI_JOB_TOKEN` as the password that can be used, but this token will
no longer be valid as soon as the deployment job finishes. This means that
Kubernetes can run the application, but in case it should be restarted or
executed somewhere else, it cannot be accessed again.

521 522
#### Migrations

523 524
> [Introduced][ce-21955] in GitLab 11.4

525 526
Database initialization and migrations for PostgreSQL can be configured to run
within the application pod by setting the project variables `DB_INITIALIZE` and
527 528
`DB_MIGRATE` respectively.

529 530 531 532 533 534 535 536 537
If present, `DB_INITIALIZE` will be run as a shell command within an
application pod as a helm post-install hook. As some applications will
not run without a successful database initialization step, GitLab will
deploy the first release without the application deployment and only the
database initialization step. After the database initialization completes,
GitLab will deploy a second release with the application deployment as
normal.

Note that a post-install hook means that if any deploy succeeds,
538 539 540 541 542
`DB_INITIALIZE` will not be processed thereafter.

If present, `DB_MIGRATE` will be run as a shell command within an application pod as
a helm pre-upgrade hook.

543
For example, in a Rails application:
544

545
- `DB_INITIALIZE` can be set to `cd /app && RAILS_ENV=production
546
  bin/setup`
547
- `DB_MIGRATE` can be set to `cd /app && RAILS_ENV=production bin/update`
548

549 550 551
NOTE: **Note:**
The `/app` path is the directory of your project inside the docker image
as [configured by
552 553
Herokuish](https://github.com/gliderlabs/herokuish#paths)

554 555
> [Introduced][ce-19507] in GitLab 11.0.

556
For internal and private projects a [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token)
557 558 559 560 561 562
will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved. This Deploy Token
can be used for permanent access to the registry.

Note: **Note**
When the GitLab Deploy Token has been manually revoked, it won't be automatically created.

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
563 564
### Auto Monitoring

565
NOTE: **Note:**
566
Check the [requirements](#requirements) for Auto Monitoring to make this stage
567 568
work.

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
569 570 571 572 573 574
Once your application is deployed, Auto Monitoring makes it possible to monitor
your application's server and response metrics right out of the box. Auto
Monitoring uses [Prometheus](../../user/project/integrations/prometheus.md) to
get system metrics such as CPU and memory usage directly from
[Kubernetes](../../user/project/integrations/prometheus_library/kubernetes.md),
and response metrics such as HTTP error rates, latency, and throughput from the
575
[NGINX server](../../user/project/integrations/prometheus_library/nginx_ingress.md).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
576

577
The metrics include:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
578

579 580
- **Response Metrics:** latency, throughput, error rate
- **System Metrics:** CPU utilization, memory utilization
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
581

Dylan Griffith's avatar
Dylan Griffith committed
582
In order to make use of monitoring you need to:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
583

584
1. [Deploy Prometheus](../../user/project/integrations/prometheus.md) into your Kubernetes cluster
585 586
1. If you would like response metrics, ensure you are running at least version
   0.9.0 of NGINX Ingress and
587
   [enable Prometheus metrics](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/custom-vts-metrics-prometheus/nginx-vts-metrics-conf.yaml).
588 589 590
1. Finally, [annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
   the NGINX Ingress deployment to be scraped by Prometheus using
   `prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
591

592 593
To view the metrics, open the
[Monitoring dashboard for a deployed environment](../../ci/environments.md#monitoring-environments).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
594

595
![Auto Metrics](img/auto_monitoring.png)
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
596

597
## Customizing
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
598

599 600 601
While Auto DevOps provides great defaults to get you started, you can customize
almost everything to fit your needs; from custom [buildpacks](#custom-buildpacks),
to [`Dockerfile`s](#custom-dockerfile), [Helm charts](#custom-helm-chart), or
602
even copying the complete [CI/CD configuration](#customizing-gitlab-ciyml)
603
into your project to enable staging and canary deployments, and more.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
604

605
### Custom buildpacks
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
606 607

If the automatic buildpack detection fails for your project, or if you want to
608 609
use a custom buildpack, you can override the buildpack(s) using a project variable
or a `.buildpacks` file in your project:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
610 611 612

- **Project variable** - Create a project variable `BUILDPACK_URL` with the URL
  of the buildpack to use.
613
- **`.buildpacks` file** - Add a file in your project's repo called  `.buildpacks`
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
614
  and add the URL of the buildpack to use on a line in the file. If you want to
615
  use multiple buildpacks, you can enter them in, one on each line.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
616

617 618
CAUTION: **Caution:**
Using multiple buildpacks isn't yet supported by Auto DevOps.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
619 620 621 622 623 624 625 626 627 628

### Custom `Dockerfile`

If your project has a `Dockerfile` in the root of the project repo, Auto DevOps
will build a Docker image based on the Dockerfile rather than using buildpacks.
This can be much faster and result in smaller images, especially if your
Dockerfile is based on [Alpine](https://hub.docker.com/_/alpine/).

### Custom Helm Chart

629 630 631
Auto DevOps uses [Helm](https://helm.sh/) to deploy your application to Kubernetes.
You can override the Helm chart used by bundling up a chart into your project
repo or by specifying a project variable:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
632

633
- **Bundled chart** - If your project has a `./chart` directory with a `Chart.yaml`
634
  file in it, Auto DevOps will detect the chart and use it instead of the [default
635
  one](https://gitlab.com/charts/auto-deploy-app).
636
  This can be a great way to control exactly how your application is deployed.
Marcel Amirault's avatar
Marcel Amirault committed
637
- **Project variable** - Create a [project variable](../../ci/variables/README.md#variables)
638
  `AUTO_DEVOPS_CHART` with the URL of a custom chart to use or create two project variables `AUTO_DEVOPS_CHART_REPOSITORY` with the URL of a custom chart repository and `AUTO_DEVOPS_CHART` with the path to the chart.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
639

640 641 642 643 644
### Custom Helm chart per environment **[PREMIUM]**

You can specify the use of a custom Helm chart per environment by scoping the environment variable 
to the desired environment. See [Limiting environment scopes of variables](https://docs.gitlab.com/ee/ci/variables/#limiting-environment-scopes-of-variables-premium).

645
### Customizing `.gitlab-ci.yml`
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
646 647

If you want to modify the CI/CD pipeline used by Auto DevOps, you can copy the
648 649 650 651
[Auto DevOps template] into your project's repo and edit as you see fit.

Assuming that your project is new or it doesn't have a `.gitlab-ci.yml` file
present:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
652

653
1. From your project home page, either click on the "Set up CI/CD" button, or click
654 655 656 657 658
   on the plus button and (`+`), then "New file"
1. Pick `.gitlab-ci.yml` as the template type
1. Select "Auto-DevOps" from the template dropdown
1. Edit the template or add any jobs needed
1. Give an appropriate commit message and hit "Commit changes"
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
659

660 661 662
TIP: **Tip:** The Auto DevOps template includes useful comments to help you
customize it. For example, if you want deployments to go to a staging environment
instead of directly to a production one, you can enable the `staging` job by
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
663 664 665 666
renaming `.staging` to `staging`. Then make sure to uncomment the `when` key of
the `production` job to turn it into a manual action instead of deploying
automatically.

667 668 669 670 671 672 673 674 675 676 677 678 679 680 681
### PostgreSQL database support

In order to support applications that require a database,
[PostgreSQL][postgresql] is provisioned by default. The credentials to access
the database are preconfigured, but can be customized by setting the associated
[variables](#environment-variables). These credentials can be used for defining a
`DATABASE_URL` of the format:

```yaml
postgres://user:password@postgres-host:postgres-port/postgres-database
```

### Environment variables

The following variables can be used for setting up the Auto DevOps domain,
682
providing a custom Helm chart, or scaling your application. PostgreSQL can
683 684
also be customized, and you can easily use a [custom buildpack](#custom-buildpacks).

685 686
| **Variable**                 | **Description**                                                                                                                                                                                                               |
| ------------                 | ---------------                                                                                                                                                                                                               |
687
| `AUTO_DEVOPS_DOMAIN`         | The [Auto DevOps domain](#auto-devops-base-domain). By default, set automatically by the [Auto DevOps setting](#enablingdisabling-auto-devops). This variable is deprecated and [is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959). Use `KUBE_INGRESS_BASE_DOMAIN` instead. |
688
| `AUTO_DEVOPS_CHART`          | The Helm Chart used to deploy your apps; defaults to the one [provided by GitLab](https://gitlab.com/charts/auto-deploy-app).                                                             |
689
| `AUTO_DEVOPS_CHART_REPOSITORY` | The Helm Chart repository used to search for charts; defaults to `https://charts.gitlab.io`. |
690 691 692 693
| `REPLICAS`                   | The number of replicas to deploy; defaults to 1.                                                                                                                                                                              |
| `PRODUCTION_REPLICAS`        | The number of replicas to deploy in the production environment. This takes precedence over `REPLICAS`; defaults to 1.                                                                                                         |
| `CANARY_REPLICAS`            | The number of canary replicas to deploy for [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html); defaults to 1                                                                              |
| `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1  |
694 695
| `ADDITIONAL_HOSTS`           | Fully qualified domain names specified as a comma-separated list that are added to the ingress hosts.                                                                                                                         |
| `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`.                                      |
696 697 698
| `POSTGRES_ENABLED`           | Whether PostgreSQL is enabled; defaults to `"true"`. Set to `false` to disable the automatic deployment of PostgreSQL.                                                                                                        |
| `POSTGRES_USER`              | The PostgreSQL user; defaults to `user`. Set it to use a custom username.                                                                                                                                                     |
| `POSTGRES_PASSWORD`          | The PostgreSQL password; defaults to `testing-password`. Set it to use a custom password.                                                                                                                                     |
Evan Read's avatar
Evan Read committed
699
| `POSTGRES_DB`                | The PostgreSQL database name; defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/README.md#predefined-environment-variables). Set it to use a custom database name.                               |
700
| `POSTGRES_VERSION`           | The PostgreSQL version; defaults to `9.6.2` |
701
| `BUILDPACK_URL`              | The buildpack's full URL. It can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`, for example `https://github.com/heroku/heroku-buildpack-ruby.git#v142` |
702 703
| `SAST_CONFIDENCE_LEVEL`      | The minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High; defaults to `3`.|
| `DEP_SCAN_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled; defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).|
704 705
| `DB_INITIALIZE`              | From GitLab 11.4, this variable can be used to specify the command to run to initialize the application's PostgreSQL database. It runs inside the application pod. |
| `DB_MIGRATE`                 | From GitLab 11.4, this variable can be used to specify the command to run to migrate the application's PostgreSQL database. It runs inside the application pod. |
706
| `STAGING_ENABLED`            | From GitLab 10.8, this variable can be used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). |
707 708
| `CANARY_ENABLED`             | From GitLab 11.0, this variable can be used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments-premium). |
| `INCREMENTAL_ROLLOUT_MODE`| From GitLab 11.4, this variable, if present, can be used to enable an [incremental rollout](#incremental-rollout-to-production-premium) of your application for the production environment.<br/>Set to: <ul><li>`manual`, for manual deployment jobs.</li><li>`timed`, for automatic rollout deployments with a 5 minute delay each one.</li></ul> |
709
| `TEST_DISABLED`              | From GitLab 11.0, this variable can be used to disable the `test` job. If the variable is present, the job will not be created. |
710
| `CODE_QUALITY_DISABLED`       | From GitLab 11.0, this variable can be used to disable the `codequality` job. If the variable is present, the job will not be created. |
711 712
| `SAST_DISABLED`              | From GitLab 11.0, this variable can be used to disable the `sast` job. If the variable is present, the job will not be created. |
| `DEPENDENCY_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dependency_scanning` job. If the variable is present, the job will not be created. |
713
| `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast:container` job. If the variable is present, the job will not be created. |
714 715 716
| `REVIEW_DISABLED`            | From GitLab 11.0, this variable can be used to disable the `review` and the manual `review:stop` job. If the variable is present, these jobs will not be created. |
| `DAST_DISABLED`              | From GitLab 11.0, this variable can be used to disable the `dast` job. If the variable is present, the job will not be created. |
| `PERFORMANCE_DISABLED`       | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. |
717
| `K8S_SECRET_*`               | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) will be made available by Auto DevOps as environment variables to the deployed application. |
718
| `KUBE_INGRESS_BASE_DOMAIN`   | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/index.md#base-domain) for more information. |
719 720 721

TIP: **Tip:**
Set up the replica variables using a
Marcel Amirault's avatar
Marcel Amirault committed
722
[project variable](../../ci/variables/README.md#variables)
723 724 725 726 727 728 729
and scale your application by just redeploying it!

CAUTION: **Caution:**
You should *not* scale your application using Kubernetes directly. This can
cause confusion with Helm not detecting the change, and subsequent deploys with
Auto DevOps can undo your changes.

730
#### Application secret variables
731 732 733 734

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/49056) in GitLab 11.7.

Some applications need to define secret variables that are
735
accessible by the deployed application. Auto DevOps detects variables where the key starts with
736 737 738 739 740 741
`K8S_SECRET_` and make these prefixed variables available to the
deployed application, as environment variables.

To configure your application variables:

1. Go to your project's **Settings > CI/CD**, then expand the section
742
   called **Variables**.
743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773

2. Create a CI Variable, ensuring the key is prefixed with
   `K8S_SECRET_`. For example, you can create a variable with key
`K8S_SECRET_RAILS_MASTER_KEY`.

3. Run an Auto Devops pipeline either by manually creating a new
   pipeline or by pushing a code change to GitLab.

Auto DevOps pipelines will take your application secret variables to
populate a Kubernetes secret. This secret is unique per environment.
When deploying your application, the secret is loaded as environment
variables in the container running the application. Following the
example above, you can see the secret below containing the
`RAILS_MASTER_KEY` variable.

```sh
$ kubectl get secret production-secret -n minimal-ruby-app-54 -o yaml
apiVersion: v1
data:
  RAILS_MASTER_KEY: MTIzNC10ZXN0
kind: Secret
metadata:
  creationTimestamp: 2018-12-20T01:48:26Z
  name: production-secret
  namespace: minimal-ruby-app-54
  resourceVersion: "429422"
  selfLink: /api/v1/namespaces/minimal-ruby-app-54/secrets/production-secret
  uid: 57ac2bfd-03f9-11e9-b812-42010a9400e4
type: Opaque
```

774
CAUTION: **Caution:**
775
Variables with multiline values are not currently supported due to
776 777
limitations with the current Auto DevOps scripting environment.

778 779 780 781 782
NOTE: **Note:**
Environment variables are generally considered immutable in a Kubernetes
pod. Therefore, if you update an application secret without changing any
code then manually create a new pipeline, you will find that any running
application pods will not have the updated secrets. In this case, you
783 784
can either push a code update to GitLab to force the Kubernetes
Deployment to recreate pods or manually delete running pods to
785 786
cause Kubernetes to create new pods with updated secrets.

787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805
#### Advanced replica variables setup

Apart from the two replica-related variables for production mentioned above,
you can also use others for different environments.

There's a very specific mapping between Kubernetes' label named `track`,
GitLab CI/CD environment names, and the replicas environment variable.
The general rule is: `TRACK_ENV_REPLICAS`. Where:

- `TRACK`: The capitalized value of the `track`
  [Kubernetes label](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
  in the Helm Chart app definition. If not set, it will not be taken into account
  to the variable name.
- `ENV`: The capitalized environment name of the deploy job that is set in
  `.gitlab-ci.yml`.

That way, you can define your own `TRACK_ENV_REPLICAS` variables with which
you will be able to scale the pod's replicas easily.

806 807 808
In the example below, the environment's name is `qa` and it deploys the track
`foo` which would result in looking for the `FOO_QA_REPLICAS` environment
variable:
809 810 811 812 813 814 815

```yaml
QA testing:
  stage: deploy
  environment:
    name: qa
  script:
816
  - deploy foo
817 818
```

819 820
The track `foo` being referenced would also need to be defined in the
application's Helm chart, like:
821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841

```yaml
replicaCount: 1
image:
  repository: gitlab.example.com/group/project
  tag: stable
  pullPolicy: Always
  secrets:
    - name: gitlab-registry
application:
  track: foo
  tier: web
service:
  enabled: true
  name: web
  type: ClusterIP
  url: http://my.host.com/
  externalPort: 5000
  internalPort: 5000
```

842 843 844 845 846
#### Deploy policy for staging and production environments

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/160)
in GitLab 10.8.

847 848 849
TIP: **Tip:**
You can also set this inside your [project's settings](#deployment-strategy).

850 851 852 853 854 855 856
The normal behavior of Auto DevOps is to use Continuous Deployment, pushing
automatically to the `production` environment every time a new pipeline is run
on the default branch. However, there are cases where you might want to use a
staging environment and deploy to production manually. For this scenario, the
`STAGING_ENABLED` environment variable was introduced.

If `STAGING_ENABLED` is defined in your project (e.g., set `STAGING_ENABLED` to
Marcel Amirault's avatar
Marcel Amirault committed
857
`1` as a CI/CD variable), then the application will be automatically deployed
858 859 860
to a `staging` environment, and a  `production_manual` job will be created for
you when you're ready to manually deploy to production.

861 862 863 864 865 866 867 868 869
#### Deploy policy for canary environments **[PREMIUM]**

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/171)
in GitLab 11.0.

A [canary environment](https://docs.gitlab.com/ee/user/project/canary_deployments.html) can be used
before any changes are deployed to production.

If `CANARY_ENABLED` is defined in your project (e.g., set `CANARY_ENABLED` to
Marcel Amirault's avatar
Marcel Amirault committed
870
`1` as a CI/CD variable) then two manual jobs will be created:
871 872 873 874 875

- `canary` which will deploy the application to the canary environment
- `production_manual` which is to be used by you when you're ready to manually
  deploy to production.

876 877 878 879
#### Incremental rollout to production **[PREMIUM]**

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5415) in GitLab 10.8.

880 881 882
TIP: **Tip:**
You can also set this inside your [project's settings](#deployment-strategy).

883 884 885 886 887
When you have a new version of your app to deploy in production, you may want
to use an incremental rollout to replace just a few pods with the latest code.
This will allow you to first check how the app is behaving, and later manually
increasing the rollout up to 100%.

888 889
If `INCREMENTAL_ROLLOUT_MODE` is set to `manual` in your project, then instead
of the standard `production` job, 4 different
890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912
[manual jobs](../../ci/pipelines.md#manual-actions-from-the-pipeline-graph)
will be created:

1. `rollout 10%`
1. `rollout 25%`
1. `rollout 50%`
1. `rollout 100%`

The percentage is based on the `REPLICAS` variable and defines the number of
pods you want to have for your deployment. If you say `10`, and then you run
the `10%` rollout job, there will be `1` new pod + `9` old ones.

To start a job, click on the play icon next to the job's name. You are not
required to go from `10%` to `100%`, you can jump to whatever job you want.
You can also scale down by running a lower percentage job, just before hitting
`100%`. Once you get to `100%`, you cannot scale down, and you'd have to roll
back by redeploying the old version using the
[rollback button](../../ci/environments.md#rolling-back-changes) in the
environment page.

Below, you can see how the pipeline will look if the rollout or staging
variables are defined.

913 914 915 916 917
Without `INCREMENTAL_ROLLOUT_MODE` and without `STAGING_ENABLED`:

![Staging and rollout disabled](img/rollout_staging_disabled.png)

Without `INCREMENTAL_ROLLOUT_MODE` and with `STAGING_ENABLED`:
918

919
![Staging enabled](img/staging_enabled.png)
920

921
With `INCREMENTAL_ROLLOUT_MODE` set to `manual` and without `STAGING_ENABLED`:
922

923
![Rollout enabled](img/rollout_enabled.png)
924

925 926 927 928 929 930 931 932 933 934 935 936 937 938 939
With `INCREMENTAL_ROLLOUT_MODE` set to `manual` and with `STAGING_ENABLED`

![Rollout and staging enabled](img/rollout_staging_enabled.png)

CAUTION: **Caution:**
Before GitLab 11.4 this feature was enabled by the presence of the
`INCREMENTAL_ROLLOUT_ENABLED` environment variable.
This configuration is deprecated and will be removed in the future.

#### Timed incremental rollout to production **[PREMIUM]**

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7545) in GitLab 11.4.

TIP: **Tip:**
You can also set this inside your [project's settings](#deployment-strategy).
940

941
This configuration based on
942
[incremental rollout to production](#incremental-rollout-to-production-premium).
943

944
Everything behaves the same way, except:
945

946 947 948 949 950 951
- It's enabled by setting the `INCREMENTAL_ROLLOUT_MODE` variable to `timed`.
- Instead of the standard `production` job, the following jobs with a 5 minute delay between each are created:
    1. `timed rollout 10%`
    1. `timed rollout 25%`
    1. `timed rollout 50%`
    1. `timed rollout 100%`
952

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
953 954
## Currently supported languages

955
NOTE: **Note:**
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
956 957 958 959 960 961 962 963 964 965
Not all buildpacks support Auto Test yet, as it's a relatively new
enhancement. All of Heroku's [officially supported
languages](https://devcenter.heroku.com/articles/heroku-ci#currently-supported-languages)
support it, and some third-party buildpacks as well e.g., Go, Node, Java, PHP,
Python, Ruby, Gradle, Scala, and Elixir all support Auto Test, but notably the
multi-buildpack does not.

As of GitLab 10.0, the supported buildpacks are:

```
966 967 968 969 970 971 972 973 974 975 976 977 978
- heroku-buildpack-multi     v1.0.0
- heroku-buildpack-ruby      v168
- heroku-buildpack-nodejs    v99
- heroku-buildpack-clojure   v77
- heroku-buildpack-python    v99
- heroku-buildpack-java      v53
- heroku-buildpack-gradle    v23
- heroku-buildpack-scala     v78
- heroku-buildpack-play      v26
- heroku-buildpack-php       v122
- heroku-buildpack-go        v72
- heroku-buildpack-erlang    fa17af9
- buildpack-nginx            v8
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
979 980
```

981 982 983 984 985 986 987
## Limitations

The following restrictions apply.

### Private project support

CAUTION: **Caution:** Private project support in Auto DevOps is experimental.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed