README.md 6.09 KB
Newer Older
Mark Lapierre's avatar
Mark Lapierre committed
1
# GitLab QA - End-to-end tests for GitLab
2

Mark Lapierre's avatar
Mark Lapierre committed
3 4 5 6 7
This directory contains [end-to-end tests](doc/development/testing_guide/end_to_end_tests.md)
for GitLab. It includes the test framework and the tests themselves.

The tests can be found in `qa/specs/features` (not to be confused with the unit
tests for the test framework, which are in `spec/`).
8

9
It is part of the [GitLab QA project](https://gitlab.com/gitlab-org/gitlab-qa).
10

11
## What is it?
12

Mark Lapierre's avatar
Mark Lapierre committed
13
GitLab QA is an end-to-end tests suite for GitLab.
14

Mark Lapierre's avatar
Mark Lapierre committed
15
These are black-box and entirely click-driven end-to-end tests you can run
16 17 18 19 20 21
against any existing instance.

## How does it work?

1. When we release a new version of GitLab, we build a Docker images for it.
1. Along with GitLab Docker Images we also build and publish GitLab QA images.
Mark Lapierre's avatar
Mark Lapierre committed
22
1. GitLab QA project uses these images to execute end-to-end tests.
Stan Hu's avatar
Stan Hu committed
23

24 25 26 27 28 29 30 31 32
## Validating GitLab views / partials / selectors in merge requests

We recently added a new CI job that is going to be triggered for every push
event in CE and EE projects. The job is called `qa:selectors` and it will
verify coupling between page objects implemented as a part of GitLab QA
and corresponding views / partials / selectors in CE / EE.

Whenever `qa:selectors` job fails in your merge request, you are supposed to
fix [page objects](qa/page/README.md). You should also trigger end-to-end tests
Toon Claes's avatar
Toon Claes committed
33
using `package-and-qa` manual action, to test if everything works fine.
34

Stan Hu's avatar
Stan Hu committed
35 36 37
## How can I use it?

You can use GitLab QA to exercise tests on any live instance! For example, the
38
following call would login to a local [GDK] instance and run all specs in
Stan Hu's avatar
Stan Hu committed
39 40 41
`qa/specs/features`:

```
42
bin/qa Test::Instance::All http://localhost:3000
Stan Hu's avatar
Stan Hu committed
43 44
```

45 46 47
Note: If you want to run tests requiring SSH against GDK, you
will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md).

48 49 50
### Writing tests

1. [Using page objects](qa/page/README.md)
51
2. [Style guide](STYLE_GUIDE.md)
52

53 54 55
### Running specific tests

You can also supply specific tests to run as another parameter. For example, to
56
run the repository-related specs, you can execute:
Stan Hu's avatar
Stan Hu committed
57 58

```
59
bin/qa Test::Instance::All http://localhost -- qa/specs/features/browser_ui/3_create/repository
60 61 62 63 64 65
```

Since the arguments would be passed to `rspec`, you could use all `rspec`
options there. For example, passing `--backtrace` and also line number:

```
66
bin/qa Test::Instance::All http://localhost -- qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb:6 --backtrace
67 68
```

69 70 71
Note that the separator `--` is required; all subsequent options will be
ignored by the QA framework and passed to `rspec`.

72 73 74 75 76 77 78 79 80
### Overriding the authenticated user

Unless told otherwise, the QA tests will run as the default `root` user seeded
by the GDK.

If you need to authenticate as a different user, you can provide the
`GITLAB_USERNAME` and `GITLAB_PASSWORD` environment variables:

```
81
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance::All https://gitlab.example.com
Stan Hu's avatar
Stan Hu committed
82 83
```

84 85 86 87 88
If your user doesn't have permission to default sandbox group
`gitlab-qa-sandbox`, you could also use another sandbox group by giving
`GITLAB_SANDBOX_NAME`:

```
89
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
90 91
```

92
All [supported environment variables are here](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-environment-variables).
93

94 95 96 97 98 99
### Sending additional cookies

The environment variable `QA_COOKIES` can be set to send additional cookies
on every request. This is necessary on gitlab.com to direct traffic to the
canary fleet. To do this set `QA_COOKIES="gitlab_canary=true"`.

100
To set multiple cookies, separate them with the `;` character, for example: `QA_COOKIES="cookie1=value;cookie2=value2"`
101 102


103 104 105 106 107 108 109 110 111 112
### Building a Docker image to test

Once you have made changes to the CE/EE repositories, you may want to build a
Docker image to test locally instead of waiting for the `gitlab-ce-qa` or
`gitlab-ee-qa` nightly builds. To do that, you can run from this directory:

```sh
docker build -t gitlab/gitlab-ce-qa:nightly .
```

113
[GDK]: https://gitlab.com/gitlab-org/gitlab-development-kit/
114 115 116 117 118 119 120 121 122 123

### Quarantined tests

Tests can be put in quarantine by assigning `:quarantine` metadata. This means
they will be skipped unless run with `--tag quarantine`. This can be used for
tests that are expected to fail while a fix is in progress (similar to how
[`skip` or `pending`](https://relishapp.com/rspec/rspec-core/v/3-8/docs/pending-and-skipped-examples)
 can be used).

```
124
bin/qa Test::Instance::All http://localhost -- --tag quarantine
125 126 127 128 129 130 131 132 133 134
```

If `quarantine` is used with other tags, tests will only be run if they have at
least one of the tags other than `quarantine`. This is different from how RSpec
tags usually work, where all tags are inclusive.

For example, suppose one test has `:smoke` and `:quarantine` metadata, and
another test has `:ldap` and `:quarantine` metadata. If the tests are run with
`--tag smoke --tag quarantine`, only the first test will run. The test with
`:ldap` will not run even though it also has `:quarantine`.
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156

### Running tests with a feature flag enabled

Tests can be run with with a feature flag enabled by using the command-line
option `--enable-feature FEATURE_FLAG`. For example, to enable the feature flag
that enforces Gitaly request limits, you would use the command:

```
bin/qa Test::Instance::All http://localhost --enable-feature gitaly_enforce_requests_limits
```

This will instruct the QA framework to enable the `gitaly_enforce_requests_limits`
feature flag ([via the API](https://docs.gitlab.com/ee/api/features.html)), run
all the tests in the `Test::Instance::All` scenario, and then disable the
feature flag again.

Note: the QA framework doesn't currently allow you to easily toggle a feature
flag during a single test, [as you can in unit tests](https://docs.gitlab.com/ee/development/feature_flags.html#specs),
but [that capability is planned](https://gitlab.com/gitlab-org/quality/team-tasks/issues/77).

Note also that the `--` separator isn't used because `--enable-feature` is a QA
framework option, not an `rspec` option.