Commit c78861bc authored by Kamil Trzciński's avatar Kamil Trzciński

Allow to recursively expand includes

This change introduces a support for nesting the includes,
allowing to evaluate them in context of the target,
by properly respecting the relative inclusions and user permissions
of another projects, or templates.
parent c9ecc71a
---
title: Allow to recursively expand includes
merge_request: 24356
author:
type: added
...@@ -1594,6 +1594,9 @@ You can only use files that are currently tracked by Git on the same branch ...@@ -1594,6 +1594,9 @@ You can only use files that are currently tracked by Git on the same branch
your configuration file is on. In other words, when using a `include:local`, make your configuration file is on. In other words, when using a `include:local`, make
sure that both `.gitlab-ci.yml` and the local file are on the same branch. sure that both `.gitlab-ci.yml` and the local file are on the same branch.
All [nested includes](#nested-includes) will be executed in the scope of the same project,
so it is possible to use local, project, remote or template includes.
NOTE: **Note:** NOTE: **Note:**
Including local files through Git submodules paths is not supported. Including local files through Git submodules paths is not supported.
...@@ -1606,7 +1609,7 @@ include: ...@@ -1606,7 +1609,7 @@ include:
### `include:file` ### `include:file`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53903) in GitLab 11.7. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53903) in GitLab 11.9.
To include files from another private project under the same GitLab instance, To include files from another private project under the same GitLab instance,
use `include:file`. This file is referenced using full paths relative to the use `include:file`. This file is referenced using full paths relative to the
...@@ -1635,6 +1638,10 @@ include: ...@@ -1635,6 +1638,10 @@ include:
file: '/templates/.gitlab-ci-template.yml' file: '/templates/.gitlab-ci-template.yml'
``` ```
All nested includes will be executed in the scope of the target project,
so it is possible to used local (relative to target project), project, remote
or template includes.
### `include:template` ### `include:template`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53445) in GitLab 11.7. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53445) in GitLab 11.7.
...@@ -1650,6 +1657,9 @@ include: ...@@ -1650,6 +1657,9 @@ include:
- template: Auto-DevOps.gitlab-ci.yml - template: Auto-DevOps.gitlab-ci.yml
``` ```
All nested includes will be executed only with the permission of the user,
so it is possible to use project, remote or template includes.
### `include:remote` ### `include:remote`
`include:remote` can be used to include a file from a different location, `include:remote` can be used to include a file from a different location,
...@@ -1662,10 +1672,16 @@ include: ...@@ -1662,10 +1672,16 @@ include:
- remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml' - remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml'
``` ```
NOTE: **Note for GitLab admins:** All nested includes will be executed without context as public user, so only another remote,
In order to include files from another repository inside your local network, or public project, or template is allowed.
you may need to enable the **Allow requests to the local network from hooks and services** checkbox
located in the **Admin area > Settings > Network > Outbound requests** section. ### Nested includes
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53903) in GitLab 11.7.
Nested includes allow you to compose a set of includes.
A total of 50 includes is allowed.
Duplicate includes are considered a configuration error.
### `include` examples ### `include` examples
...@@ -1834,6 +1850,51 @@ In this case, if `install_dependencies` and `deploy` were not repeated in ...@@ -1834,6 +1850,51 @@ In this case, if `install_dependencies` and `deploy` were not repeated in
`.gitlab-ci.yml`, they would not be part of the script for the `production` `.gitlab-ci.yml`, they would not be part of the script for the `production`
job in the combined CI configuration. job in the combined CI configuration.
#### Using nested includes
The examples below show how includes can be nested from different sources
using a combination of different methods.
In this example, `.gitlab-ci.yml` includes local the file `/.gitlab-ci/another-config.yml`:
```yaml
includes:
- local: /.gitlab-ci/another-config.yml
```
The `/.gitlab-ci/another-config.yml` includes a template and the `/templates/docker-workflow.yml` file
from another project:
```yaml
includes:
- template: Bash.gitlab-ci.yml
- project: /group/my-project
file: /templates/docker-workflow.yml
```
The `/templates/docker-workflow.yml` present in `/group/my-project` includes two local files
of the `/group/my-project`:
```yaml
includes:
- local: : /templates/docker-build.yml
- local: : /templates/docker-testing.yml
```
Our `/templates/docker-build.yml` present in `/group/my-project` adds a `docker-build` job:
```yaml
docker-build:
script: docker build -t my-image .
```
Our second `/templates/docker-test.yml` present in `/group/my-project` adds a `docker-test` job:
```yaml
docker-test:
script: docker run my-image /run/tests.sh
```
## `extends` ## `extends`
> Introduced in GitLab 11.3. > Introduced in GitLab 11.3.
......
...@@ -84,7 +84,8 @@ module Gitlab ...@@ -84,7 +84,8 @@ module Gitlab
Config::External::Processor.new(config, Config::External::Processor.new(config,
project: project, project: project,
sha: sha || project.repository.root_ref_sha, sha: sha || project.repository.root_ref_sha,
user: user).perform user: user,
expandset: Set.new).perform
end end
end end
end end
......
...@@ -12,7 +12,7 @@ module Gitlab ...@@ -12,7 +12,7 @@ module Gitlab
YAML_WHITELIST_EXTENSION = /.+\.(yml|yaml)$/i.freeze YAML_WHITELIST_EXTENSION = /.+\.(yml|yaml)$/i.freeze
Context = Struct.new(:project, :sha, :user) Context = Struct.new(:project, :sha, :user, :expandset)
def initialize(params, context) def initialize(params, context)
@params = params @params = params
...@@ -43,13 +43,27 @@ module Gitlab ...@@ -43,13 +43,27 @@ module Gitlab
end end
def to_hash def to_hash
@hash ||= Gitlab::Config::Loader::Yaml.new(content).load! expanded_content_hash
rescue Gitlab::Config::Loader::FormatError
nil
end end
protected protected
def expanded_content_hash
return unless content_hash
strong_memoize(:expanded_content_yaml) do
expand_includes(content_hash)
end
end
def content_hash
strong_memoize(:content_yaml) do
Gitlab::Config::Loader::Yaml.new(content).load!
end
rescue Gitlab::Config::Loader::FormatError
nil
end
def validate! def validate!
validate_location! validate_location!
validate_content! if errors.none? validate_content! if errors.none?
...@@ -73,6 +87,14 @@ module Gitlab ...@@ -73,6 +87,14 @@ module Gitlab
errors.push("Included file `#{location}` does not have valid YAML syntax!") errors.push("Included file `#{location}` does not have valid YAML syntax!")
end end
end end
def expand_includes(hash)
External::Processor.new(hash, **expand_context).perform
end
def expand_context
{ project: nil, sha: nil, user: nil, expandset: context.expandset }
end
end end
end end
end end
......
...@@ -31,6 +31,13 @@ module Gitlab ...@@ -31,6 +31,13 @@ module Gitlab
def fetch_local_content def fetch_local_content
context.project.repository.blob_data_at(context.sha, location) context.project.repository.blob_data_at(context.sha, location)
end end
def expand_context
super.merge(
project: context.project,
sha: context.sha,
user: context.user)
end
end end
end end
end end
......
...@@ -64,6 +64,13 @@ module Gitlab ...@@ -64,6 +64,13 @@ module Gitlab
project.commit(ref_name).try(:sha) project.commit(ref_name).try(:sha)
end end
end end
def expand_context
super.merge(
project: project,
sha: sha,
user: context.user)
end
end end
end end
end end
......
...@@ -7,6 +7,8 @@ module Gitlab ...@@ -7,6 +7,8 @@ module Gitlab
class Mapper class Mapper
include Gitlab::Utils::StrongMemoize include Gitlab::Utils::StrongMemoize
MAX_INCLUDES = 50
FILE_CLASSES = [ FILE_CLASSES = [
External::File::Remote, External::File::Remote,
External::File::Template, External::File::Template,
...@@ -14,25 +16,34 @@ module Gitlab ...@@ -14,25 +16,34 @@ module Gitlab
External::File::Project External::File::Project
].freeze ].freeze
AmbigiousSpecificationError = Class.new(StandardError) Error = Class.new(StandardError)
AmbigiousSpecificationError = Class.new(Error)
DuplicateIncludesError = Class.new(Error)
TooManyIncludesError = Class.new(Error)
def initialize(values, project:, sha:, user:, expandset:)
raise Error, 'Expanded needs to be `Set`' unless expandset.is_a?(Set)
def initialize(values, project:, sha:, user:)
@locations = Array.wrap(values.fetch(:include, [])) @locations = Array.wrap(values.fetch(:include, []))
@project = project @project = project
@sha = sha @sha = sha
@user = user @user = user
@expandset = expandset
end end
def process def process
return [] if locations.empty?
locations locations
.compact .compact
.map(&method(:normalize_location)) .map(&method(:normalize_location))
.each(&method(:verify_duplicates!))
.map(&method(:select_first_matching)) .map(&method(:select_first_matching))
end end
private private
attr_reader :locations, :project, :sha, :user attr_reader :locations, :project, :sha, :user, :expandset
# convert location if String to canonical form # convert location if String to canonical form
def normalize_location(location) def normalize_location(location)
...@@ -51,6 +62,23 @@ module Gitlab ...@@ -51,6 +62,23 @@ module Gitlab
end end
end end
def verify_duplicates!(location)
if expandset.count >= MAX_INCLUDES
raise TooManyIncludesError, "Maximum of #{MAX_INCLUDES} nested includes are allowed!"
end
# We scope location to context, as this allows us to properly support
# relative incldues, and similarly looking relative in another project
# does not trigger duplicate error
scoped_location = location.merge(
context_project: project,
context_sha: sha)
unless expandset.add?(scoped_location)
raise DuplicateIncludesError, "Include `#{location.to_json}` was already included!"
end
end
def select_first_matching(location) def select_first_matching(location)
matching = FILE_CLASSES.map do |file_class| matching = FILE_CLASSES.map do |file_class|
file_class.new(location, context) file_class.new(location, context)
...@@ -63,7 +91,7 @@ module Gitlab ...@@ -63,7 +91,7 @@ module Gitlab
def context def context
strong_memoize(:context) do strong_memoize(:context) do
External::File::Base::Context.new(project, sha, user) External::File::Base::Context.new(project, sha, user, expandset)
end end
end end
end end
......
...@@ -7,11 +7,11 @@ module Gitlab ...@@ -7,11 +7,11 @@ module Gitlab
class Processor class Processor
IncludeError = Class.new(StandardError) IncludeError = Class.new(StandardError)
def initialize(values, project:, sha:, user:) def initialize(values, project:, sha:, user:, expandset:)
@values = values @values = values
@external_files = External::Mapper.new(values, project: project, sha: sha, user: user).process @external_files = External::Mapper.new(values, project: project, sha: sha, user: user, expandset: expandset).process
@content = {} @content = {}
rescue External::Mapper::AmbigiousSpecificationError => e rescue External::Mapper::Error => e
raise IncludeError, e.message raise IncludeError, e.message
end end
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
require 'fast_spec_helper' require 'fast_spec_helper'
describe Gitlab::Ci::Config::External::File::Base do describe Gitlab::Ci::Config::External::File::Base do
let(:context) { described_class::Context.new(nil, 'HEAD', nil) } let(:context) { described_class::Context.new(nil, 'HEAD', nil, Set.new) }
let(:test_class) do let(:test_class) do
Class.new(described_class) do Class.new(described_class) do
...@@ -79,4 +79,20 @@ describe Gitlab::Ci::Config::External::File::Base do ...@@ -79,4 +79,20 @@ describe Gitlab::Ci::Config::External::File::Base do
end end
end end
end end
describe '#to_hash' do
context 'with includes' do
let(:location) { 'some/file/config.yml' }
let(:content) { 'include: { template: Bash.gitlab-ci.yml }'}
before do
allow_any_instance_of(test_class)
.to receive(:content).and_return(content)
end
it 'does expand hash to include the template' do
expect(subject.to_hash).to include(:before_script)
end
end
end
end end
...@@ -4,8 +4,10 @@ require 'spec_helper' ...@@ -4,8 +4,10 @@ require 'spec_helper'
describe Gitlab::Ci::Config::External::File::Local do describe Gitlab::Ci::Config::External::File::Local do
set(:project) { create(:project, :repository) } set(:project) { create(:project, :repository) }
set(:user) { create(:user) }
let(:context) { described_class::Context.new(project, '12345', nil) } let(:sha) { '12345' }
let(:context) { described_class::Context.new(project, sha, user, Set.new) }
let(:params) { { local: location } } let(:params) { { local: location } }
let(:local_file) { described_class.new(params, context) } let(:local_file) { described_class.new(params, context) }
...@@ -103,4 +105,36 @@ describe Gitlab::Ci::Config::External::File::Local do ...@@ -103,4 +105,36 @@ describe Gitlab::Ci::Config::External::File::Local do
expect(local_file.error_message).to eq("Local file `#{location}` does not exist!") expect(local_file.error_message).to eq("Local file `#{location}` does not exist!")
end end
end end
describe '#expand_context' do
let(:location) { 'location.yml' }
subject { local_file.send(:expand_context) }
it 'inherits project, user and sha' do
is_expected.to include(user: user, project: project, sha: sha)
end
end
describe '#to_hash' do
context 'properly includes another local file in the same repository' do
let(:location) { 'some/file/config.yml' }
let(:content) { 'include: { local: another-config.yml }'}
let(:another_location) { 'another-config.yml' }
let(:another_content) { 'rspec: JOB' }
before do
allow(project.repository).to receive(:blob_data_at).with(sha, location)
.and_return(content)
allow(project.repository).to receive(:blob_data_at).with(sha, another_location)
.and_return(another_content)
end
it 'does expand hash to include the template' do
expect(local_file.to_hash).to include(:rspec)
end
end
end
end end
...@@ -3,12 +3,13 @@ ...@@ -3,12 +3,13 @@
require 'spec_helper' require 'spec_helper'
describe Gitlab::Ci::Config::External::File::Project do describe Gitlab::Ci::Config::External::File::Project do
set(:context_project) { create(:project) }
set(:project) { create(:project, :repository) } set(:project) { create(:project, :repository) }
set(:user) { create(:user) } set(:user) { create(:user) }
let(:context_user) { user } let(:context_user) { user }
let(:context) { described_class::Context.new(nil, '12345', context_user) } let(:context) { described_class::Context.new(context_project, '12345', context_user, Set.new) }
let(:subject) { described_class.new(params, context) } let(:project_file) { described_class.new(params, context) }
before do before do
project.add_developer(user) project.add_developer(user)
...@@ -19,7 +20,7 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -19,7 +20,7 @@ describe Gitlab::Ci::Config::External::File::Project do
let(:params) { { file: 'file.yml', project: 'project' } } let(:params) { { file: 'file.yml', project: 'project' } }
it 'should return true' do it 'should return true' do
expect(subject).to be_matching expect(project_file).to be_matching
end end
end end
...@@ -27,7 +28,7 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -27,7 +28,7 @@ describe Gitlab::Ci::Config::External::File::Project do
let(:params) { { file: 'file.yml' } } let(:params) { { file: 'file.yml' } }
it 'should return false' do it 'should return false' do
expect(subject).not_to be_matching expect(project_file).not_to be_matching
end end
end end
...@@ -35,7 +36,7 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -35,7 +36,7 @@ describe Gitlab::Ci::Config::External::File::Project do
let(:params) { { project: 'project' } } let(:params) { { project: 'project' } }
it 'should return false' do it 'should return false' do
expect(subject).not_to be_matching expect(project_file).not_to be_matching
end end
end end
...@@ -43,7 +44,7 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -43,7 +44,7 @@ describe Gitlab::Ci::Config::External::File::Project do
let(:params) { {} } let(:params) { {} }
it 'should return false' do it 'should return false' do
expect(subject).not_to be_matching expect(project_file).not_to be_matching
end end
end end
end end
...@@ -61,15 +62,15 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -61,15 +62,15 @@ describe Gitlab::Ci::Config::External::File::Project do
end end
it 'should return true' do it 'should return true' do
expect(subject).to be_valid expect(project_file).to be_valid
end end
context 'when user does not have permission to access file' do context 'when user does not have permission to access file' do
let(:context_user) { create(:user) } let(:context_user) { create(:user) }
it 'should return false' do it 'should return false' do
expect(subject).not_to be_valid expect(project_file).not_to be_valid
expect(subject.error_message).to include("Project `#{project.full_path}` not found or access denied!") expect(project_file.error_message).to include("Project `#{project.full_path}` not found or access denied!")
end end
end end
end end
...@@ -86,7 +87,7 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -86,7 +87,7 @@ describe Gitlab::Ci::Config::External::File::Project do
end end
it 'should return true' do it 'should return true' do
expect(subject).to be_valid expect(project_file).to be_valid
end end
end end
...@@ -102,8 +103,8 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -102,8 +103,8 @@ describe Gitlab::Ci::Config::External::File::Project do
end end
it 'should return false' do it 'should return false' do
expect(subject).not_to be_valid expect(project_file).not_to be_valid
expect(subject.error_message).to include("Project `#{project.full_path}` file `/file.yml` is empty!") expect(project_file.error_message).to include("Project `#{project.full_path}` file `/file.yml` is empty!")
end end
end end
...@@ -113,8 +114,8 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -113,8 +114,8 @@ describe Gitlab::Ci::Config::External::File::Project do
end end
it 'should return false' do it 'should return false' do
expect(subject).not_to be_valid expect(project_file).not_to be_valid
expect(subject.error_message).to include("Project `#{project.full_path}` reference `I-Do-Not-Exist` does not exist!") expect(project_file.error_message).to include("Project `#{project.full_path}` reference `I-Do-Not-Exist` does not exist!")
end end
end end
...@@ -124,8 +125,8 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -124,8 +125,8 @@ describe Gitlab::Ci::Config::External::File::Project do
end end
it 'should return false' do it 'should return false' do
expect(subject).not_to be_valid expect(project_file).not_to be_valid
expect(subject.error_message).to include("Project `#{project.full_path}` file `/invalid-file.yml` does not exist!") expect(project_file.error_message).to include("Project `#{project.full_path}` file `/invalid-file.yml` does not exist!")
end end
end end
...@@ -135,12 +136,22 @@ describe Gitlab::Ci::Config::External::File::Project do ...@@ -135,12 +136,22 @@ describe Gitlab::Ci::Config::External::File::Project do
end end
it 'should return false' do it 'should return false' do
expect(subject).not_to be_valid expect(project_file).not_to be_valid
expect(subject.error_message).to include('Included file `/invalid-file` does not have YAML extension!') expect(project_file.error_message).to include('Included file `/invalid-file` does not have YAML extension!')
end end
end end
end end
describe '#expand_context' do
let(:params) { { file: 'file.yml', project: project.full_path, ref: 'master' } }
subject { project_file.send(:expand_context) }
it 'inherits user, and target project and sha' do
is_expected.to include(user: user, project: project, sha: project.commit('master').id)
end
end
private private
def stub_project_blob(ref, path) def stub_project_blob(ref, path)
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
require 'spec_helper' require 'spec_helper'
describe Gitlab::Ci::Config::External::File::Remote do describe Gitlab::Ci::Config::External::File::Remote do
let(:context) { described_class::Context.new(nil, '12345', nil) } let(:context) { described_class::Context.new(nil, '12345', nil, Set.new) }
let(:params) { { remote: location } } let(:params) { { remote: location } }
let(:remote_file) { described_class.new(params, context) } let(:remote_file) { described_class.new(params, context) }
let(:location) { 'https://gitlab.com/gitlab-org/gitlab-ce/blob/1234/.gitlab-ci-1.yml' } let(:location) { 'https://gitlab.com/gitlab-org/gitlab-ce/blob/1234/.gitlab-ci-1.yml' }
...@@ -181,4 +181,14 @@ describe Gitlab::Ci::Config::External::File::Remote do ...@@ -181,4 +181,14 @@ describe Gitlab::Ci::Config::External::File::Remote do
end end
end end
end end