GitLab steht wegen Wartungsarbeiten am Montag, den 10. Mai, zwischen 17:00 und 19:00 Uhr nicht zur Verfügung.

Commit 39203f1a authored by Kamil Trzcinski's avatar Kamil Trzcinski

Pre-create all builds for Pipeline when a trigger is received

This change simplifies a Pipeline processing by introducing a special new status: created.
This status is used for all builds that are created for a pipeline.
We are then processing next stages and queueing some of the builds (created -> pending) or skipping them (created -> skipped).
This makes it possible to simplify and solve a few ordering problems with how previously builds were scheduled.
This also allows us to visualise a full pipeline (with created builds).

This also removes an after_touch used for updating a pipeline state parameters.
Right now in various places we explicitly call a reload_status! on pipeline to force it to be updated and saved.
parent 2facade8
...@@ -17,6 +17,7 @@ v 8.11.0 (unreleased) ...@@ -17,6 +17,7 @@ v 8.11.0 (unreleased)
- Cache the commit author in RequestStore to avoid extra lookups in PostReceive - Cache the commit author in RequestStore to avoid extra lookups in PostReceive
- Expand commit message width in repo view (ClemMakesApps) - Expand commit message width in repo view (ClemMakesApps)
- Cache highlighted diff lines for merge requests - Cache highlighted diff lines for merge requests
- Pre-create all builds for a Pipeline when the new Pipeline is created !5295
- Fix of 'Commits being passed to custom hooks are already reachable when using the UI' - Fix of 'Commits being passed to custom hooks are already reachable when using the UI'
- Fix awardable button mutuality loading spinners (ClemMakesApps) - Fix awardable button mutuality loading spinners (ClemMakesApps)
- Add support for using RequestStore within Sidekiq tasks via SIDEKIQ_REQUEST_STORE env variable - Add support for using RequestStore within Sidekiq tasks via SIDEKIQ_REQUEST_STORE env variable
......
...@@ -6,7 +6,7 @@ class Projects::BuildsController < Projects::ApplicationController ...@@ -6,7 +6,7 @@ class Projects::BuildsController < Projects::ApplicationController
def index def index
@scope = params[:scope] @scope = params[:scope]
@all_builds = project.builds @all_builds = project.builds.relevant
@builds = @all_builds.order('created_at DESC') @builds = @all_builds.order('created_at DESC')
@builds = @builds =
case @scope case @scope
......
...@@ -134,8 +134,8 @@ def define_note_vars ...@@ -134,8 +134,8 @@ def define_note_vars
end end
def define_status_vars def define_status_vars
@statuses = CommitStatus.where(pipeline: pipelines) @statuses = CommitStatus.where(pipeline: pipelines).relevant
@builds = Ci::Build.where(pipeline: pipelines) @builds = Ci::Build.where(pipeline: pipelines).relevant
end end
def assign_change_commit_vars(mr_source_branch) def assign_change_commit_vars(mr_source_branch)
......
...@@ -160,7 +160,7 @@ def new ...@@ -160,7 +160,7 @@ def new
@diff_notes_disabled = true @diff_notes_disabled = true
@pipeline = @merge_request.pipeline @pipeline = @merge_request.pipeline
@statuses = @pipeline.statuses if @pipeline @statuses = @pipeline.statuses.relevant if @pipeline
@note_counts = Note.where(commit_id: @commits.map(&:id)). @note_counts = Note.where(commit_id: @commits.map(&:id)).
group(:commit_id).count group(:commit_id).count
...@@ -362,7 +362,7 @@ def define_show_vars ...@@ -362,7 +362,7 @@ def define_show_vars
@commits_count = @merge_request.commits.count @commits_count = @merge_request.commits.count
@pipeline = @merge_request.pipeline @pipeline = @merge_request.pipeline
@statuses = @pipeline.statuses if @pipeline @statuses = @pipeline.statuses.relevant if @pipeline
if @merge_request.locked_long_ago? if @merge_request.locked_long_ago?
@merge_request.unlock_mr @merge_request.unlock_mr
......
...@@ -19,7 +19,7 @@ def new ...@@ -19,7 +19,7 @@ def new
end end
def create def create
@pipeline = Ci::CreatePipelineService.new(project, current_user, create_params).execute @pipeline = Ci::CreatePipelineService.new(project, current_user, create_params).execute(ignore_skip_ci: true, save_on_errors: false)
unless @pipeline.persisted? unless @pipeline.persisted?
render 'new' render 'new'
return return
......
...@@ -16,7 +16,7 @@ class Build < CommitStatus ...@@ -16,7 +16,7 @@ class Build < CommitStatus
scope :with_artifacts_not_expired, ->() { with_artifacts.where('artifacts_expire_at IS NULL OR artifacts_expire_at > ?', Time.now) } scope :with_artifacts_not_expired, ->() { with_artifacts.where('artifacts_expire_at IS NULL OR artifacts_expire_at > ?', Time.now) }
scope :with_expired_artifacts, ->() { with_artifacts.where('artifacts_expire_at < ?', Time.now) } scope :with_expired_artifacts, ->() { with_artifacts.where('artifacts_expire_at < ?', Time.now) }
scope :last_month, ->() { where('created_at > ?', Date.today - 1.month) } scope :last_month, ->() { where('created_at > ?', Date.today - 1.month) }
scope :manual_actions, ->() { where(when: :manual) } scope :manual_actions, ->() { where(when: :manual).relevant }
mount_uploader :artifacts_file, ArtifactUploader mount_uploader :artifacts_file, ArtifactUploader
mount_uploader :artifacts_metadata, ArtifactUploader mount_uploader :artifacts_metadata, ArtifactUploader
...@@ -65,17 +65,11 @@ def retry(build, user = nil) ...@@ -65,17 +65,11 @@ def retry(build, user = nil)
end end
end end
state_machine :status, initial: :pending do state_machine :status do
after_transition pending: :running do |build| after_transition pending: :running do |build|
build.execute_hooks build.execute_hooks
end end
# We use around_transition to create builds for next stage as soon as possible, before the `after_*` is executed
around_transition any => [:success, :failed, :canceled] do |build, block|
block.call
build.pipeline.create_next_builds(build) if build.pipeline
end
after_transition any => [:success, :failed, :canceled] do |build| after_transition any => [:success, :failed, :canceled] do |build|
build.update_coverage build.update_coverage
build.execute_hooks build.execute_hooks
...@@ -461,7 +455,7 @@ def predefined_variables ...@@ -461,7 +455,7 @@ def predefined_variables
def build_attributes_from_config def build_attributes_from_config
return {} unless pipeline.config_processor return {} unless pipeline.config_processor
pipeline.config_processor.build_attributes(name) pipeline.config_processor.build_attributes(name)
end end
end end
......
...@@ -13,11 +13,10 @@ class Pipeline < ActiveRecord::Base ...@@ -13,11 +13,10 @@ class Pipeline < ActiveRecord::Base
has_many :trigger_requests, dependent: :destroy, class_name: 'Ci::TriggerRequest', foreign_key: :commit_id has_many :trigger_requests, dependent: :destroy, class_name: 'Ci::TriggerRequest', foreign_key: :commit_id
validates_presence_of :sha validates_presence_of :sha
validates_presence_of :ref
validates_presence_of :status validates_presence_of :status
validate :valid_commit_sha validate :valid_commit_sha
# Invalidate object and save if when touched
after_touch :update_state
after_save :keep_around_commits after_save :keep_around_commits
# ref can't be HEAD or SHA, can only be branch/tag name # ref can't be HEAD or SHA, can only be branch/tag name
...@@ -90,12 +89,16 @@ def cancelable? ...@@ -90,12 +89,16 @@ def cancelable?
def cancel_running def cancel_running
builds.running_or_pending.each(&:cancel) builds.running_or_pending.each(&:cancel)
reload_status!
end end
def retry_failed(user) def retry_failed(user)
builds.latest.failed.select(&:retryable?).each do |build| builds.latest.failed.select(&:retryable?).each do |build|
Ci::Build.retry(build, user) Ci::Build.retry(build, user)
end end
reload_status!
end end
def latest? def latest?
...@@ -109,37 +112,6 @@ def triggered? ...@@ -109,37 +112,6 @@ def triggered?
trigger_requests.any? trigger_requests.any?
end end
def create_builds(user, trigger_request = nil)
##
# We persist pipeline only if there are builds available
#
return unless config_processor
build_builds_for_stages(config_processor.stages, user,
'success', trigger_request) && save
end
def create_next_builds(build)
return unless config_processor
# don't create other builds if this one is retried
latest_builds = builds.latest
return unless latest_builds.exists?(build.id)
# get list of stages after this build
next_stages = config_processor.stages.drop_while { |stage| stage != build.stage }
next_stages.delete(build.stage)
# get status for all prior builds
prior_builds = latest_builds.where.not(stage: next_stages)
prior_status = prior_builds.status
# build builds for next stage that has builds available
# and save pipeline if we have builds
build_builds_for_stages(next_stages, build.user, prior_status,
build.trigger_request) && save
end
def retried def retried
@retried ||= (statuses.order(id: :desc) - statuses.latest) @retried ||= (statuses.order(id: :desc) - statuses.latest)
end end
...@@ -151,6 +123,14 @@ def coverage ...@@ -151,6 +123,14 @@ def coverage
end end
end end
def config_builds_attributes
return [] unless config_processor
config_processor.
builds_for_ref(ref, tag?, trigger_requests.first).
sort_by { |build| build[:stage_idx] }
end
def has_warnings? def has_warnings?
builds.latest.ignored.any? builds.latest.ignored.any?
end end
...@@ -182,10 +162,6 @@ def ci_yaml_file ...@@ -182,10 +162,6 @@ def ci_yaml_file
end end
end end
def skip_ci?
git_commit_message =~ /\[(ci skip|skip ci)\]/i if git_commit_message
end
def environments def environments
builds.where.not(environment: nil).success.pluck(:environment).uniq builds.where.not(environment: nil).success.pluck(:environment).uniq
end end
...@@ -207,39 +183,33 @@ def notes ...@@ -207,39 +183,33 @@ def notes
Note.for_commit_id(sha) Note.for_commit_id(sha)
end end
def process!
Ci::ProcessPipelineService.new(project, user).execute(self)
reload_status!
end
def predefined_variables def predefined_variables
[ [
{ key: 'CI_PIPELINE_ID', value: id.to_s, public: true } { key: 'CI_PIPELINE_ID', value: id.to_s, public: true }
] ]
end end
private def reload_status!
def build_builds_for_stages(stages, user, status, trigger_request)
##
# Note that `Array#any?` implements a short circuit evaluation, so we
# build builds only for the first stage that has builds available.
#
stages.any? do |stage|
CreateBuildsService.new(self).
execute(stage, user, status, trigger_request).
any?(&:active?)
end
end
def update_state
statuses.reload statuses.reload
self.status = if yaml_errors.blank? self.status =
statuses.latest.status || 'skipped' if yaml_errors.blank?
else statuses.latest.status || 'skipped'
'failed' else
end 'failed'
end
self.started_at = statuses.started_at self.started_at = statuses.started_at
self.finished_at = statuses.finished_at self.finished_at = statuses.finished_at
self.duration = statuses.latest.duration self.duration = statuses.latest.duration
save save
end end
private
def keep_around_commits def keep_around_commits
return unless project return unless project
......
...@@ -5,7 +5,7 @@ class CommitStatus < ActiveRecord::Base ...@@ -5,7 +5,7 @@ class CommitStatus < ActiveRecord::Base
self.table_name = 'ci_builds' self.table_name = 'ci_builds'
belongs_to :project, class_name: '::Project', foreign_key: :gl_project_id belongs_to :project, class_name: '::Project', foreign_key: :gl_project_id
belongs_to :pipeline, class_name: 'Ci::Pipeline', foreign_key: :commit_id, touch: true belongs_to :pipeline, class_name: 'Ci::Pipeline', foreign_key: :commit_id
belongs_to :user belongs_to :user
delegate :commit, to: :pipeline delegate :commit, to: :pipeline
...@@ -25,28 +25,36 @@ class CommitStatus < ActiveRecord::Base ...@@ -25,28 +25,36 @@ class CommitStatus < ActiveRecord::Base
scope :ordered, -> { order(:name) } scope :ordered, -> { order(:name) }
scope :ignored, -> { where(allow_failure: true, status: [:failed, :canceled]) } scope :ignored, -> { where(allow_failure: true, status: [:failed, :canceled]) }
state_machine :status, initial: :pending do state_machine :status do
event :queue do event :queue do
transition skipped: :pending transition [:created, :skipped] => :pending
end end
event :run do event :run do
transition pending: :running transition pending: :running
end end
event :skip do
transition [:created, :pending] => :skipped
end
event :drop do event :drop do
transition [:pending, :running] => :failed transition [:created, :pending, :running] => :failed
end end
event :success do event :success do
transition [:pending, :running] => :success transition [:created, :pending, :running] => :success
end end
event :cancel do event :cancel do
transition [:pending, :running] => :canceled transition [:created, :pending, :running] => :canceled
end
after_transition created: [:pending, :running] do |commit_status|
commit_status.update_attributes queued_at: Time.now
end end
after_transition pending: :running do |commit_status| after_transition [:created, :pending] => :running do |commit_status|
commit_status.update_attributes started_at: Time.now commit_status.update_attributes started_at: Time.now
end end
...@@ -54,13 +62,20 @@ class CommitStatus < ActiveRecord::Base ...@@ -54,13 +62,20 @@ class CommitStatus < ActiveRecord::Base
commit_status.update_attributes finished_at: Time.now commit_status.update_attributes finished_at: Time.now
end end
after_transition [:pending, :running] => :success do |commit_status| after_transition [:created, :pending, :running] => :success do |commit_status|
MergeRequests::MergeWhenBuildSucceedsService.new(commit_status.pipeline.project, nil).trigger(commit_status) MergeRequests::MergeWhenBuildSucceedsService.new(commit_status.pipeline.project, nil).trigger(commit_status)
end end
after_transition any => :failed do |commit_status| after_transition any => :failed do |commit_status|
MergeRequests::AddTodoWhenBuildFailsService.new(commit_status.pipeline.project, nil).execute(commit_status) MergeRequests::AddTodoWhenBuildFailsService.new(commit_status.pipeline.project, nil).execute(commit_status)
end end
# We use around_transition to process pipeline on next stages as soon as possible, before the `after_*` is executed
around_transition any => [:success, :failed, :canceled] do |commit_status, block|
block.call
commit_status.pipeline.process! if commit_status.pipeline
end
end end
delegate :sha, :short_sha, to: :pipeline delegate :sha, :short_sha, to: :pipeline
......
module Statuseable module Statuseable
extend ActiveSupport::Concern extend ActiveSupport::Concern
AVAILABLE_STATUSES = %w(pending running success failed canceled skipped) AVAILABLE_STATUSES = %w[created pending running success failed canceled skipped]
STARTED_STATUSES = %w[running success failed skipped]
ACTIVE_STATUSES = %w[pending running]
COMPLETED_STATUSES = %w[success failed canceled]
class_methods do class_methods do
def status_sql def status_sql
builds = all.select('count(*)').to_sql scope = all.relevant
success = all.success.select('count(*)').to_sql builds = scope.select('count(*)').to_sql
ignored = all.ignored.select('count(*)').to_sql if all.respond_to?(:ignored) success = scope.success.select('count(*)').to_sql
ignored = scope.ignored.select('count(*)').to_sql if scope.respond_to?(:ignored)
ignored ||= '0' ignored ||= '0'
pending = all.pending.select('count(*)').to_sql pending = scope.pending.select('count(*)').to_sql
running = all.running.select('count(*)').to_sql running = scope.running.select('count(*)').to_sql
canceled = all.canceled.select('count(*)').to_sql canceled = scope.canceled.select('count(*)').to_sql
skipped = all.skipped.select('count(*)').to_sql skipped = scope.skipped.select('count(*)').to_sql
deduce_status = "(CASE deduce_status = "(CASE
WHEN (#{builds})=0 THEN NULL WHEN (#{builds})=0 THEN NULL
...@@ -48,7 +52,8 @@ def finished_at ...@@ -48,7 +52,8 @@ def finished_at
included do included do
validates :status, inclusion: { in: AVAILABLE_STATUSES } validates :status, inclusion: { in: AVAILABLE_STATUSES }
state_machine :status, initial: :pending do state_machine :status, initial: :created do
state :created, value: 'created'
state :pending, value: 'pending' state :pending, value: 'pending'
state :running, value: 'running' state :running, value: 'running'
state :failed, value: 'failed' state :failed, value: 'failed'
...@@ -57,6 +62,8 @@ def finished_at ...@@ -57,6 +62,8 @@ def finished_at
state :skipped, value: 'skipped' state :skipped, value: 'skipped'
end end
scope :created, -> { where(status: 'created') }
scope :relevant, -> { where.not(status: 'created') }
scope :running, -> { where(status: 'running') } scope :running, -> { where(status: 'running') }
scope :pending, -> { where(status: 'pending') } scope :pending, -> { where(status: 'pending') }
scope :success, -> { where(status: 'success') } scope :success, -> { where(status: 'success') }
...@@ -68,14 +75,14 @@ def finished_at ...@@ -68,14 +75,14 @@ def finished_at
end end
def started? def started?
!pending? && !canceled? && started_at STARTED_STATUSES.include?(status) && started_at
end end
def active? def active?
running? || pending? ACTIVE_STATUSES.include?(status)
end end
def complete? def complete?
canceled? || success? || failed? COMPLETED_STATUSES.include?(status)
end end
end end
module Ci
class CreateBuildsService
def initialize(pipeline)
@pipeline = pipeline
@config = pipeline.config_processor
end
def execute(stage, user, status, trigger_request = nil)
builds_attrs = @config.builds_for_stage_and_ref(stage, @pipeline.ref, @pipeline.tag, trigger_request)
# check when to create next build
builds_attrs = builds_attrs.select do |build_attrs|
case build_attrs[:when]
when 'on_success'
status == 'success'
when 'on_failure'
status == 'failed'
when 'always', 'manual'
%w(success failed).include?(status)
end
end
# don't create the same build twice
builds_attrs.reject! do |build_attrs|
@pipeline.builds.find_by(ref: @pipeline.ref,
tag: @pipeline.tag,
trigger_request: trigger_request,
name: build_attrs[:name])
end
builds_attrs.map do |build_attrs|
build_attrs.slice!(:name,
:commands,
:tag_list,
:options,
:allow_failure,
:stage,
:stage_idx,
:environment,
:when,
:yaml_variables)
build_attrs.merge!(pipeline: @pipeline,
ref: @pipeline.ref,
tag: @pipeline.tag,
trigger_request: trigger_request,
user: user,
project: @pipeline.project)
# TODO: The proper implementation for this is in
# https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5295
build_attrs[:status] = 'skipped' if build_attrs[:when] == 'manual'
##
# We do not persist new builds here.
# Those will be persisted when @pipeline is saved.
#
@pipeline.builds.new(build_attrs)
end
end
end
end
module Ci
class CreatePipelineBuildsService < BaseService
attr_reader :pipeline
def execute(pipeline)
@pipeline = pipeline
new_builds.map do |build_attributes|
create_build(build_attributes)