cbuildbot: Use build 'deadline' to abort slaves when master quits.
When builds are launched in the master-slave model, we'd like to abort the
slaves about the same time when the master times out on them.
After this CL,
- The master posts a reasonable deadline for the build to cidb, and waits for
slaves to finish till that deadline.
- The slaves abort themselves if the deadline set by their master is hit.
- CQ Master: waits for a long time in the sync stage for CLs to be +2'ed, so it
extends the deadline after the sync stage so that slaves have enough time to
finish their build.
BUG=chromium:431836
TEST=(1) ./cbuildbot/run_tests
(2) Launch a master trybot and verify that a sensible deadline is posted to
cidb.
(3) Launch a slave trybot pointing to the earlier master trybot as the
master, and verify that the slave posts the same deadline and aborts
when the deadline is hit.
Change-Id: Idebd09630eda1be10afae462d38f41ff913004de
Reviewed-on: https://chromium-review.googlesource.com/234853
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
diff --git a/scripts/cbuildbot.py b/scripts/cbuildbot.py
index c663592..25ce51b 100644
--- a/scripts/cbuildbot.py
+++ b/scripts/cbuildbot.py
@@ -1857,9 +1857,6 @@
# cgroups would kill gets killed, etc.
stack.Add(critical_section.ForkWatchdog)
- if options.timeout > 0:
- stack.Add(timeout_util.FatalTimeout, options.timeout)
-
if not options.buildbot:
build_config = cbuildbot_config.OverrideConfigForTrybot(
build_config, options)
@@ -1881,4 +1878,30 @@
_SetupCidb(options, build_config)
retry_stats.SetupStats()
+ # For master-slave builds: Update slave's timeout using master's published
+ # deadline.
+ if options.buildbot and options.master_build_id is not None:
+ slave_timeout = None
+ if cidb.CIDBConnectionFactory.IsCIDBSetup():
+ cidb_handle = cidb.CIDBConnectionFactory.GetCIDBConnectionForBuilder()
+ if cidb_handle:
+ slave_timeout = cidb_handle.GetTimeToDeadline(options.master_build_id)
+
+ if slave_timeout is not None:
+ # Cut me some slack. We artificially add a a small time here to the
+ # slave_timeout because '0' is handled specially, and because we don't
+ # want to timeout while trying to set things up.
+ slave_timeout = slave_timeout + 20
+ if options.timeout == 0 or slave_timeout < options.timeout:
+ logging.info('Updating slave build timeout to %d seconds enforced '
+ 'by the master',
+ slave_timeout)
+ options.timeout = slave_timeout
+ else:
+ logging.warning('Could not get master deadline for master-slave build. '
+ 'Can not set slave timeout.')
+
+ if options.timeout > 0:
+ stack.Add(timeout_util.FatalTimeout, options.timeout)
+
_RunBuildStagesWrapper(options, build_config)