Move cbuildbot temporary files into a separate root.
Constrain cbuildbot efforts into a specific tempdir per run, so
that we no longer have to worry about cruft left leading to the
builders running out of space.
Note that this wiping means we can't see what was left; instrumentation
for that requires a secondary CL to deal with (this effort is purely
to control the mess/breakage now, then we'll fix the offenders
after).
Finally, the underlying tempfile decorators/machinery we use
is updated to do env manipulation (thus subprocesses use that
tmp), and updating of python internals so any points between
the setup/teardown use the tempspace.
In tightening this down, a corner case in RunCommands tempfile usage
is spotted and dealt with- it previously would fail if given a TMP
that pointed at a nonexistant directory. Since RunCommand's tempfile
usage is known safe (it deletes the file immediately, thus can't leak
even during a sigkill) an explicit fallback to /tmp is allowed.
BUG=chromium-os:30750
TEST=Pause cbuildbot in the middle of a run and verify it stores
temporary directories in this location. Remote trybot run.
TEST=cbuildbot <run> # ctrl+c it, verify the stated location is
# wiped.
Change-Id: I065d857e10c59fd2115dfaf0cd106e2cb3195978
Reviewed-on: https://gerrit.chromium.org/gerrit/22285
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
diff --git a/scripts/cbuildbot.py b/scripts/cbuildbot.py
index 480b85e..baa2d1f 100644
--- a/scripts/cbuildbot.py
+++ b/scripts/cbuildbot.py
@@ -1118,19 +1118,25 @@
with cleanup.EnforcedCleanupSection() as critical_section:
with sudo.SudoKeepAlive():
- with cros_lib.AllowDisabling(options.cgroups,
- cgroups.SimpleContainChildren, 'cbuildbot'):
- # Mark everything between EnforcedCleanupSection and here as having to
- # be rolled back via the contextmanager cleanup handlers. This ensures
- # that sudo bits cannot outlive cbuildbot, that anything cgroups
- # would kill gets killed, etc.
- critical_section.ForkWatchdog()
+ # If we're in resume mode, use our parents tempdir rather than
+ # nesting another layer.
+ with cros_lib.AllowDisabling(
+ not options.resume, osutils.TempDirContextManager, 'cbuildbot-tmp'):
+ logging.debug("Cbuildbot tempdir is %r.", os.environ.get('TMP'))
+ with cros_lib.AllowDisabling(options.cgroups,
+ cgroups.SimpleContainChildren,
+ 'cbuildbot'):
+ # Mark everything between EnforcedCleanupSection and here as having to
+ # be rolled back via the contextmanager cleanup handlers. This
+ # ensures that sudo bits cannot outlive cbuildbot, that anything
+ # cgroups would kill gets killed, etc.
+ critical_section.ForkWatchdog()
- with cros_lib.AllowDisabling(options.timeout > 0,
- cros_lib.Timeout, options.timeout):
- if not options.buildbot:
- build_config = cbuildbot_config.OverrideConfigForTrybot(
- build_config,
- options.remote_trybot)
+ with cros_lib.AllowDisabling(options.timeout > 0,
+ cros_lib.Timeout, options.timeout):
+ if not options.buildbot:
+ build_config = cbuildbot_config.OverrideConfigForTrybot(
+ build_config,
+ options.remote_trybot)
- _RunBuildStagesWrapper(options, build_config)
+ _RunBuildStagesWrapper(options, build_config)