buildbot: switch buildbot & cros_sdk to xz
Previously submitted in https://gerrit.chromium.org/gerrit/23967
Requires https://gerrit.chromium.org/gerrit/35098 to work properly.
This makes cros_sdk prefer .tar.xz archives over .tbz2, while
changing the chroot builder to produce and upload .tar.xz on the next
iteration.
Simple benchmarks:
tar: 1.847.654.400 cros-sdk-29.05.12.103036.tar
xz: 431.625.428 cros-sdk-29.05.12.103036.tar.xz
Enc: real 13m41.632s user 13m37.130s sys 0m01.470s
Dec: real 0m52.440s user 0m34.510s sys 0m01.390s
xz -9 (max): 350.064.736 cros-sdk-29.05.12.103036.tar.xz
Enc: real 16m01.348s user 15m56.610s sys 0m01.680s
Dec: real 0m28.328s user 0m26.670s sys 0m01.530s
pixz: 457.186.584 cros-sdk-29.05.12.103036.tpxz
Enc: real 1m00.774s user 21m14.320s sys 0m03.700s
Dec: real 0m02.898s user 0m46.030s sys 0m02.230s
pixz -9: 376.264.212 cros-sdk-29.05.12.103036.tpxz
Enc: real 1m30.333s user 23m37.600s sys 0m14.440s
Dec: real 0m03.781s user 0m36.260s sys 0m03.370s
pbzip2: 575.609.262 cros-sdk-29.05.12.103036.tbz2
Enc: real 0m18.216s user 7m01.470s sys 0m02.940s
Dec: real 0m06.980s user 2m17.220s sys 0m03.100s
bzip2: 574.026.150 cros-sdk-29.05.12.103036.tbz2
Enc: real 3m49.258s user 3m43.500s sys 0m1.470s
Dec: real 1m21.931s user 1m13.000s sys 0m2.360s
Current modes are:
pbzip2 for encoding
bzip2 for decoding, or pbzip2 on machines that have it (most don't)
New are:
xz -9 for encoding
xz for decoding
This causes a slowdown on the side of cros-sdk bot, but speedup
for decoding on most dev machines.
BUG=chromium-os:19287
TEST=trybot chromiumos-sdk
TEST=run unit tests
TEST=pylint
CQ-DEPEND=I2df2ee76eaf9d31f6fcc7aa54a3e6a1820224e5d
Change-Id: Ie15c2e327955e61c652299d9a503695cd5eb912b
Reviewed-on: https://gerrit.chromium.org/gerrit/35149
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
diff --git a/scripts/upload_prebuilts.py b/scripts/upload_prebuilts.py
index 08d578e..3f7ad78 100644
--- a/scripts/upload_prebuilts.py
+++ b/scripts/upload_prebuilts.py
@@ -410,8 +410,7 @@
cmd = ['tar', '-I', bzip2, '-cf', tarfile]
excluded_paths = ('usr/lib/debug', 'usr/local/autotest', 'packages',
'tmp')
- for path in excluded_paths:
- cmd.append('--exclude=%s/*' % path)
+ cmd.extend('--exclude=%s/*' % path for path in excluded_paths)
cmd.append('.')
cros_build_lib.SudoRunCommand(cmd, cwd=os.path.join(cwd, boardname))
else:
@@ -426,7 +425,7 @@
# FIXME(zbehan): Why does version contain the prefix "chroot-"?
version_str = version[len('chroot-'):]
remote_tarfile = \
- '%s/cros-sdk-%s.tbz2' % (_SDK_GS_BUCKET, version_str)
+ '%s/cros-sdk-%s.tar.xz' % (_SDK_GS_BUCKET, version_str)
# For SDK, also upload the manifest which is guaranteed to exist
# by the builderstage.
_GsUpload(tarfile + '.Manifest', remote_tarfile + '.Manifest',
@@ -435,11 +434,10 @@
# scripts rely.
pointerfile = os.path.join(tmpdir, 'cros-sdk-latest.conf')
remote_pointerfile = '%s/cros-sdk-latest.conf' % _SDK_GS_BUCKET
- osutils.WriteFile(pointerfile, 'LATEST_SDK=%s' % version_str)
+ osutils.WriteFile(pointerfile, 'LATEST_SDK="%s"' % version_str)
_GsUpload(pointerfile, remote_pointerfile, self._acl)
_GsUpload(tarfile, remote_tarfile, self._acl)
-
def _GetTargets(self):
"""Retuns the list of targets to use."""
targets = self._slave_targets[:]