Re-land "Split up artifacts to download some in the foreground/background.""
This refactors the devserver to place artifacts into their own classes and
enables us to download certain artifacts in the foreground and some in the
background. I also add wait_for_status, an RPC you can call using the same
archive_url to determine whether background artifacts have finished
downloading (a blocking call).
My next change is to modify the dynamic_suite code to do:
Download()
Process Duts
Wait_For_Status
Run tests.
BUG=chromium-os:27285
TEST=Ran all unittests + pylint -- added a whole bunch of tests.
Ran downloader from localhost?downloader and saw artifacts staged correctly.
Change-Id: I14c888dcc08cf4df2719915fe2fe88d52fbcdc62
Reviewed-on: https://gerrit.chromium.org/gerrit/19244
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
diff --git a/gsutil_util.py b/gsutil_util.py
new file mode 100644
index 0000000..5567a60
--- /dev/null
+++ b/gsutil_util.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2012 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Module containing gsutil helper methods."""
+
+import subprocess
+
+GSUTIL_ATTEMPTS = 5
+
+
+class GSUtilError(Exception):
+ """Exception raises when we run into an error running gsutil."""
+ pass
+
+
+def GSUtilRun(cmd, err_msg):
+ """Runs a GSUTIL command up to GSUTIL_ATTEMPTS number of times.
+
+ Returns:
+ stdout of the called gsutil command.
+ Raises:
+ subprocess.CalledProcessError if all attempt to run gsutil cmd fails.
+ """
+ proc = None
+ for _attempt in range(GSUTIL_ATTEMPTS):
+ # Note processes can hang when capturing from stderr. This command
+ # specifically doesn't pipe stderr.
+ proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+ stdout, _stderr = proc.communicate()
+ if proc.returncode == 0:
+ return stdout
+ else:
+ raise GSUtilError('%s GSUTIL cmd %s failed with return code %d' % (
+ err_msg, cmd, proc.returncode))
+
+
+def DownloadFromGS(src, dst):
+ """Downloads object from gs_url |src| to |dst|.
+
+ Raises:
+ GSUtilError: if an error occurs during the download.
+ """
+ cmd = 'gsutil cp %s %s' % (src, dst)
+ msg = 'Failed to download "%s".' % src
+ GSUtilRun(cmd, msg)