parallel_emerge: Use a minimum of 4 threads to fetch
Because fetch jobs are IO bound, we can use more processes to take full
advantage of the network while waiting for other CPU bound jobs. This CL
increases number of fetch jobs to at least 4.
BUG=brillo:1023
TEST=unittests
Change-Id: Ie601fcf1dfd3856b2d46e3bde3f47c63ed70a35a
Reviewed-on: https://chromium-review.googlesource.com/270091
Reviewed-by: Bertrand Simonnet <bsimonnet@chromium.org>
Commit-Queue: Nam Nguyen <namnguyen@chromium.org>
Tested-by: Nam Nguyen <namnguyen@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
diff --git a/scripts/parallel_emerge.py b/scripts/parallel_emerge.py
index c1cb2be..ae1e6ec 100644
--- a/scripts/parallel_emerge.py
+++ b/scripts/parallel_emerge.py
@@ -1324,7 +1324,9 @@
# jobs.
procs = min(self._total_jobs,
emerge.opts.pop("--jobs", multiprocessing.cpu_count()))
- self._build_procs = self._unpack_procs = self._fetch_procs = max(1, procs)
+ self._build_procs = self._unpack_procs = max(1, procs)
+ # Fetch is IO bound, we can use more processes.
+ self._fetch_procs = max(4, procs)
self._load_avg = emerge.opts.pop("--load-average", None)
self._job_queue = multiprocessing.Queue()
self._print_queue = multiprocessing.Queue()