autoninja - ninja wrapper to make goma easier
Using goma requires the developer to remember which build directories
use goma and which don't so that they can pass an appropriate -j number.
Getting this wrong makes builds slower, either by under utilizing
resources or by causing a self-inflicted DOS attack. Usage:
autoninja -C out/debug
autoninja looks at the settings for the specified build directory and
then selects either -j num_cores*20 or no -j flag based on the
use_goma setting.
You can set the NINJA_CORE_MULTIPLIER variable to change from the
default 20* multiplier. You can also use NINJA_CORE_ADDITION if you
want non-goma builds to specify -j with an offset to the number of
cores, such as this Linux command:
NINJA_CORE_ADDITION=-2 autoninja -C out/release base
This will tell autoninja to pass -j to ninja with num_cores-2 as the
parameter.
On Windows you can have a ninja.bat file (ahead of ninja on the path)
such that autoninja will automatically be used. It should contain this:
@call autoninja.bat %*
Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655
Reviewed-on: https://chromium-review.googlesource.com/517662
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
diff --git a/autoninja.py b/autoninja.py
new file mode 100755
index 0000000..056b9c1
--- /dev/null
+++ b/autoninja.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2017 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""
+This script (intended to be invoked by autoninja or autoninja.bat) detects
+whether a build is using goma. If so it runs with a large -j value, and
+otherwise it chooses a small one. This auto-adjustment makes using goma simpler
+and safer, and avoids errors that can cause slow goma builds or swap-storms
+on non-goma builds.
+"""
+
+import multiprocessing
+import os
+import re
+import sys
+
+j_specified = False
+output_dir = '.'
+for index, arg in enumerate(sys.argv[1:]):
+ if arg == '-j':
+ j_specified = True
+ if arg == '-C':
+ # + 1 to get the next argument and +1 because we trimmed off sys.argv[0]
+ output_dir = sys.argv[index + 2]
+
+use_goma = False
+try:
+ with open(os.path.join(output_dir, 'args.gn')) as file_handle:
+ for line in file_handle:
+ # This regex pattern copied from create_installer_archive.py
+ m = re.match('^\s*use_goma\s*=\s*true(\s*$|\s*#.*$)', line)
+ if m:
+ use_goma = True
+except IOError:
+ pass
+
+if sys.platform.startswith('win'):
+ # Specify ninja.exe on Windows so that ninja.bat can call autoninja and not
+ # be called back.
+ args = ['ninja.exe'] + sys.argv[1:]
+else:
+ args = ['ninja'] + sys.argv[1:]
+
+num_cores = multiprocessing.cpu_count()
+if not j_specified:
+ if use_goma:
+ args.append('-j')
+ core_multiplier = int(os.environ.get("NINJA_CORE_MULTIPLIER", "20"))
+ args.append('%d' % (num_cores * core_multiplier))
+ else:
+ core_addition = os.environ.get("NINJA_CORE_ADDITION")
+ if core_addition:
+ core_addition = int(core_addition)
+ args.append('-j')
+ args.append('%d' % (num_cores + core_addition))
+
+# Specify a maximum CPU load so that running builds in two different command
+# prompts won't overload the system too much. This is not reliable enough to
+# be used to auto-adjust between goma/non-goma loads, but it is a nice
+# fallback load balancer.
+args.append('-l')
+args.append('%d' % num_cores)
+
+print ' '.join(args)