cros: speed up basic commandline parsing
Our current design imports all subcommand modules before doing anything.
As we add more modules, and the modules themselves get larger, this does
not scale well at all. As it stands now, doing nothing or just passing
in --help takes over 500msec.
The reason for this design was to enable flexibility: the name of the
module implementing a subcommand did not have to match the subcommand,
and we could put more than one subcommand in a module using decorators.
In practice, we've never used this functionality -- we've always had a
one-to-one matching between the module name & the subcommand it held.
Lets make this a hard requirement to regain performance. We no longer
have to import every single subcommand every time (and all their unique
but unused modules), we only have to import & execute the specific one
the user has requested.
This cuts the no-op time by half for `cros`, and shaves 300msec off all
`cros` command users run.
BUG=chromium:868820, chromium:1022507
TEST=`cros --help` is faster
TEST=`cros lint ...` still works
Change-Id: Icc1cc9493c12dae94f5459a027fca37cecfa0aef
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1903449
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Chris McDonald <cjmcdonald@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
diff --git a/scripts/cros.py b/scripts/cros.py
index 40a5cb2..8368b29 100644
--- a/scripts/cros.py
+++ b/scripts/cros.py
@@ -21,27 +21,36 @@
from chromite.lib import cros_logging as logging
-def GetOptions(my_commands):
+def GetOptions(cmd_name=None):
"""Returns the parser to use for commandline parsing.
Args:
- my_commands: A dictionary mapping subcommand names to classes.
+ cmd_name: The subcommand to import & add.
Returns:
A commandline.ArgumentParser object.
"""
- parser = commandline.ArgumentParser(caching=True, default_log_level='notice')
+ # We need to omit help for the base parser so that we're able to parse out the
+ # subcommand for further parsing.
+ parser = commandline.ArgumentParser(caching=True, default_log_level='notice',
+ add_help=cmd_name is not None)
- if my_commands:
- subparsers = parser.add_subparsers(title='Subcommands')
- for cmd_name in sorted(my_commands.keys()):
- class_def = my_commands[cmd_name]
+ subparsers = parser.add_subparsers(title='Subcommands', dest='subcommand')
+ subparsers.required = True
+
+ # We add all the commands so `cros --help ...` looks reasonable.
+ # We add them in order also so the --help output is stable for users.
+ for subcommand in sorted(command.ListCommands()):
+ if subcommand == cmd_name:
+ class_def = command.ImportCommand(cmd_name)
epilog = getattr(class_def, 'EPILOG', None)
sub_parser = subparsers.add_parser(
cmd_name, description=class_def.__doc__, epilog=epilog,
caching=class_def.use_caching_options,
formatter_class=commandline.argparse.RawDescriptionHelpFormatter)
class_def.AddParser(sub_parser)
+ else:
+ subparsers.add_parser(subcommand, add_help=False)
return parser
@@ -53,13 +62,20 @@
def main(argv):
try:
- parser = GetOptions(command.ListCommands())
- # Cros currently does nothing without a subcmd. Print help if no args are
- # specified.
+ # The first time we parse the commandline is only to figure out what
+ # subcommand the user wants to run. This allows us to avoid importing
+ # all subcommands which can be quite slow. This works because there is
+ # no way in Python to list all subcommands and their help output in a
+ # single run.
+ parser = GetOptions()
if not argv:
parser.print_help()
return 1
+ namespace, _ = parser.parse_known_args(argv)
+ # The user has selected a subcommand now, so get the full parser after we
+ # import the single subcommand.
+ parser = GetOptions(namespace.subcommand)
namespace = parser.parse_args(argv)
subcommand = namespace.command_class(namespace)
try:
@@ -69,8 +85,7 @@
raise
except Exception as e:
code = 1
- logging.error('cros %s failed before completing.',
- subcommand.command_name)
+ logging.error('cros %s failed before completing.', namespace.subcommand)
if namespace.debug:
raise
else: