parallel_emerge: Suppress multiprocessing exceptions during shutdown.
Specifically, due to multiprocessing sucking, we wind up seeing the
following quite frequently (line wrapped):
<parallel-emerge output>
Done
Exception in thread QueueFeederThread (most likely raised during
interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 525, in __bootstrap_inner
File "/usr/lib64/python2.6/threading.py", line 477, in run
File "/usr/lib64/python2.6/multiprocessing/queues.py", line 233, in _feed
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
Roughly, this (and those like it) come about due to threads still
in existance during late vm shutdown. Those threads are around due
to multiprocessing leaving those threads to be GC'd while marking
the threads as daemon (meaning the VM *can* shutdown regardless of
if they're still running).
Needless to say, this is stupid. This is fixed in 2.7 via
http://bugs.python.org/issue4106. We're not on 2.7 however, and
I'm tired of seeing this noisy traceback.
This CL basically manipulates the shutdown in a similar fashion,
suppressing this fully in my testing (under enough load, it may
still be possible for the error to slip out; that would be forkbomb
levels of load I suspect).
Roughly, addressing this via:
1) Reduced the pointless sys.exit usage; that triggers an Exception,
resulting in the vm strongly refererencing the traceback which
strongly references each frame it crashed through. One way to
keep instances around.
2) Explicitly forcing of gc collection. In testing, this tactic alone
seems to be enough to suppress the TB although that's mostly timing.
The step is necessary either way to ensure that PyFinalize's GC
run doesn't get first crack at these threads (if it does, by that
time the VM is mid shutdown thus the exception).
3) Explicitly go looking for those damn threads and try to join them
with a 1 second timeout. In testing, this tactic alone seems to be
enough to suppress the TB. This is basically a variation of what
the upstream fix was.
BUG=None
TEST=None, realistically. Fundamentally it's a race; heavy system
load seems to exacerbate it's frequency, but this can very.
Having a build_packages running in the background (full load)
while abusing setup_board to create a new target from scratch
was the recent 75% scenario- that said that scenario has broken
down in the past (right now locally, it's occuring, thus this
patch. YMMV however).
Change-Id: I2c7c63b23a75808fd33eeefb05d70e73c8463b99
Reviewed-on: https://gerrit.chromium.org/gerrit/23340
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
diff --git a/scripts/parallel_emerge.py b/scripts/parallel_emerge.py
index aa6876b..31865d7 100644
--- a/scripts/parallel_emerge.py
+++ b/scripts/parallel_emerge.py
@@ -17,6 +17,7 @@
import codecs
import copy
import errno
+import gc
import heapq
import multiprocessing
import os
@@ -24,6 +25,7 @@
import signal
import sys
import tempfile
+import threading
import time
import traceback
@@ -78,7 +80,6 @@
print
print "The --rebuild option rebuilds packages whenever their dependencies"
print "are changed. This ensures that your build is correct."
- sys.exit(1)
# Global start time
@@ -1551,7 +1552,23 @@
def main(argv):
+ try:
+ return real_main(argv)
+ finally:
+ # Work around multiprocessing sucking and not cleaning up after itself.
+ # http://bugs.python.org/issue4106;
+ # Step one; ensure GC is ran *prior* to the VM starting shutdown.
+ gc.collect()
+ # Step two; go looking for those threads and try to manually reap
+ # them if we can.
+ for x in threading.enumerate():
+ # Filter on the name, and ident; if ident is None, the thread
+ # wasn't started.
+ if x.name == 'QueueFeederThread' and x.ident is not None:
+ x.join(1)
+
+def real_main(argv):
parallel_emerge_args = argv[:]
deps = DepGraphGenerator()
deps.Initialize(parallel_emerge_args)
@@ -1559,10 +1576,10 @@
if emerge.action is not None:
argv = deps.ParseParallelEmergeArgs(argv)
- sys.exit(emerge_main(argv))
+ return emerge_main(argv)
elif not emerge.cmdline_packages:
Usage()
- sys.exit(1)
+ return 1
# Unless we're in pretend mode, there's not much point running without
# root access. We need to be able to install packages.
@@ -1572,7 +1589,7 @@
# dependency cache. This is important for performance.
if "--pretend" not in emerge.opts and portage.secpass < 2:
print "parallel_emerge: superuser access is required."
- sys.exit(1)
+ return 1
if "--quiet" not in emerge.opts:
cmdline_packages = " ".join(emerge.cmdline_packages)
@@ -1631,4 +1648,4 @@
os.execvp("sudo", args)
print "Done"
- sys.exit(0)
+ return 0