cbuildbot should complain when exceptions don't fail the build.

If the build succeeds, but an unhandled exception is thrown anyway, this
indicates a bug in cbuildbot. We should complain about this and print a
traceback.

This helps when debugging unforeseen bugs in cbuildbot. For example, we had
a bug where a stage was incorrectly throwing an exception, and the failure
was being suppressed due to a bug in cgroups.

I've also fixed cbuildbot_stages to not throw exceptions for errors that
were forgiven.

BUG=chromium-os:29170
TEST=pylint, unit tests, example run with fake exception added.

Change-Id: I239e3a1ff57258e4c16543b8a5a60b2af3002c51
Reviewed-on: https://gerrit.chromium.org/gerrit/19857
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
diff --git a/scripts/cbuildbot.py b/scripts/cbuildbot.py
index 4a126b2..9f1ba17 100644
--- a/scripts/cbuildbot.py
+++ b/scripts/cbuildbot.py
@@ -299,6 +299,7 @@
   def Run(self):
     """Main runner for this builder class.  Runs build and prints summary."""
     print_report = True
+    exception_thrown = False
     success = True
     try:
       self.Initialize()
@@ -316,7 +317,9 @@
         success = self._ReExecuteInBuildroot(sync_instance)
       else:
         self.RunStages()
-
+    except Exception:
+      exception_thrown = True
+      raise
     finally:
       if print_report:
         self._WriteCheckpoint()
@@ -324,6 +327,12 @@
         results_lib.Results.Report(sys.stdout, self.archive_urls,
                                    self.release_tag)
         success = results_lib.Results.BuildSucceededSoFar()
+        if exception_thrown and success:
+          success = False
+          print >> sys.stderr, """
+@@@STEP_FAILURE@@@
+Exception thrown, but all stages marked successful. This is an internal error,
+because the stage that threw the exception should be marked as failing."""
 
     return success