system_log_manager: Refactoring SystemLogManager and related arguments

This CL does these things:
1. Refactors SystemLogManager so it will always create a thread to scan
logs periodically. In each scan it will clear logs, and optionally sync
logs to server if the time difference between last sync and the current
time is larger then sync_log_period_secs.

2. SystemLogManger now provides two function for goofy to call.
KickToSync will kick SystemLogManager thread to sync logs. KickToClear
will kick SystemLogManager thread to clear logs.

3. Changes test list options:
  a. scan_log_period_secs: The period to scan logs including clearing
  logs and optionally sync logs.
  b. sync_log_period_secs: The period to sync logs. If it is None, then
  background log syncing is disabled.
  c. enable_sync_logs: If it is True, goofy will call KickToSync when
  goofy think there is important event like low battery of disk full.
  This parameter is not passed to SystemLogManager.

4. Change generic test list argument to demonstrate the use case where
background syncing log is done every 10 minuts, foreground syncing log
is enabled, and clearing log is done every 2 minutes.

5. Adds a CatchException decorator in debug_utils. It is useful when a
manager wants to suppress its exception in its own thread without
bothering the user(goofy).

BUG=chrome-os-partner:21904
TEST=unittest

Change-Id: I710fbd6d2f31d99dade0c1d532512babd80ac5af
Reviewed-on: https://chromium-review.googlesource.com/182037
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
diff --git a/py/goofy/goofy.py b/py/goofy/goofy.py
index a4fa28d..5477966 100755
--- a/py/goofy/goofy.py
+++ b/py/goofy/goofy.py
@@ -290,7 +290,7 @@
       self.log_watcher = None
     if self.system_log_manager:
       if self.system_log_manager.IsThreadRunning():
-        self.system_log_manager.StopSyncThread()
+        self.system_log_manager.Stop()
       self.system_log_manager = None
     if self.prespawner:
       logging.info('Stopping prespawner')
@@ -1400,16 +1400,15 @@
     if self.test_list.options.sync_event_log_period_secs:
       self.log_watcher.StartWatchThread()
 
-    # Note that we create a system log manager even if
-    # sync_log_period_secs isn't set (no background
-    # syncing), since we may kick it to sync logs in its
-    # thread.
-    if self.test_list.options.enable_sync_log:
-      self.system_log_manager = SystemLogManager(
-        sync_log_paths=self.test_list.options.sync_log_paths,
-        sync_period_sec=self.test_list.options.sync_log_period_secs,
-        clear_log_paths=self.test_list.options.clear_log_paths)
-      self.system_log_manager.StartSyncThread()
+    # Creates a system log manager to scan logs periocially.
+    # A scan includes clearing logs and optionally syncing logs if
+    # enable_syng_log is True. We kick it to sync logs.
+    self.system_log_manager = SystemLogManager(
+      sync_log_paths=self.test_list.options.sync_log_paths,
+      sync_log_period_secs=self.test_list.options.sync_log_period_secs,
+      scan_log_period_secs=self.test_list.options.scan_log_period_secs,
+      clear_log_paths=self.test_list.options.clear_log_paths)
+    self.system_log_manager.Start()
 
     self.update_system_info()
 
@@ -1726,8 +1725,8 @@
                              charger_connected=ac_present,
                              critical=critical_low_battery)
           self.log_watcher.KickWatchThread()
-          if self.system_log_manager:
-            self.system_log_manager.KickSyncThread()
+          if self.test_list.options.enable_sync_log:
+            self.system_log_manager.KickToSync()
     except: # pylint: disable=W0702
       logging.exception('Unable to check battery or notify shopfloor')
     finally:
@@ -1755,8 +1754,8 @@
       self.log_watcher.KickWatchThread()
 
       # Syncs files to server
-      if self.system_log_manager:
-        self.system_log_manager.KickSyncThread(
+      if self.test_list.options.enable_sync_log:
+        self.system_log_manager.KickToSync(
             core_dump_files, self.core_dump_manager.ClearFiles)
 
   def check_log_rotation(self):