openstack team mailing list archive

[MORITA Kazutaka <morita.kazutaka@xxxxxxxxx>]

At Fri, 22 Jun 2012 00:00:26 +0800,
Kuo Hugo wrote:
> 
> Hi folks ,
> 
> We're facing an issue related to the bug as below
> 
> /dev/log rotations can cause object-server failures
>                                 https://bugs.launchpad.net/swift/+bug/780025
> 
> My Swift version : 1.4.9
> 
> But I found that not only object-server but also all swift related workers
> those log through rsyslog.
> There's a easy way to reproduce it ,
> 1. Run swift-bench
> 2. restart/stop rsyslog during swift-bench progress....
> 
> You can see that all CPU usage reach to 100%
> 
> Should it be an additional bug ? If so , I can file it .
> 
> Is there anyway to improve this behavior ? I expect that all swift workers
> should keep working even though that rsyslog dead or restart.

I've faced with the same problem and found that it was a bug of the
python logging module.  I think the following patch against the module
would solve the problem.

diff --git a/logging/handlers.py b/logging/handlers.py
index 756baf0..d2a042a 100644
--- a/logging/handlers.py
+++ b/logging/handlers.py
@@ -727,7 +727,11 @@ class SysLogHandler(logging.Handler):
         except socket.error:
             self.socket.close()
             self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-            self.socket.connect(address)
+            try:
+                self.socket.connect(address)
+            except socket.error:
+                self.socket.close()
+                raise

     # curious: when talking to the unix-domain '/dev/log' socket, a
     #   zero-terminator seems to be required.  this string is placed