openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #13582
Re: [Swift] rsyslog daemon reloading causes swift related services hangs and CPU reach to 100%
nice suggestion ,
We decide to use UPD now
2012/6/22 MORITA Kazutaka <morita.kazutaka@xxxxxxxxx>
> At Fri, 22 Jun 2012 00:00:26 +0800,
> Kuo Hugo wrote:
> >
> > Hi folks ,
> >
> > We're facing an issue related to the bug as below
> >
> > /dev/log rotations can cause object-server failures
> >
> https://bugs.launchpad.net/swift/+bug/780025
> >
> > My Swift version : 1.4.9
> >
> > But I found that not only object-server but also all swift related
> workers
> > those log through rsyslog.
> > There's a easy way to reproduce it ,
> > 1. Run swift-bench
> > 2. restart/stop rsyslog during swift-bench progress....
> >
> > You can see that all CPU usage reach to 100%
> >
> > Should it be an additional bug ? If so , I can file it .
> >
> > Is there anyway to improve this behavior ? I expect that all swift
> workers
> > should keep working even though that rsyslog dead or restart.
>
> I've faced with the same problem and found that it was a bug of the
> python logging module. I think the following patch against the module
> would solve the problem.
>
> diff --git a/logging/handlers.py b/logging/handlers.py
> index 756baf0..d2a042a 100644
> --- a/logging/handlers.py
> +++ b/logging/handlers.py
> @@ -727,7 +727,11 @@ class SysLogHandler(logging.Handler):
> except socket.error:
> self.socket.close()
> self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> - self.socket.connect(address)
> + try:
> + self.socket.connect(address)
> + except socket.error:
> + self.socket.close()
> + raise
>
> # curious: when talking to the unix-domain '/dev/log' socket, a
> # zero-terminator seems to be required. this string is placed
>
--
+Hugo Kuo+
tonytkdk@xxxxxxxxx
+ <tonytkdk@xxxxxxxxx>886 935004793
Follow ups
References