← Back to team overview

openstack team mailing list archive

Re: [Swift] rsyslog daemon reloading causes swift related services hangs and CPU reach to 100%

 

nice suggestion ,
We decide to use UPD now

2012/6/22 MORITA Kazutaka <morita.kazutaka@xxxxxxxxx>

> At Fri, 22 Jun 2012 00:00:26 +0800,
> Kuo Hugo wrote:
> >
> > Hi folks ,
> >
> > We're facing an issue related to the bug as below
> >
> > /dev/log rotations can cause object-server failures
> >
> https://bugs.launchpad.net/swift/+bug/780025
> >
> > My Swift version : 1.4.9
> >
> > But I found that not only object-server but also all swift related
> workers
> > those log through rsyslog.
> > There's a easy way to reproduce it ,
> > 1. Run swift-bench
> > 2. restart/stop rsyslog during swift-bench progress....
> >
> > You can see that all CPU usage reach to 100%
> >
> > Should it be an additional bug ? If so , I can file it .
> >
> > Is there anyway to improve this behavior ? I expect that all swift
> workers
> > should keep working even though that rsyslog dead or restart.
>
> I've faced with the same problem and found that it was a bug of the
> python logging module.  I think the following patch against the module
> would solve the problem.
>
> diff --git a/logging/handlers.py b/logging/handlers.py
> index 756baf0..d2a042a 100644
> --- a/logging/handlers.py
> +++ b/logging/handlers.py
> @@ -727,7 +727,11 @@ class SysLogHandler(logging.Handler):
>         except socket.error:
>             self.socket.close()
>             self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> -            self.socket.connect(address)
> +            try:
> +                self.socket.connect(address)
> +            except socket.error:
> +                self.socket.close()
> +                raise
>
>     # curious: when talking to the unix-domain '/dev/log' socket, a
>     #   zero-terminator seems to be required.  this string is placed
>



-- 
+Hugo Kuo+
tonytkdk@xxxxxxxxx
+ <tonytkdk@xxxxxxxxx>886 935004793

Follow ups

References