graphite-dev team mailing list archive

[ziggy <question135957@xxxxxxxxxxxxxxxxxxxxx>]

Question #135957 on Graphite changed:
https://answers.launchpad.net/graphite/+question/135957

    Status: Solved => Open

ziggy is still having a problem:
Something weird is going on with this.  After getting SIGTERM, I see
carbon stop listening for new metrics, but it sometimes takes a _long_
time to write out the remaining queue to disk and then exit.  I've seen
similar behavior other times where it is lightly loaded but will not
write data to disk.  Here are log messages from console.log highlighting
the behavior:

07/12/2010 07:54:51 :: Received SIGTERM, shutting down.
07/12/2010 07:54:51 :: (Port 2003 Closed)
07/12/2010 07:54:51 :: Stopping factory <twisted.internet.protocol.Factory instance at 0x2b0fa158d128>
07/12/2010 07:54:51 :: (Port 2004 Closed)
07/12/2010 07:54:51 :: Stopping factory <twisted.internet.protocol.Factory instance at 0x2b0fa2b955a8>
07/12/2010 07:54:52 :: (Port 7002 Closed)
07/12/2010 07:54:52 :: Stopping factory <twisted.internet.protocol.Factory instance at 0x2b0fa2b95878>
07/12/2010 07:54:56 :: Sorted 21877 cache queues in 0.017809 seconds
07/12/2010 07:54:57 :: Sorted 21877 cache queues in 0.018691 seconds
...
07/12/2010 08:00:18 :: Sorted 21582 cache queues in 0.021431 seconds
07/12/2010 08:00:20 :: Sorted 21582 cache queues in 0.021100 seconds
...
07/12/2010 08:39:51 :: Sorted 19222 cache queues in 0.018025 seconds
07/12/2010 08:39:53 :: Sorted 19222 cache queues in 0.018327 seconds
07/12/2010 08:39:55 :: Sorted 19222 cache queues in 0.018079 seconds
...
07/12/2010 08:59:01 :: Sorted 18101 cache queues in 0.017155 seconds
07/12/2010 08:59:03 :: Sorted 18101 cache queues in 0.016907 seconds

and over an hour after sending SIGTERM it still is burning CPU (top
shows ~50%) and making very slow progress writing the queue to disk and
exiting.  Any ideas what is going on?

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.