graphite-dev team mailing list archive

[Ben Meier <question217341@xxxxxxxxxxxxxxxxxxxxx>]

New question #217341 on Graphite:
https://answers.launchpad.net/graphite/+question/217341

As part of a monitoring and metrics application we are using Carbon to store the various statistics generated by Diamond collectors and the monitored application. One of the decisions was that the entire application would run as a single process with the various daemons running as threads. This would hopefully help to make it perform the same on both Unix and Windows systems. (We have made some small modifications to allow carbon to operate on windows, but these changes don't affect the behaviour below.)

This all worked out fine in development and testing, with both carbon-cache and carbon-aggregator seemingly running fine and accepting data from diamond collectors, and graphite-web displaying the data appropriately. Unfortunately after running like this for more than a few minutes CPU usage quickly spikes to 100% (or more with multiple CPUs). Memory usage does not grow and everything seems to perform fine and work appropriately except for the high CPU usage. This does not happen when no data (neither carbon instrumentation nor diamond) is sent to carbon.

Is there any way to launch both daemons in a single python process? I know the reactor object is global and that there can only ever be one, but are there any other global conflicts between the two daemons that would cause this behaviour?

Are they perhaps sharing the same internal buffer and filling it exponentially via an aggregator-cache-aggregator-cache loop?

The code I am using to launch both daemons is as follows:
-------------------------------------------

. . . (relevant import and setup here)

daemons = ['carbon-cache', 'carbon-aggregator'] # settings.CARBON_DAEMONS

print("Carbon Daemons = %s" % daemons)work

topsvc = service.MultiService()

for program in daemons:
    twistd_options = ["--no_save", "--nodaemon", program]

    if settings.CARBON_CONFIG != None:
        twistd_options.append('--config='+settings.CARBON_CONFIG)

    config = ServerOptions()
    config.parseOptions(twistd_options)
    config['originalname'] = program

    plg = config.loadedPlugins[config.subCommand]
    ser = plg.makeService(config.subOptions)
    ser.setServiceParent(topsvc)

topsvc.startService()

from twisted.internet import reactor

reactor.run()

-------------------------------------------

This also behaves fine if I launch the daemons seperately using the same code and simply changing the 'daemons' variable to ['carbon-cache'] and ['carbon-aggregator'] respectively.

We are using the 0.9.x branch.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.