graphite-dev team mailing list archive

[Patrick O'Brien <question173171@xxxxxxxxxxxxxxxxxxxxx>]

New question #173171 on Graphite:
https://answers.launchpad.net/graphite/+question/173171

carbon.conf file:
[relay]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
RELAY_METHOD = rules
DESTINATIONS = 127.0.0.1:2005:np, 127.0.0.1:2007:prod
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True

relay-rules.conf
[np]
pattern = (?=(^((?!npvm).)*$))(?=.*(_np_|_dev_|_qa_))
servers = 127.0.0.1:2005

# You must have exactly one section with 'default = true'
[prod]
default = true
servers = 127.0.0.1:2007

When I try to run carbon-relay.py:

Starting carbon-relay (instance a)
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-relay.py", line 30, in <module>
    run_twistd_plugin(__file__)
  File "/opt/graphite/lib/carbon/util.py", line 81, in run_twistd_plugin
    runApp(config)
  File "/usr/lib64/python2.6/site-packages/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 411, in run
    self.application = self.createOrGetApplication()
  File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 489, in createOrGetApplication
    ser = plg.makeService(self.config.subOptions)
  File "/opt/graphite/lib/twisted/plugins/carbon_relay_plugin.py", line 21, in makeService
    return service.createRelayService(options)
  File "/opt/graphite/lib/carbon/service.py", line 177, in createRelayService
    router = RelayRulesRouter(settings["relay-rules"])
  File "/opt/graphite/lib/carbon/routers.py", line 24, in __init__
    self.rules = loadRelayRules()
TypeError: loadRelayRules() takes exactly 1 argument (0 given)

This happens even if I specify the rules file with --rules

If I look at routers.py here is the offending line:

    self.rules = loadRelayRules()

So I hard code in the path to my rules files and try to start carbon-relay:

Starting carbon-relay (instance a)
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-relay.py", line 30, in <module>
    run_twistd_plugin(__file__)
  File "/opt/graphite/lib/carbon/util.py", line 81, in run_twistd_plugin
    runApp(config)
  File "/usr/lib64/python2.6/site-packages/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 411, in run
    self.application = self.createOrGetApplication()
  File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 489, in createOrGetApplication
    ser = plg.makeService(self.config.subOptions)
  File "/opt/graphite/lib/twisted/plugins/carbon_relay_plugin.py", line 21, in makeService
    return service.createRelayService(options)
  File "/opt/graphite/lib/carbon/service.py", line 177, in createRelayService
    router = RelayRulesRouter(settings["relay-rules"])
  File "/opt/graphite/lib/carbon/routers.py", line 24, in __init__
    self.rules = loadRelayRules(self.rules_path)
  File "/opt/graphite/lib/carbon/relayrules.py", line 26, in loadRelayRules
    "'destinations' list" % (path, section))
ValueError: Rules file /opt/graphite/conf/relay-rules.conf section np does not define a 'destinations' list

Ok, I did not know destinations was supposed to be in the relay-rules.conf, I thought that was supposed to be in carbon.conf. Anyway, just to see what happens, I add destinations to the relay-rules.conf file so it looks like this:

[np]
pattern = (?=(^((?!npvm).)*$))(?=.*(_np_|_dev_|_qa_))
servers = 127.0.0.1:2005
destinations = 127.0.0.1:2005:np

# You must have exactly one section with 'default = true'
[prod]
default = true
servers = 127.0.0.1:2007
destinations = 127.0.0.1:2007:prod

I also tried this:

[np]
pattern = (?=(^((?!npvm).)*$))(?=.*(_np_|_dev_|_qa_))
servers = 127.0.0.1:2005
destinations = 127.0.0.1:2005

# You must have exactly one section with 'default = true'
[prod]
default = true
servers = 127.0.0.1:2007
destinations = 127.0.0.1:2007

With both of those in place, carbon-relay finally starts:
Starting carbon-relay (instance a)

here are the logs:
console.log for carbon-relay-a:
03/10/2011 21:46:52 :: Log opened.
03/10/2011 21:46:52 :: twistd 8.2.0 (/usr/bin/python 2.6.5) starting up.
03/10/2011 21:46:52 :: reactor class: twisted.internet.epollreactor.EPollReactor.
03/10/2011 21:46:52 :: twisted.internet.protocol.ServerFactory starting on 2003
03/10/2011 21:46:52 :: Starting factory <twisted.internet.protocol.ServerFactory instance at 0x10ed7a0>
03/10/2011 21:46:52 :: twisted.internet.protocol.ServerFactory starting on 2004
03/10/2011 21:46:52 :: Starting factory <twisted.internet.protocol.ServerFactory instance at 0x112a0e0>
03/10/2011 21:46:52 :: Starting factory CarbonClientFactory(127.0.0.1:2007:prod)
03/10/2011 21:46:52 :: Starting factory CarbonClientFactory(127.0.0.1:2005:np)
03/10/2011 21:47:52 :: Unhandled error in Deferred:
03/10/2011 21:47:52 :: Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/twisted/internet/base.py", line 1128, in run
    self.mainLoop()
  File "/usr/lib64/python2.6/site-packages/twisted/internet/base.py", line 1137, in mainLoop
    self.runUntilCurrent()
  File "/usr/lib64/python2.6/site-packages/twisted/internet/base.py", line 757, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/lib64/python2.6/site-packages/twisted/internet/task.py", line 114, in __call__
    d = defer.maybeDeferred(self.f, *self.a, **self.kw)
--- <exception caught here> ---
  File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 106, in maybeDeferred
    result = f(*args, **kw)
  File "/opt/graphite/lib/carbon/instrumentation.py", line 100, in recordMetrics
    prefix = 'destinations.%s.' % receiverProto.destinationName
exceptions.AttributeError: MetricLineReceiver instance has no attribute 'destinationName'

carbon-relay is accepting metrics, the individual carbon-cache processes are running and configured correctly and I think I've followed everything from the documentation, so I have no idea what I am doing wrong. We got to about 400,000 metrics/minute off of a single carbon-cache instance before stuff starting hitting the fan so it seems like we'll need to roll out a few more servers and instances to keep up with the data input.

Any insight or advice would be much appreciated.


-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.