graphite-dev team mailing list archive

[Regnoult <question290796@xxxxxxxxxxxxxxxxxxxxx>]

Question #290796 on Graphite changed:
https://answers.launchpad.net/graphite/+question/290796

Description changed to:
Hi,

I'm testing the aggregator and something is unexpected.
Here is my aggregation rule:
<env>.<system>.<host>.<app>.<metrics>.sum_10 (10) = sum <env>.<system>.<host>.<app>.<<metrics>>.raw$

Now I've added this line in carbon/clients.py:321
def sendDatapoint(self, metric, datapoint):
    for destination in self.router.getDestinations(metric):
->  log.clients("Sending metric to destination %s" % (metric))
      self.client_factories[destination].sendDatapoint(metric, datapoint)
and carbon/aggregator/buffer.py:69:
        state.events.metricGenerated(self.metric_path, datapoint)
->    log.aggregator("Metric generated %s" % self.metric_path)
        state.instrumentation.increment('aggregateDatapointsSent')


I'm sending these 2 every 5 secs (in python):
"test.bash.stats", random.randint(0,100)
"TEST.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw", random.randint(0,100)

And here are the logs when starting my aggregator in DEBUG:
-------------------------------------------------
13/04/2016 15:11:36 :: [console] Log opened.
13/04/2016 15:11:36 :: [console] twistd 16.0.0 (/usr/bin/python 2.7.6) starting up.
13/04/2016 15:11:36 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
13/04/2016 15:11:36 :: [console] CarbonReceiverFactory starting on 2023
13/04/2016 15:11:36 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f1533a7c908>
13/04/2016 15:11:36 :: [console] CarbonReceiverFactory starting on 2024
13/04/2016 15:11:36 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f1533a7c7e8>
13/04/2016 15:11:36 :: [console] Starting factory CarbonClientFactory(10.33.21.38:2014:None)
13/04/2016 15:11:36 :: [clients] CarbonClientFactory(XX.XX.XX.XX:2014:None)::startedConnecting (XX.XX.XX.XX:2014)
13/04/2016 15:11:36 :: [clients] CarbonClientProtocol(XX.XX.XX.XX::2014:None)::connectionMade
13/04/2016 15:12:23 :: [listener] MetricPickleReceiver connection with YY.YY.YY.YY:55986 established
13/04/2016 15:12:23 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:23 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:23 :: [aggregator] Allocating new metric buffer for LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:23 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:28 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:28 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:28 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:33 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:33 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:33 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:33 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:33 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:33 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:38 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:38 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:38 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:43 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:43 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:43 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:43 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:43 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:43 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:48 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:48 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:48 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:53 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:53 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:53 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:53 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:53 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:53 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:58 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:58 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:58 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
-------------------------------------------------

1. raw data is being passed over. Isn't the aggregator supposed to retain these values and pass them?
2. This line shows that the output of an aggregation is being re-aggregated:
13/04/2016 15:12:53 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.

So I want to send the least amount of data possible, so just the aggregated output calculated every X seconds. I've seen the config item: FORWARD_ALL = True
But if I switch it to False, here are the logs:
-------------------------------------------------
13/04/2016 15:26:46 :: [console] twistd 16.0.0 (/usr/bin/python 2.7.6) starting up.
13/04/2016 15:26:46 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
13/04/2016 15:26:46 :: [console] CarbonReceiverFactory starting on 2023
13/04/2016 15:26:46 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f74fdf1f758>
13/04/2016 15:26:46 :: [console] CarbonReceiverFactory starting on 2024
13/04/2016 15:26:46 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f74fdf1f638>
13/04/2016 15:26:46 :: [console] Starting factory CarbonClientFactory(10.33.21.38:2014:None)
13/04/2016 15:26:46 :: [clients] CarbonClientFactory(XX.XX.XX.XX::2014:None)::startedConnecting (XX.XX.XX.XX::2014)
13/04/2016 15:26:46 :: [clients] CarbonClientProtocol(XX.XX.XX.XX::2014:None)::connectionMade
13/04/2016 15:26:52 :: [listener] MetricPickleReceiver connection with YY.YY.YY.YY:57884 established
13/04/2016 15:26:52 :: [aggregator] Allocating new metric buffer for LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:02 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:12 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:22 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:32 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:42 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:52 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:28:02 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
-------------------------------------------------

As you can see nothing is being sent. Not the test.bash.stats, nor .raw, nor .sum10
Is there something I am doing wrong or I didn't understand the purpose of the aggregator?

Regards
Francois

-- 
You received this question notification because your team graphite-dev
is an answer contact for Graphite.