graphite-dev team mailing list archive

Thread
Date
[Question #290796]: Aggregation rule and non aggregated data rules

To: graphite-dev@xxxxxxxxxxxxxxxxxxx
From: Regnoult <question290796@xxxxxxxxxxxxxxxxxxxxx>
Date: Wed, 13 Apr 2016 14:32:40 -0000
Reply-to: question290796@xxxxxxxxxxxxxxxxxxxxx
Sender: bounces@xxxxxxxxxxxxx
New question #290796 on Graphite:
https://answers.launchpad.net/graphite/+question/290796

Hi,

I'm testing the aggregator and something is unexpected.
Here is my aggregation rule:
<env>.<system>.<host>.<app>.<metrics>.sum_10 (10) = sum <env>.<system>.<host>.<app>.<<metrics>>.raw$

Now I've added this line in carbon/clients.py:321
def sendDatapoint(self, metric, datapoint):
    for destination in self.router.getDestinations(metric):
->  log.clients("Sending metric to destination %s" % (metric))
      self.client_factories[destination].sendDatapoint(metric, datapoint)
and carbon/aggregator/buffer.py:69:
        state.events.metricGenerated(self.metric_path, datapoint)
->    log.aggregator("Metric generated %s" % self.metric_path)
        state.instrumentation.increment('aggregateDatapointsSent')


I'm sending these 2 every 5 secs (in python):
"test.bash.stats", random.randint(0,100)
"TEST.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw", random.randint(0,100)

And here are the logs when starting my aggregator in DEBUG:
13/04/2016 15:11:36 :: [console] Log opened.
13/04/2016 15:11:36 :: [console] twistd 16.0.0 (/usr/bin/python 2.7.6) starting up.
13/04/2016 15:11:36 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
13/04/2016 15:11:36 :: [console] CarbonReceiverFactory starting on 2023
13/04/2016 15:11:36 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f1533a7c908>
13/04/2016 15:11:36 :: [console] CarbonReceiverFactory starting on 2024
13/04/2016 15:11:36 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f1533a7c7e8>
13/04/2016 15:11:36 :: [console] Starting factory CarbonClientFactory(10.33.21.38:2014:None)
13/04/2016 15:11:36 :: [clients] CarbonClientFactory(10.33.21.38:2014:None)::startedConnecting (10.33.21.38:2014)
13/04/2016 15:11:36 :: [clients] CarbonClientProtocol(10.33.21.38:2014:None)::connectionMade
13/04/2016 15:12:23 :: [listener] MetricPickleReceiver connection with 10.33.17.109:55986 established
13/04/2016 15:12:23 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:23 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:23 :: [aggregator] Allocating new metric buffer for LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:23 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:28 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:28 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:28 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:33 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:33 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:33 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:33 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:33 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:33 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:38 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:38 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:38 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:43 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:43 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:43 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:43 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:43 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:43 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:48 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:48 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:48 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:53 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:53 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:53 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:12:53 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:53 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:53 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw
13/04/2016 15:12:58 :: [console] Couldn't match metric test.bash.stats with any aggregation rule. Passing on un-aggregated.
13/04/2016 15:12:58 :: [clients] Sending metric to destination test.bash.stats
13/04/2016 15:12:58 :: [clients] Sending metric to destination LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.raw

1. raw data is being passed over. Isn't the aggregator supposed to retain these values and pass them?
2. This line shows that the output of an aggregation is being re-aggregated:
13/04/2016 15:12:53 :: [console] Couldn't match metric LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10 with any aggregation rule. Passing on un-aggregated.

So I want to send the least amount of data possible, so just the aggregated output calculated every X seconds. I've seen the config item: FORWARD_ALL = True
But if I switch it to False, here are the logs:

13/04/2016 15:26:46 :: [console] twistd 16.0.0 (/usr/bin/python 2.7.6) starting up.
13/04/2016 15:26:46 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
13/04/2016 15:26:46 :: [console] CarbonReceiverFactory starting on 2023
13/04/2016 15:26:46 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f74fdf1f758>
13/04/2016 15:26:46 :: [console] CarbonReceiverFactory starting on 2024
13/04/2016 15:26:46 :: [console] Starting factory <carbon.service.CarbonReceiverFactory instance at 0x7f74fdf1f638>
13/04/2016 15:26:46 :: [console] Starting factory CarbonClientFactory(10.33.21.38:2014:None)
13/04/2016 15:26:46 :: [clients] CarbonClientFactory(10.33.21.38:2014:None)::startedConnecting (10.33.21.38:2014)
13/04/2016 15:26:46 :: [clients] CarbonClientProtocol(10.33.21.38:2014:None)::connectionMade
13/04/2016 15:26:52 :: [listener] MetricPickleReceiver connection with 10.33.17.109:57884 established
13/04/2016 15:26:52 :: [aggregator] Allocating new metric buffer for LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:02 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:12 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:22 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:32 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:42 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:27:52 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10
13/04/2016 15:28:02 :: [aggregator] Metric generated LIVE.boxA.something.applicationXX.QUERY_TIMES.ODB.CURSSQL.sum_10

As you can see nothing is being sent. Not the test.bash.stats, nor .raw, nor .sum10
Is there something I am doing wrong or I didn't understand the purpose of the aggregator?

Regards
Francois



-- 
You received this question notification because your team graphite-dev
is an answer contact for Graphite.