graphite-dev team mailing list archive

[Rodion Vynnychenko <question201602@xxxxxxxxxxxxxxxxxxxxx>]

New question #201602 on Graphite:
https://answers.launchpad.net/graphite/+question/201602

I'm trying to verify that graphite will handle 100k metrics per minute properly, so I've made a script posting 1666 metrics per second to a single carbon-cache instance on localhost.

The problem is that creates are really slow, I started running this script  every second "find /mnt/whisper/ -type f | wc -l >> carbon-counts" and it turned out that only about 10 new metrics are created. At the same time "iostat -dxk 10"  shows:

ubuntu@ip-10-64-153-96:~$ iostat -dxk 10
Linux 3.2.0-23-virtual (ip-10-64-153-96) 	06/27/2012 	_x86_64_	(1 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.32    0.59    1.01     8.11    29.78    47.25     0.07   43.87   23.65   55.72   3.33   0.53
xvdb              0.00   123.70    0.10  136.45     0.34  1040.59    15.25    27.98  204.87    6.85  205.01   1.24  17.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.21    0.00    0.62     0.00     3.32    10.67     0.00    1.33    0.00    1.33   0.67   0.04
xvdb              0.00   301.04    0.00   61.89     0.00  1451.71    46.91     7.12  114.97    0.00  114.97   1.27   7.85

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.52    0.00    1.14     0.00    12.84    22.55     0.00    1.09    0.00    1.09   0.73   0.08
xvdb              0.00   274.22    0.00   31.78     0.00  1224.02    77.03     1.16   36.56    0.00   36.56   0.47   1.49

So there doesn't seem to be any substantial disk activity. 

MAX_CACHE_SIZE is set to 1000000, MAX_CREATES_PER_MINUTE = inf. Top shows this:

ubuntu@ip-10-64-153-96:~$ top | grep carbon-cache
 1398 root      20   0  460m 378m 1968 S 38.4 10.1  40:52.62 carbon-cache.py                                                                                                                                                          
15838 root      20   0     8    4    0 R  3.8  0.0   0:00.02 carbon-cache.py                                                                                                                                                          
 1398 root      20   0  460m 378m 1968 S 58.8 10.1  40:54.39 carbon-cache.py                                                                                                                                                          
 1398 root      20   0  460m 378m 1968 S 64.7 10.1  40:56.34 carbon-cache.py                                                                                                                                                          
 1398 root      20   0  460m 378m 1968 S 55.7 10.1  40:58.02 carbon-cache.py           

This is running on a ec2.medium instance with 1 CPU and 4GB memory. 

How do I increase the rate at which creates happen?

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.