graphite-dev team mailing list archive

[Steve Keller <question109514@xxxxxxxxxxxxxxxxxxxxx>]

New question #109514 on Graphite:
https://answers.launchpad.net/graphite/+question/109514

I'm running 0.9.6.  Not all data points that I send to carbon are getting into the whisper data files.  I suspected this because I saw holes in the graphs, so I devised a test by siphoning off about 1 1/2 hours of data from my data stream, and ran a second client against the siphoned file (it has ~52000 lines in it).  

carbon.conf is the default example file

storage-schemas.conf:

[opsview_stg]
priority = 100
pattern = ^staging\.*
retentions = 300:25920,900:70080
[opsview_bugtest]
priority = 99
pattern = ^bugtest\.*
retentions = 300:25920,900:70080

I ran my client against the data in the test file, and it seemed to complete fine, but not every data point was stored in whisper.  From my client's log, for example, the following lines were sent  (many more lines were sent of course, but this is the grepped output to show the problem):

bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271896572
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271896872
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271897172
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271897472
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271897772
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898072
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898372
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898672
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898972
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271899272
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271899572
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271899872
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271900172
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271900472
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271900772
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271901072
bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271901367

Yet when I look at the data in the whisper data file I see this:

whisper-fetch.py --from=1271896200 --until=1271901600 /opt/graphite/storage/whisper/bugtest/aibpdap1_mycompany_com/ssh_check_linux_disk_-opt/available.wsp
1271896500      None
1271896800      None
1271897100      None
1271897400      None
1271897700      None
1271898000      512.000000
1271898300      None
1271898600      None
1271898900      None
1271899200      512.000000
1271899500      None
1271899800      None
1271900100      None
1271900400      512.000000
1271900700      None
1271901000      None
1271901300      512.000000
1271901600      None

I suspect there is some weird buffer overrun going on, but I don't know what.  

To test this, I restricted the output to just data from the machines aibpdap1 and aibpdap2, added a third config to storage-schemas.conf

[opsview_liltest]
priority = 99
pattern = ^liltest\.*
retentions = 300:25920,900:70080

Result, the data got stored OK.

whisper-fetch.py --from=1271896200 --until=1271901600 /opt/graphite/storage/whisper/liltest/aibpdap1_mycompany_com/ssh_check_linux_disk_-opt/available.wsp
1271896500      512.000000
1271896800      512.000000
1271897100      512.000000
1271897400      512.000000
1271897700      512.000000
1271898000      512.000000
1271898300      512.000000
1271898600      512.000000
1271898900      512.000000
1271899200      512.000000
1271899500      512.000000
1271899800      512.000000
1271900100      512.000000
1271900400      512.000000
1271900700      512.000000
1271901000      512.000000
1271901300      512.000000
1271901600      None

My client is configured to send data to the carbon-cache daemon in groups of about 400 data lines.  I've tried reducing this to no avail, and tried sleeping between sends, also to no avail.

**** The following may be a related problem ****

I can't look at the logs because the only log file is console.log.  I see in the code where there should be others, but they don't appear.

Any help gratefully accepted.

Thanks,
Steve Keller

 

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.