graphite-dev team mailing list archive

[Yee-Ting Li <yee379@xxxxxxxxx>]

hi - great project :)

i'm using mleinart's graphite-megacarbon latest branch with ceres in a vain attempt to reduce the disk space that my data occupies; basically i'm pushing a bunch of network switch counters every 15 seconds.

so, i feed data into ceres and everything seems great. however, after a few hours, the logs give me a bunch of 'database write operation failed' errors. this is repeatable…. adding an exception string to the try/except shows:

"database write operation failed: (<type 'exceptions.IOError'>) [Errno 28] No space left on device: '/opt/graphite/storage/ceres/ptolemy/<REDACTED>/<REDACTED>/port_stats/rfc2863/unicast_pkts_out/1341947505@15.slice'"

(the redacted bits is just my fully qualified name of my switch/router and port)

my disk is definitely not full. and in fact the file/slice does not exist.

i haven't had too much time to look into the ceres code yet, but i did a quick hack to try to re-create the database (basically running the database creation bit of write_cached_datapoints() again) if i get this error, but still nothing. and no exceptions from database.create()

after i stop the daemon, and start it again - all writes will report this same error and i get no new data into the system. a 'rm -rf' on the ceres store fixes things.

please forgive my attachment, but as they say, a picture shows a thousand words. rates are relative low for now - will likely want to goto about 50 times that in production - as long as i can actually write data continuously ;)

any ideas?

Yee.

Attachment: Screen Shot 2012-07-10 at 22.51.24.png
Description: PNG image