graphite-dev team mailing list archive

[Michael Leinartas <mleinartas@xxxxxxxxx>]

Hey, sorry I missed this message over a month ago!

If the file doesn't exist then I'd guess it's trying to write out a
new file - unlike Whisper, Ceres will continue to create new files for
a metric as time goes on (hinted at by the timestamp in the name).

Have you been able to work through this issue? If not, my guess is
that you're running out of inodes - While it's somewhat more space
efficient than Whisper (not using space for points that aren't stored
yet) it does end up with a lot more files. You should be able to
confirm this by trying to touch a file on the same partition as a
non-root user.

Let us know how it works out!

thanks
michael

On Wed, Jul 11, 2012 at 1:03 AM, Yee-Ting Li <yee379@xxxxxxxxx> wrote:
> hi - great project :)
>
> i'm using mleinart's graphite-megacarbon latest branch with ceres in a vain attempt to reduce the disk space that my data occupies; basically i'm pushing a bunch of network switch counters every 15 seconds.
>
> so, i feed data into ceres and everything seems great. however, after a few hours, the logs give me a bunch of 'database write operation failed' errors. this is repeatable…. adding an exception string to the try/except shows:
>
> "database write operation failed: (<type 'exceptions.IOError'>) [Errno 28] No space left on device: '/opt/graphite/storage/ceres/ptolemy/<REDACTED>/<REDACTED>/port_stats/rfc2863/unicast_pkts_out/1341947505@15.slice'"
>
> (the redacted bits is just my fully qualified name of my switch/router and port)
>
> my disk is definitely not full. and in fact the file/slice does not exist.
>
> i haven't had too much time to look into the ceres code yet, but i did a quick hack to try to re-create the database (basically running the database creation bit of write_cached_datapoints() again) if i get this error, but still nothing. and no exceptions from database.create()
>
> after i stop the daemon, and start it again - all writes will report this same error and i get no new data into the system. a 'rm -rf' on the ceres store fixes things.
>
> please forgive my attachment, but as they say, a picture shows a thousand words. rates are relative low for now - will likely want to goto about 50 times that in production - as long as i can actually write data continuously ;)
>
> any ideas?
>
> Yee.
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to     : graphite-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help   : https://help.launchpad.net/ListHelp
>