graphite-dev team mailing list archive

[Yee-Ting Li <yee379@xxxxxxxxx>]

hi Michael,

terribly sorry for the very late response. i actually reverted back to using whisper as i was unable to determine what was causing the issues: you're pointer on the inode exhaustion is a good one and i'll try it out….

so i have a new machine i can test on (same as the last one) and i'm trying to run megacarbon. i'm a little confused over the unified relay/write daemons. i expected that i could set a daemon with the following in daemon.conf:

PIPELINE = relay, write

but i get the following error when trying to start the instance:

carbon.conf.ConfigError: <ConfigError(daemon.conf: Exactly one 'write' or 'relay' must exist in PIPELINE)>

however, if i try to configure just a relay (like with 0.9.10), with

PIPELINE = relay

however, i get the following a different error:  

carbon.conf.ConfigError: <ConfigError(Missing expected configuration "CACHE_SIZE_LOW_WATERMARK")>

am i doing something wrong?

cheers,  


On Wednesday, 29 August 2012 at 10:23, Michael Leinartas wrote:

> Hey, sorry I missed this message over a month ago!
>  
> If the file doesn't exist then I'd guess it's trying to write out a
> new file - unlike Whisper, Ceres will continue to create new files for
> a metric as time goes on (hinted at by the timestamp in the name).
>  
> Have you been able to work through this issue? If not, my guess is
> that you're running out of inodes - While it's somewhat more space
> efficient than Whisper (not using space for points that aren't stored
> yet) it does end up with a lot more files. You should be able to
> confirm this by trying to touch a file on the same partition as a
> non-root user.
>  
> Let us know how it works out!
>  
> thanks
> michael
>  
> On Wed, Jul 11, 2012 at 1:03 AM, Yee-Ting Li <yee379@xxxxxxxxx (mailto:yee379@xxxxxxxxx)> wrote:
> > hi - great project :)
> >  
> > i'm using mleinart's graphite-megacarbon latest branch with ceres in a vain attempt to reduce the disk space that my data occupies; basically i'm pushing a bunch of network switch counters every 15 seconds.
> >  
> > so, i feed data into ceres and everything seems great. however, after a few hours, the logs give me a bunch of 'database write operation failed' errors. this is repeatable…. adding an exception string to the try/except shows:
> >  
> > "database write operation failed: (<type 'exceptions.IOError'>) [Errno 28] No space left on device: '/opt/graphite/storage/ceres/ptolemy/<REDACTED>/<REDACTED>/port_stats/rfc2863/unicast_pkts_out/1341947505@15.slice (mailto:1341947505@15.slice)'"
> >  
> > (the redacted bits is just my fully qualified name of my switch/router and port)
> >  
> > my disk is definitely not full. and in fact the file/slice does not exist.
> >  
> > i haven't had too much time to look into the ceres code yet, but i did a quick hack to try to re-create the database (basically running the database creation bit of write_cached_datapoints() again) if i get this error, but still nothing. and no exceptions from database.create()
> >  
> > after i stop the daemon, and start it again - all writes will report this same error and i get no new data into the system. a 'rm -rf' on the ceres store fixes things.
> >  
> > please forgive my attachment, but as they say, a picture shows a thousand words. rates are relative low for now - will likely want to goto about 50 times that in production - as long as i can actually write data continuously ;)
> >  
> > any ideas?
> >  
> > Yee.
> >  
> >  
> >  
> > _______________________________________________
> > Mailing list: https://launchpad.net/~graphite-dev
> > Post to : graphite-dev@xxxxxxxxxxxxxxxxxxx (mailto:graphite-dev@xxxxxxxxxxxxxxxxxxx)
> > Unsubscribe : https://launchpad.net/~graphite-dev
> > More help : https://help.launchpad.net/ListHelp
>  
>  
>  
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : graphite-dev@xxxxxxxxxxxxxxxxxxx (mailto:graphite-dev@xxxxxxxxxxxxxxxxxxx)
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp