graphite-dev team mailing list archive

[jdoe12345 <question252717@xxxxxxxxxxxxxxxxxxxxx>]

New question #252717 on Graphite:
https://answers.launchpad.net/graphite/+question/252717

Hi all,

   I have one host setup as my graphite installation (one relay, 2 writers). Everything was working peachy until I increased the number of open files allowed for the account running the carbon daemons to open more files (32k). I was getting EMFILE in my relay console.log, hence the change.

After I restart the carbon daemons (relay and two writers), after about 20 minutes (metrics are written to disk fine), zero metrics are written, and I cannot see them in the web interface. The relay is still receiving metrics (verified via strace), but I don't see the writers writing any data.

I've enabled write logging on the writers, and they even say that writes are being made to the writer logfile. 

However, if I use whisper-dump.py, the last recorded metric is within that 20 minute window mentioned above. I monitor the disks with iostat, and after that 20 minute period, everything goes dark.

I am willing to bet this has to do with open files, but I'm just stuck on what to look for, or better ways to debug the carbon relay/writers.

Anyone have any idea?

THanks!

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.