graphite-dev team mailing list archive

[Kurt <question246825@xxxxxxxxxxxxxxxxxxxxx>]

New question #246825 on Graphite:
https://answers.launchpad.net/graphite/+question/246825

I am having an issue with graphite/carbon.
     
I am seeing errors in the webapp and carbon-cache log. I am using carbon-relay to relay to two local instances of carbon cache. I am running version 0.9.10.
     
    carbon-relay.conf contains the following:
    DESTINATIONS = 127.0.0.1:2200:a, 127.0.0.1:2201:b
    RELAY_METHOD = consistent-hashing
    REPLICATION_FACTOR = 1
     
    and local_settings.py has:
    CARBONLINK_HOSTS = ["127.0.0.1:7000:a", "127.0.0.1:7001:b"]
     
FWIW, CARBONLINK_HOSTS was misconfigured, pointing at the pickle receiver ports, which was causing connection lost errors in the webapp logs. Fixed that, then I stated getting errors shown below. I am not sure if this is a hash ring issue, or maybe an issue due to old code that would be remedied by running new code. Either way, I would like to get this optimized.
     
    ----Begin Log Output----
     
    /opt/graphite/storage/log/carbon-cache/carbon-cache-a/console.log:
     
    10/04/2014 00:24:36 :: Unhandled Error
    Traceback (most recent call last):
      File "/opt/graphite/local/lib/python2.7/site-packages/twisted/python/log.py", line 84, in callWithLogger
        return callWithContext({"system": lp}, func, *args, **kw)
      File "/opt/graphite/local/lib/python2.7/site-packages/twisted/python/log.py", line 69, in callWithContext
        return context.call({ILogContext: newCtx}, func, *args, **kw)
      File "/opt/graphite/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/opt/graphite/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
    --- <exception caught here> ---
      File "/opt/graphite/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 581, in _doReadOrWrite
        why = selectable.doRead()
      File "/opt/graphite/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 199, in doRead
        rval = self.protocol.dataReceived(data)
      File "/opt/graphite/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 755, in dataReceived
        self.stringReceived(packet)
      File "/opt/graphite/lib/carbon/protocols.py", line 100, in stringReceived
        for (metric, datapoint) in datapoints:
    exceptions.ValueError: too many values to unpack
     
    ------
     
    graphite1:/opt/graphite/storage/log/webapp/exception.log:
     
    Thu Apr 10 00:26:27 2014 :: Exception Caught
    Traceback (most recent call last):
      File "/opt/graphite/webapp/graphite/render/datalib.py", line 226, in fetchData
        cachedResults = CarbonLink.query(dbFile.real_metric)
      File "/opt/graphite/webapp/graphite/render/datalib.py", line 135, in query
        results = self.send_request(request)
      File "/opt/graphite/webapp/graphite/render/datalib.py", line 161, in send_request
        result = self.recv_response(conn)
      File "/opt/graphite/webapp/graphite/render/datalib.py", line 173, in recv_response
        len_prefix = recv_exactly(conn, 4)
      File "/opt/graphite/webapp/graphite/render/datalib.py", line 188, in recv_exactly
        raise Exception("Connection lost")
    Exception: Connection lost


Posted on pastebin: http://pastebin.com/h7nEVjZQ

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.