yahoo-eng-team team mailing list archive

[Morgan Fainberg <morgan.fainberg@xxxxxxxxx>]

Targeted to Icehouse based upon original request from -infra.

** Also affects: keystone/icehouse
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1334466

Title:
  Eventlet Log Spamming on Client Disconnect (Broken Pipe)

Status in OpenStack Identity (Keystone):
  In Progress
Status in Keystone icehouse series:
  In Progress

Bug description:
  If a client makes a request to keystone, and then disconnects before
  keystone responds, it is possible to fill up the logs (INFO) with
  eventlet tracebacks due to broken pipe:

  2014-06-24 23:30:29.729 31440 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Jun/2014 23:30:29] "POST /v3/auth/tokens HTTP/1.1" 201 0 100.313719
  2014-06-24 23:30:29.731 31440 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 399, in handle_one_response
      write(''.join(towrite))
    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 349, in write
      _writelines(towrite)
    File "/usr/lib/python2.7/socket.py", line 334, in writelines
      self.flush()
    File "/usr/lib/python2.7/socket.py", line 303, in flush
      self._sock.sendall(view[write_offset:write_offset+buffer_size])
    File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
      tail = self.send(data, flags)
    File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
      total_sent += fd.send(data[total_sent:], flags)
  error: [Errno 32] Broken pipe

  Example (900k line file) due to this [WARNING THIS LINK MIGHT KILL YOUR BROWSER]:
  http://logs.openstack.org/66/99766/2/check/check-grenade-dsvm/9fd33e1/logs/old/screen-key.txt.gz?level=INFO

  We should override the required HTTPProtocol class and gracefully
  handle the traceback. If we would like to keep the information, a
  single log-line per incident would be sufficient instead of ~14.

  This should be considered for a backport to Icehouse to help limit log
  spam there as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1334466/+subscriptions