yahoo-eng-team team mailing list archive

[Morgan Fainberg <morgan.fainberg@xxxxxxxxx>]

Public bug reported:

If a client makes a request to keystone, and then disconnects before
keystone responds, it is possible to fill up the logs (INFO) with
eventlet tracebacks due to broken pipe:

2014-06-24 23:30:29.729 31440 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Jun/2014 23:30:29] "POST /v3/auth/tokens HTTP/1.1" 201 0 100.313719
2014-06-24 23:30:29.731 31440 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 399, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 349, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 32] Broken pipe

Example (900k line file) due to this [WARNING THIS LINK MIGHT KILL YOUR BROWSER]:
http://logs.openstack.org/66/99766/2/check/check-grenade-dsvm/9fd33e1/logs/old/screen-key.txt.gz?level=INFO

We should override the required HTTPProtocol class and gracefully handle
the traceback. If we would like to keep the information, a single log-
line per incident would be sufficient instead of ~14.

This should be considered for a backport to Icehouse to help limit log
spam there as well.

** Affects: keystone
     Importance: Critical
         Status: Triaged


** Tags: icehouse-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1334466

Title:
  Eventlet Log Spamming on Client Disconnect (Broken Pipe)

Status in OpenStack Identity (Keystone):
  Triaged

Bug description:
  If a client makes a request to keystone, and then disconnects before
  keystone responds, it is possible to fill up the logs (INFO) with
  eventlet tracebacks due to broken pipe:

  2014-06-24 23:30:29.729 31440 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Jun/2014 23:30:29] "POST /v3/auth/tokens HTTP/1.1" 201 0 100.313719
  2014-06-24 23:30:29.731 31440 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 399, in handle_one_response
      write(''.join(towrite))
    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 349, in write
      _writelines(towrite)
    File "/usr/lib/python2.7/socket.py", line 334, in writelines
      self.flush()
    File "/usr/lib/python2.7/socket.py", line 303, in flush
      self._sock.sendall(view[write_offset:write_offset+buffer_size])
    File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
      tail = self.send(data, flags)
    File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
      total_sent += fd.send(data[total_sent:], flags)
  error: [Errno 32] Broken pipe

  Example (900k line file) due to this [WARNING THIS LINK MIGHT KILL YOUR BROWSER]:
  http://logs.openstack.org/66/99766/2/check/check-grenade-dsvm/9fd33e1/logs/old/screen-key.txt.gz?level=INFO

  We should override the required HTTPProtocol class and gracefully
  handle the traceback. If we would like to keep the information, a
  single log-line per incident would be sufficient instead of ~14.

  This should be considered for a backport to Icehouse to help limit log
  spam there as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1334466/+subscriptions