yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1479662] [NEW] Vmware: Fail to create image cache "Bad file descriptor"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Tibi <kodokuu@xxxxxxxxx>
Date: Thu, 30 Jul 2015 07:55:59 -0000
Reply-to: Bug 1479662 <1479662@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have issue when nova try to create cache for new image. My image is
store in my control node.

Create image good.
Launch image with cache good.
Launch image without cache not good (fail create cache).

For reproduce  :

Glance (http) <=====>  apache proxy (https) <======> nova compute

nova compute log :

2015-07-29 09:12:50.248 910 DEBUG oslo_vmware.rw_handles [-] Creating HTTP connection to write to file with size = 15032385536 and URL = https://vcenter01:443/folder/vmware_temp/d18b930c-13ce-4ce7-8285-f48b9c9dc40c/9b6845ac-56d0-4699-8b90-847acd7cfadf/9b6845ac-56d0-4699-8b90-847acd7cfadf-flat.vmdk?dsName=DS-03&dcPath=datacenter-opk-01. _create_write_connection /usr/lib/python2.7/site-packages/oslo_vmware/rw_handles.py:91
2015-07-29 09:12:50.277 910 ERROR nova.virt.vmwareapi.io_util [-] Read/Write data failed
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util Traceback (most recent call last):
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/io_util.py", line 175, in _inner
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self.input.read(None)
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/oslo_vmware/rw_handles.py", line 613, in read
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = next(self._iter)
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/oslo_vmware/rw_handles.py", line 621, in get_next
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for data in self._glance_read_iter:
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 394, in integrity_iter
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for chunk in iter:
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 280, in _close_after_stream
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for chunk in response.iter_content(chunk_size=chunk_size):
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/requests/models.py", line 673, in generate
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for chunk in self.raw.stream(chunk_size, decode_content=True):
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 307, in stream
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self.read(amt=amt, decode_content=decode_content)
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 243, in read
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self._fp.read(amt)
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/httplib.py", line 567, in read
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     s = self.fp.read(amt)
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/socket.py", line 380, in read
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self._sock.recv(left)
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 57, in read
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     trampoline(self.fd.fileno(),
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/socket.py", line 224, in meth
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     return getattr(self._sock,name)(*args)
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/socket.py", line 170, in _dummy
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     raise error(EBADF, 'Bad file descriptor')
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util error: [Errno 9] Bad file descriptor
2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util
2015-07-29 09:12:50.282 910 ERROR nova.virt.vmwareapi.images [-] Transfer data failed
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images Traceback (most recent call last):
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images   File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/images.py", line 192, in start_transfer
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images     read_event.wait()
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images     return hubs.get_hub().switch()
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images     return self.greenlet.switch()
2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images error: [Errno 9] Bad file descriptor

Glance log :

2015-07-29 09:12:50.267 1310 INFO eventlet.wsgi.server [req-a350fc34-2ae7-4e77-8d8c-78ceb8513b00 2d3e8f76f3f9429c869e5a085837d2f1 d6f612ae52a9411f86ab5e2a79b4fd7e - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 477, in handle_one_response
    write(b''.join(towrite))
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write
    _writelines(towrite)
  File "/usr/lib64/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib64/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer


I resolve this issue with :

Glance (http)  <======> nova compute
and 
Glance (https) <======> nova compute.

So nova no support if glance use a proxy for expose https.

** Affects: glance
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: vmware

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1479662

Title:
  Vmware: Fail to create image cache "Bad file descriptor"

Status in Glance:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  I have issue when nova try to create cache for new image. My image is
  store in my control node.

  Create image good.
  Launch image with cache good.
  Launch image without cache not good (fail create cache).

  For reproduce  :

  Glance (http) <=====>  apache proxy (https) <======> nova compute

  nova compute log :

  2015-07-29 09:12:50.248 910 DEBUG oslo_vmware.rw_handles [-] Creating HTTP connection to write to file with size = 15032385536 and URL = https://vcenter01:443/folder/vmware_temp/d18b930c-13ce-4ce7-8285-f48b9c9dc40c/9b6845ac-56d0-4699-8b90-847acd7cfadf/9b6845ac-56d0-4699-8b90-847acd7cfadf-flat.vmdk?dsName=DS-03&dcPath=datacenter-opk-01. _create_write_connection /usr/lib/python2.7/site-packages/oslo_vmware/rw_handles.py:91
  2015-07-29 09:12:50.277 910 ERROR nova.virt.vmwareapi.io_util [-] Read/Write data failed
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util Traceback (most recent call last):
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/io_util.py", line 175, in _inner
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self.input.read(None)
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/oslo_vmware/rw_handles.py", line 613, in read
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = next(self._iter)
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/oslo_vmware/rw_handles.py", line 621, in get_next
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for data in self._glance_read_iter:
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 394, in integrity_iter
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for chunk in iter:
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 280, in _close_after_stream
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for chunk in response.iter_content(chunk_size=chunk_size):
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/requests/models.py", line 673, in generate
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     for chunk in self.raw.stream(chunk_size, decode_content=True):
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 307, in stream
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self.read(amt=amt, decode_content=decode_content)
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 243, in read
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self._fp.read(amt)
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/httplib.py", line 567, in read
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     s = self.fp.read(amt)
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/socket.py", line 380, in read
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     data = self._sock.recv(left)
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 57, in read
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     trampoline(self.fd.fileno(),
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/socket.py", line 224, in meth
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     return getattr(self._sock,name)(*args)
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util   File "/usr/lib64/python2.7/socket.py", line 170, in _dummy
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util     raise error(EBADF, 'Bad file descriptor')
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util error: [Errno 9] Bad file descriptor
  2015-07-29 09:12:50.277 910 TRACE nova.virt.vmwareapi.io_util
  2015-07-29 09:12:50.282 910 ERROR nova.virt.vmwareapi.images [-] Transfer data failed
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images Traceback (most recent call last):
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images   File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/images.py", line 192, in start_transfer
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images     read_event.wait()
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images     return hubs.get_hub().switch()
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images     return self.greenlet.switch()
  2015-07-29 09:12:50.282 910 TRACE nova.virt.vmwareapi.images error: [Errno 9] Bad file descriptor

  Glance log :

  2015-07-29 09:12:50.267 1310 INFO eventlet.wsgi.server [req-a350fc34-2ae7-4e77-8d8c-78ceb8513b00 2d3e8f76f3f9429c869e5a085837d2f1 d6f612ae52a9411f86ab5e2a79b4fd7e - - -] Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 477, in handle_one_response
      write(b''.join(towrite))
    File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write
      _writelines(towrite)
    File "/usr/lib64/python2.7/socket.py", line 334, in writelines
      self.flush()
    File "/usr/lib64/python2.7/socket.py", line 303, in flush
      self._sock.sendall(view[write_offset:write_offset+buffer_size])
    File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall
      tail = self.send(data, flags)
    File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
      total_sent += fd.send(data[total_sent:], flags)
  error: [Errno 104] Connection reset by peer

  
  I resolve this issue with :

  Glance (http)  <======> nova compute
  and 
  Glance (https) <======> nova compute.

  So nova no support if glance use a proxy for expose https.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1479662/+subscriptions