yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1558398] Re: instance snapshot with large root disk fails when glance api using ssl

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Stuart McLaren <1558398@xxxxxxxxxxxxxxxxxx>
Date: Fri, 18 Mar 2016 10:41:56 -0000
Reply-to: Bug 1558398 <1558398@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: glance
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1558398

Title:
  instance snapshot with large root disk fails when glance api using ssl

Status in Glance:
  Invalid

Bug description:
  in Kilo, using glance ssl with or without a load-balancer in between
  will cause instance snapshot of an instance with a large root disk
  (>~60 GB) to fail.  Smaller root disks work fine.  The flavor test was
  specifically 4 vcpu, 16GB ram, 160GB root disk, 0 ephemeral and 0
  swap.   This is when using local file storage for glance and you see
  pretty much 64GB worth of data hit the glance server disk before the
  file disappears off the server and the connection has issues.

  Glance will only say that the client closed the connection before all the data could be sent.
  WARNING glance.api.v1.upload_utils [req-a5279f16-1be8-4cbf-8625-491e25bcf5c7 766dcc03c6e0454296b449105c71db8c 1a63bbf53e8949e7928ac9e2381a0c04 - - -] Client disconnected before sending all data to backend

  This occurs when trying through horizon, openstack client, or even a
  curl to the glance api.  If you use glance purely with http this issue
  does not occur.

  When you use a load-balancer in between for ssl, (nginx in particular in my case) it will show the below error when the connection dies.  Note this setup had glance endpoints ssl and nova doing https for glance to loadbalancer, but backend glance was http:
  2016/03/15 16:28:23 [info] 23#23: *4346 SSL_read() failed (SSL: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac) while sending request to upstream, client: 10.240.118.7, server: vip-cream-1.lss.emc.com, request: "PUT /v1/images/5d4d9e36-1d66-45da-8a4e-0c1aede75cd7 HTTP/1.1", upstream: "http://10.240.118.5:9292/v1/images/5d4d9e36-1d66-45da-8a4e-0c1aede75cd7";, host: "10.240.118.24:9292" 

  and nova will show this error:
  Traceback (most recent call last):
     File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
       executor_callback))
     File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
       executor_callback)
     File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
       result = func(ctxt, **new_args)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6933, in snapshot_instance
       return self.manager.snapshot_instance(ctxt, image_id, instance)
     File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
       payload)
     File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
       six.reraise(self.type_, self.value, self.tb)
     File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
       return f(self, context, *args, **kw)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 333, in decorated_function
       LOG.warning(msg, e, instance_uuid=instance_uuid)
     File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
       six.reraise(self.type_, self.value, self.tb)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 304, in decorated_function
       return function(self, context, *args, **kwargs)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
       kwargs['instance'], e, sys.exc_info())
     File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
       six.reraise(self.type_, self.value, self.tb)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349, in decorated_function
       return function(self, context, *args, **kwargs)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function
       instance=instance)
     File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
       six.reraise(self.type_, self.value, self.tb)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
       *args, **kwargs)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3291, in snapshot_instance
       task_states.IMAGE_SNAPSHOT)
     File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3321, in _snapshot_instance
       update_task_state)
     File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1447, in snapshot
       image_file)
     File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in update
       purge_props=purge_props)
     File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 397, in update
       _reraise_translated_image_exception(image_id)
     File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 395, in update
       image_id, **image_meta)
     File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 218, in call
       return getattr(client.images, method)(*args, **kwargs)
     File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 360, in update
       resp, body = self.client.put(url, headers=hdrs, data=image_data)
     File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 268, in put
       return self._request('PUT', url, **kwargs)
     File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 209, in _request
       **kwargs)
     File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
       resp = self.send(prep, **send_kwargs)
     File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
       r = adapter.send(request, **kwargs)
     File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 388, in send
       low_conn.send(hex(len(i))[2:].encode('utf-8'))
     File "/usr/lib64/python2.7/httplib.py", line 820, in send
       self.sock.sendall(data)
     File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 101, in sendall
       tail = self.send(data)
     File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 81, in write
       return self.fd.write(data)
     File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 950, in send
       self._raise_ssl_error(self._ssl, result)
     File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 862, in _raise_ssl_error
       raise SysCallError(errno, errorcode[errno])
   SysCallError: (104, 'ECONNRESET')

  
  When you don't use a loadbalancer and have nova point to glance directly and have glance utilize ssl in the config, you see this type of error from nova:
  2016-03-16 17:39:27.097 523 ERROR oslo_messaging.rpc.dispatcher [req-84622019-ef9e-45a6-97a7-a3a7c8c45661 f9e76346bd0344a68a98887ce27d5a46 0070df81861b47a9a805643c8864c607 - - -] Exception during message handling: (32, 'EPIPE')
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6933, in snapshot_instance
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     return self.manager.snapshot_instance(ctxt, image_id, instance)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 333, in decorated_function
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 304, in decorated_function
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349, in decorated_function
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     instance=instance)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     *args, **kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3291, in snapshot_instance
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     task_states.IMAGE_SNAPSHOT)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3321, in _snapshot_instance
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     update_task_state)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1447, in snapshot
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     image_file)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in update
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     purge_props=purge_props)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 397, in update
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     _reraise_translated_image_exception(image_id)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 395, in update
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     image_id, **image_meta)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 218, in call
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     return getattr(client.images, method)(*args, **kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 360, in update
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     resp, body = self.client.put(url, headers=hdrs, data=image_data)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 268, in put
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     return self._request('PUT', url, **kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 209, in _request
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     **kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     resp = self.send(prep, **send_kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     r = adapter.send(request, **kwargs)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 390, in send
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     low_conn.send(i)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/httplib.py", line 820, in send
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     self.sock.sendall(data)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 101, in sendall
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     tail = self.send(data)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 81, in write
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     return self.fd.write(data)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 950, in send
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     self._raise_ssl_error(self._ssl, result)
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 862, in _raise_ssl_error
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher     raise SysCallError(errno, errorcode[errno])
  2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher SysCallError: (32, 'EPIPE')

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1558398/+subscriptions
References

[Bug 1558398] [NEW] instance snapshot with large root disk fails when glance api using ssl
From: Jeremy Pugh, 2016-03-17