← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1252846] Re: instance deletion fails due to neutron client error: "simultaneous read on fileno N detected"

 

*** This bug is a duplicate of bug 1229475 ***
    https://bugs.launchpad.net/bugs/1229475

Please make sure you have the proper neutronclient with this patch

https://github.com/openstack/python-
neutronclient/commit/e6a9f79066943b75e2c60b0165533d5886cc9cd8

** This bug is no longer a duplicate of bug 1251920
   Tempest failures due to failure to return console logs from an instance

** This bug has been marked a duplicate of bug 1229475
   terminate_instance(): RuntimeError: Second simultaneous read on fileno 16 detected

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1252846

Title:
  instance deletion fails due to neutron client error: "simultaneous
  read on fileno N detected"

Status in OpenStack Compute (Nova):
  New

Bug description:
  Doing performance testing with the latest OpenStack code, I regularly
  have a problem when deleting instances. When attempting to query
  neutron during instance deallocation, nova-compute apparently makes a
  greenthread-unsafe use of its neutron client:

  {u'message': 
  "Connection to neutron failed: Second simultaneous read on fileno 25 detected.  Unless you really know what you're doing, make sure that only one greenthread can read any particular socket.  Consider using a pools.Pool. If you do know what you're doing and",
  u'code': 500,
  u'details': u'
     File "/opt/stack/nova/nova/compute/manager.py", line 259, in decorated_function
       return function(self, context, *args, **kwargs)                                                                                                                                          
     File "/opt/stack/nova/nova/compute/manager.py", line 1984, in terminate_instance                                                                                                           
       do_terminate_instance(instance, bdms)                                                                                                                                                    
     File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 248, in inner                                                                                                              
       return f(*args, **kwargs)                                                                                                                                                                
     File "/opt/stack/nova/nova/compute/manager.py", line 1976, in do_terminate_instance                                                                                                        
       reservations=reservations)                                                                                                                                                               
     File "/opt/stack/nova/nova/hooks.py", line 105, in inner                                                                                                                                   
       rv = f(*args, **kwargs)                                                                                                                                                                  
     File "/opt/stack/nova/nova/compute/manager.py", line 1949, in _delete_instance                                                                                                             
       user_id=user_id)                                                                                                                                                                         
     File "/opt/stack/nova/nova/compute/manager.py", line 1919, in _delete_instance                                                                                                             
       self._shutdown_instance(context, db_inst, bdms)                                                                                                                                          
     File "/opt/stack/nova/nova/compute/manager.py", line 1854, in _shutdown_instance                                                                                                           
       self._try_deallocate_network(context, instance, requested_networks)                                                                                                                      
     File "/opt/stack/nova/nova/compute/manager.py", line 1814, in _try_deallocate_network                                                                                                      
       self._set_instance_error_state(context, instance[\'uuid\'])                                                                                                                              
     File "/opt/stack/nova/nova/compute/manager.py", line 1809, in _try_deallocate_network                                                                                                      
       self._deallocate_network(context, instance, requested_networks)                                                                                                                          
     File "/opt/stack/nova/nova/compute/manager.py", line 1492, in _deallocate_network                                                                                                          
       context, instance, requested_networks=requested_networks)                                                                                                                                
     File "/opt/stack/nova/nova/network/neutronv2/api.py", line 404, in deallocate_for_instance                                                                                                 
       data = neutronv2.get_client(context).list_ports(**search_opts)                                                                                                                           
     File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 111, in with_params                                                                                              
       ret = self.function(instance, *args, **kwargs)                                                                                                                                           
     File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 306, in list_ports                                                                                               
       **_params)                                                                                                                                                                               
     File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1253, in list                                                                                                    
       for r in self._pagination(collection, path, **params):                                                                                                                                   
     File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1266, in _pagination                                                                                             
       res = self.get(path, params=params)                                                                                                                                                      
     File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1239, in get                                                                                                     
       headers=headers, params=params)                                                                                                                                                          
     File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1221, in retry_request                                                                                           
       headers=headers, params=params)                                                                                                                                                          
     File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 1156, in do_request                                                                                              
       resp, replybody = self.httpclient.do_request(action, method, body=body)                                                                                                                  
     File "/opt/stack/python-neutronclient/neutronclient/client.py", line 190, in do_request                                                                                                    
       **kwargs)                                                                                                                                                                                
     File "/opt/stack/python-neutronclient/neutronclient/client.py", line 152, in _cs_request                                                                                                   
       raise exceptions.ConnectionFailed(reason=e)  
  ', u'created': u'2013-11-19T19:36:11Z'} 

  Note that I had to make a minor hack to my python-neutronclient to get
  this exception:

  diff --git a/neutronclient/v2_0/client.py b/neutronclient/v2_0/client.py
  index 733f339..d2ac237 100644
  --- a/neutronclient/v2_0/client.py
  +++ b/neutronclient/v2_0/client.py
  @@ -1224,6 +1224,8 @@ class Client(object):
                   if i < self.retries:
                       _logger.debug(_('Retrying connection to Neutron service'))
                       time.sleep(self.retry_interval)
  +                else:
  +                    raise
   
           raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))

  Without that patch, you just see the error "Maximum attempts reached,"
  which doesn't tell you anything.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1252846/+subscriptions