← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #24477

[Bug 1358583] Re: [OSSA 2014-038] List instances by IP results in DoS of nova-network (CVE-2014-3708)

  Thread Previous • Date Previous • • Thread Next 
** Changed in: ossa
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1358583

Title:
  [OSSA 2014-038] List instances by IP results in DoS of nova-network
  (CVE-2014-3708)

Status in OpenStack Compute (Nova):
  Fix Committed
Status in OpenStack Compute (nova) icehouse series:
  Fix Committed
Status in OpenStack Compute (nova) juno series:
  Fix Committed
Status in OpenStack Security Advisories:
  Fix Released

Bug description:
  Hi,

  On a customer install which has approximately 500 VMs in the system,
  running the following will hang:

  nova list --ip 199

  What will happen afterwards is that the nova-network process will stop
  responding for a while, a trace shows that it's receiving a huge
  amount of data.  Upon further investigation, it looks like the issue
  maybe the right here:

  https://github.com/openstack/nova/blob/stable/icehouse/nova/network/manager.py#L420

  On this installation:

  nova=> select count(*) from virtual_interfaces;
   count 
  -------
   11985
  (1 row)

  So with 1 run, we're sending almost 12K records to a single nova-
  network process which takes up a huge CPU load (and blocks it from
  doing anything else).

  What ends up happening is other things start timing out in the system,
  such as resizes and new deployments:

  2014-08-19 03:44:49.511 31562 ERROR nova.compute.manager [req-e7b6d34f-81b5-46f9-a5e9-25ccfb863cfe bac292822cdf451f81201b3c1957914f 78578deaaf3542c087101746d1ad3f50] [instance: 28bf47af-1063-473c-9c7c-bb6351e97064] Setting instance vm_state to ERROR
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064] Traceback (most recent call last):
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3547, in finish_resize
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     disk_info, image)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3490, in _finish_resize
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     migration['dest_compute'])
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 95, in wrapped
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     return func(self, context, *args, **kwargs)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 509, in setup_networks_on_host
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     self.network_rpcapi.setup_networks_on_host(context, **args)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 270, in setup_networks_on_host
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     teardown=teardown)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 361, in call
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     return self.prepare().call(ctxt, method, **kwargs)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     wait_for_reply=True, timeout=timeout)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     timeout=timeout)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     return self._send(target, ctxt, message, wait_for_reply, timeout)
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064]     raise result
  2014-08-19 03:44:49.511 31562 TRACE nova.compute.manager [instance: 28bf47af-1063-473c-9c7c-bb6351e97064] RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID dd6f75163f414ac4ade3cd629593cd2d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1358583/+subscriptions
  Thread Previous • Date Previous • • Thread Next