yahoo-eng-team team mailing list archive

[Wallace Cardoso <1801733@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
==============
The 'conductor-api' for 'rebuild_instance' has a vulnerability point for the parameter 'rebuild_instance/args/instance/nova_object.data/flavor/nova_object.data/vcpus'. When set to an invalid number of vcpus in the flavor of the instance, the compute component takes 100% of cpu consuming forever without changing the state from rebuild to active (or error). In addition, new requests to compute component are not computed, that is, the node gets out-of-service until its restart. Maybe, this bug can be a way of using a denial-of-service attack.

Steps to reproduce
=====================
1) create an instance with the flavor (VCPUS: 1, MEM: 64MB, STORAGE: 0GB) and the cirros image 0.3.4;
2) rebuild the instance with an alternative cirros image 0.4.0;
2.1) intercept the message to 'conductor' api (ComputeTaskAPI) for the method 'rebuild_instance', and change the parameter 'rebuild_instance/args/instance/nova_object.data/flavor/nova_object.data/vcpus' to 10000000000000000000001;
3) rebuild again the instance with the original image of the instance (cirros 0.3.4);
4) shelve the instance;
5) delete the instance;

Expected result
================
Even that rebuild is not an action that takes the flavor into account, should exist something for ensuring correctness of other parameters. The compute node does not stop working because of an invalid parameter.

Actual result
================
The instance does not change from rebuild to active, remaining rebuilding forever, and the compute node gets innoperating until the services be restarted. 'nova-compute' consuming 100% of cpu.

Environment
==============
I used devstack/stable/queens, a fresh Ubuntu environment.

Logs & Configs
=================
Logs attached.
The fault is injected after 11:24:16.
If you search for '10000000000000000000001', you will see the line below:
Nov  5 11:24:21 localhost nova-compute[14517]: #033[00;32mDEBUG nova.virt.hardware [#033[01;36mNone req-f97def42-9630-4165-81e5-abc0cab5c02f #033[00;36madmin admin#033[00;32m] #033[01;35m#033[00;32mBuild topologies for 10000000000000000000001 vcpu(s) 65536:65536:65536#033[00m #033[00;33m{{(pid=14517) _get_possible_cpu_topologies /opt/stack/queens/dest/nova/nova/virt/hardware.py:418}}#033[00m

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: fault-injection

** Attachment added: "Syslog"
   https://bugs.launchpad.net/bugs/1801733/+attachment/5209338/+files/newbug.sys.logs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1801733

Title:
  nova-compute consuming 100% of cpu after rebuilding with invalid data
  parameters

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ==============
  The 'conductor-api' for 'rebuild_instance' has a vulnerability point for the parameter 'rebuild_instance/args/instance/nova_object.data/flavor/nova_object.data/vcpus'. When set to an invalid number of vcpus in the flavor of the instance, the compute component takes 100% of cpu consuming forever without changing the state from rebuild to active (or error). In addition, new requests to compute component are not computed, that is, the node gets out-of-service until its restart. Maybe, this bug can be a way of using a denial-of-service attack.

  Steps to reproduce
  =====================
  1) create an instance with the flavor (VCPUS: 1, MEM: 64MB, STORAGE: 0GB) and the cirros image 0.3.4;
  2) rebuild the instance with an alternative cirros image 0.4.0;
  2.1) intercept the message to 'conductor' api (ComputeTaskAPI) for the method 'rebuild_instance', and change the parameter 'rebuild_instance/args/instance/nova_object.data/flavor/nova_object.data/vcpus' to 10000000000000000000001;
  3) rebuild again the instance with the original image of the instance (cirros 0.3.4);
  4) shelve the instance;
  5) delete the instance;

  Expected result
  ================
  Even that rebuild is not an action that takes the flavor into account, should exist something for ensuring correctness of other parameters. The compute node does not stop working because of an invalid parameter.

  Actual result
  ================
  The instance does not change from rebuild to active, remaining rebuilding forever, and the compute node gets innoperating until the services be restarted. 'nova-compute' consuming 100% of cpu.

  Environment
  ==============
  I used devstack/stable/queens, a fresh Ubuntu environment.

  Logs & Configs
  =================
  Logs attached.
  The fault is injected after 11:24:16.
  If you search for '10000000000000000000001', you will see the line below:
  Nov  5 11:24:21 localhost nova-compute[14517]: #033[00;32mDEBUG nova.virt.hardware [#033[01;36mNone req-f97def42-9630-4165-81e5-abc0cab5c02f #033[00;36madmin admin#033[00;32m] #033[01;35m#033[00;32mBuild topologies for 10000000000000000000001 vcpu(s) 65536:65536:65536#033[00m #033[00;33m{{(pid=14517) _get_possible_cpu_topologies /opt/stack/queens/dest/nova/nova/virt/hardware.py:418}}#033[00m

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1801733/+subscriptions