yahoo-eng-team team mailing list archive

[Pavlo Shchelokovskyy <1992169@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

This might somewhat be related to
https://bugs.launchpad.net/nova/+bug/1800755 and discussion there.

Recently the following problem was reported in one of our clouds:

- a homegrown self-written monitoring that polls servers diagnostics
- the monitoring script is naive and does not check the server state before requesting server diagnostics
- several servers in shutdown state
- instance_faults table is growing and ballooning database size on disk

During handling of GET /servers/<uuid>/diagnostics call for anything but RUNNING instance nova raises InstanceInvalidState exception which is then:
- stored in instance_faults table;
- returns as HTTP409 Conflict to the user.

https://opendev.org/openstack/nova/src/commit/03d2715ed492350fa11908aea0fdd0265993e284/nova/compute/manager.py#L6550-L6558

Effectively benign 'read-only' GET requests are recorded in the DB.
Also, these instance_faults entries can not purged by standard means
since the instance is not deleted yet. What's more, they won't be shown
in any API at all, since the server is also not in ERROR state.

This got me thinking - should the InvalidInstanceState be saved as instance_faults at all?
After all, usually this exception indicates not the problem (fault) with the instance, but the mismatch between instance state and requested action upon instance, which might not warrant storing it.

There's also a slight DoS potential here, but since default policy for
get diagnostics call is admin-only, this is probably not worth worrying.

** Affects: nova
     Importance: Undecided
     Assignee: Pavlo Shchelokovskyy (pshchelo)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Pavlo Shchelokovskyy (pshchelo)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1992169

Title:
  instance_faults entries are created on InstanceInvalidState exceptions

Status in OpenStack Compute (nova):
  New

Bug description:
  This might somewhat be related to
  https://bugs.launchpad.net/nova/+bug/1800755 and discussion there.

  Recently the following problem was reported in one of our clouds:

  - a homegrown self-written monitoring that polls servers diagnostics
  - the monitoring script is naive and does not check the server state before requesting server diagnostics
  - several servers in shutdown state
  - instance_faults table is growing and ballooning database size on disk

  During handling of GET /servers/<uuid>/diagnostics call for anything but RUNNING instance nova raises InstanceInvalidState exception which is then:
  - stored in instance_faults table;
  - returns as HTTP409 Conflict to the user.

  https://opendev.org/openstack/nova/src/commit/03d2715ed492350fa11908aea0fdd0265993e284/nova/compute/manager.py#L6550-L6558

  Effectively benign 'read-only' GET requests are recorded in the DB.
  Also, these instance_faults entries can not purged by standard means
  since the instance is not deleted yet. What's more, they won't be
  shown in any API at all, since the server is also not in ERROR state.

  This got me thinking - should the InvalidInstanceState be saved as instance_faults at all?
  After all, usually this exception indicates not the problem (fault) with the instance, but the mismatch between instance state and requested action upon instance, which might not warrant storing it.

  There's also a slight DoS potential here, but since default policy for
  get diagnostics call is admin-only, this is probably not worth
  worrying.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1992169/+subscriptions