yahoo-eng-team team mailing list archive

["Markus Zoeller \(markus_z\)" <mzoeller@xxxxxxxxxxxxxxxxxx>]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY


** Changed in: nova
   Importance: Medium => Undecided

** Changed in: nova
       Status: Confirmed => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1331537

Title:
  nova service-list shows nova-compute as down and is required to be
  restarted frequently in order to provision new vms

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Nova compute services in Openstack Havana go down frequently as listed
  by "nova service-list" and requires to be restarted very frequently,
  multiple times every day. All the compute nodes have the ntp times in
  sync.

  When a node shows down, it is not able to use those compute nodes for
  launching new VMs and we quickly run out of compute resources. Hence
  our workaround is to restart the Compute nodes on those servers
  hourly.

  In the nova-compute node I've found the following error and they did match with the "Updated_at" field from nova service-list.
  2014-06-07 00:21:15.690 511340 ERROR nova.servicegroup.drivers.db [-] model server went away
  2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
  2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", l ine 92, in _report_state
  5804 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db report_count = service.service_ref['report_count'] + 1
  5805 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db TypeError: 'NoneType' object has no attribute '__getitem__'
  5806 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db

  It looks like the ones that are shown as down haven't been able to update the database with the latest status and they did match with the Traceback seen above (2014-06-07 00:21:15.690) on at least two compute nodes that I have seen.
  +------------------+------------------------+--------------+----------+-------+----------------------------+-----------------+
  | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
  +------------------+------------------------+--------------+----------+-------+----------------------------+-----------------+
  | nova-compute | nova1| blabla | enabled | up | 2014-06-07T00:37:42.000000 | None |
  | nova-compute | nova2 | blabla | enabled | down | 2014-06-07T00:21:05.000000 | None |

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1331537/+subscriptions