yahoo-eng-team team mailing list archive

[Mattia Belluco <mattia.belluco@xxxxxx>]

Public bug reported:

Description
===========
After we upgraded our openstack deployment to Mitaka we started noticing more and more VMs in an error and/or shutoff state. We found out that although nova service-list was reporting all nova-compute as "up" nova-compute was logging :

ERROR nova.compute.manager OSError: [Errno 24] Too many open files

our deployment uses ceph as storage backend for ephemeral/cinder/glance
with currently around 900 osd installed.

Steps to reproduce
==================

Ask a compute node to perform several actions at once such as:
-take a live snapshot
-delete a VM
-boot a VM

Depending on the task the compute will allow for a certain number of
jobs before complaining about the number of open files.

Expected behaviour
=================
Nova-compute should be more robust in handling requests and should report when in error state.

Environment
===========
We are currently running:

nova-common                          2:13.1.2-0ubuntu2~cloud0  
nova-compute                         2:13.1.2-0ubuntu2~cloud0

hypervisor: Libvirt + KVM 
nova-compute-kvm                     2:13.1.2-0ubuntu2~cloud0  
nova-compute-libvirt                 2:13.1.2-0ubuntu2~cloud0

storage: 
ceph                                     0.94.9-1trusty

networking: neutron + openvswitch
neutron-common                           2:8.3.0-0ubuntu1.1~cloud0

Logs & Configs
==============

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "nova-compute.log"
   https://bugs.launchpad.net/bugs/1651526/+attachment/4794382/+files/nova-compute.log

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1651526

Title:
  nova-compute reports OSError: [Errno 24] Too many open files

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  After we upgraded our openstack deployment to Mitaka we started noticing more and more VMs in an error and/or shutoff state. We found out that although nova service-list was reporting all nova-compute as "up" nova-compute was logging :

  ERROR nova.compute.manager OSError: [Errno 24] Too many open files

  our deployment uses ceph as storage backend for
  ephemeral/cinder/glance with currently around 900 osd installed.

  Steps to reproduce
  ==================

  Ask a compute node to perform several actions at once such as:
  -take a live snapshot
  -delete a VM
  -boot a VM

  Depending on the task the compute will allow for a certain number of
  jobs before complaining about the number of open files.

  Expected behaviour
  =================
  Nova-compute should be more robust in handling requests and should report when in error state.

  Environment
  ===========
  We are currently running:

  nova-common                          2:13.1.2-0ubuntu2~cloud0  
  nova-compute                         2:13.1.2-0ubuntu2~cloud0

  hypervisor: Libvirt + KVM 
  nova-compute-kvm                     2:13.1.2-0ubuntu2~cloud0  
  nova-compute-libvirt                 2:13.1.2-0ubuntu2~cloud0

  storage: 
  ceph                                     0.94.9-1trusty

  networking: neutron + openvswitch
  neutron-common                           2:8.3.0-0ubuntu1.1~cloud0

  Logs & Configs
  ==============

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1651526/+subscriptions