← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #56236

[Bug 1621818] [NEW] nova-compute unexpected input/output errors on starting instances

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Our setup consists of multiple controllers and multiple hypervisors. Our
shared storage for the instances is on a nfs 4.1 export.  Using Ubuntu
16.04 LTS and Openstack Mitaka

When we launch an instance, nova updates the mtime for the _base image to let the image cache manager know the image is actively used.  I think this was added here: https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=fb6ca3e7c8a38328d384cd41c061ded6623dac90
Because of this, in our setup, we are seeing unexpected input/output errors:

Stderr: u"/bin/touch: setting times of
'/var/lib/nova/instances/_base/79e34519bacb47ad6f64c4baca4d33fd5c57d34d':
Input/output error

A full trace can be found here: http://paste.openstack.org/show/570161/

This error particularly shows itself when launching multiple instances
at once.

Also, because of this error, the instances are rescheduled.  The assigned neutron ports, however, are not deleted.  This results in multiple ip's assigned to the instances, with only one of them UP.  This also results in attached floating ip's not working ..
This is similar to https://bugs.launchpad.net/nova/+bug/1609526, nova should tell neutron, either to delete the unused port, or update it instead of creating a new one.

Some more info on our environment:
----------------------------------
Using libvirt + kvm, neutron with openvswitch L3 HA

# dpkg -l | grep nova
ii  nova-common                        2:13.0.0-0ubuntu2                   all          OpenStack Compute - common files
ii  nova-compute                       2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node base
ii  nova-compute-kvm                   2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
ii  python-nova                        2:13.0.0-0ubuntu2                   all          OpenStack Compute Python libraries
ii  python-novaclient                  2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7

# dpkg -l |grep libvirt
ii  libvirt-bin                        1.3.1-1ubuntu10.1                   amd64        programs for the libvirt library
ii  libvirt0:amd64                     1.3.1-1ubuntu10.1                   amd64        library for interfacing with different virtualization systems
ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
ii  python-libvirt                     1.3.1-1ubuntu1                      amd64        libvirt Python bindings

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Our setup consists of multiple controllers and multiple hypervisors. Our
  shared storage for the instances is on a nfs 4.1 export.  Using Ubuntu
  16.04 LTS and Openstack Mitaka
  
  When we launch an instance, nova updates the mtime for the _base image to let the image cache manager know the image is actively used.  I think this was added here: https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=fb6ca3e7c8a38328d384cd41c061ded6623dac90
  Because of this, in our setup, we are seeing unexpected input/output errors:
  
  Stderr: u"/bin/touch: setting times of
  '/var/lib/nova/instances/_base/79e34519bacb47ad6f64c4baca4d33fd5c57d34d':
  Input/output error
  
  A full trace can be found here: http://paste.openstack.org/show/570161/
  
  This error particularly shows itself when launching multiple instances
  at once.
  
  Also, because of this error, the instances are rescheduled.  The assigned neutron ports, however, are not deleted.  This results in multiple ip's assigned to the instances, with only one of them UP.  This also results in attached floating ip's not working ..
  This is similar to https://bugs.launchpad.net/nova/+bug/1609526, nova should tell neutron, either to delete the unused port, or update it instead of creating a new one.
  
  Some more info on our environment:
  ----------------------------------
  Using libvirt + kvm, neutron with openvswitch L3 HA
  
  # dpkg -l | grep nova
  ii  nova-common                        2:13.0.0-0ubuntu2                   all          OpenStack Compute - common files
  ii  nova-compute                       2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                   2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
  ii  python-nova                        2:13.0.0-0ubuntu2                   all          OpenStack Compute Python libraries
  ii  python-novaclient                  2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7
+ 
+ # dpkg -l |grep libvirt
+ ii  libvirt-bin                        1.3.1-1ubuntu10.1                   amd64        programs for the libvirt library
+ ii  libvirt0:amd64                     1.3.1-1ubuntu10.1                   amd64        library for interfacing with different virtualization systems
+ ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
+ ii  python-libvirt                     1.3.1-1ubuntu1                      amd64        libvirt Python bindings

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1621818

Title:
  nova-compute unexpected input/output errors on starting instances

Status in OpenStack Compute (nova):
  New

Bug description:
  Our setup consists of multiple controllers and multiple hypervisors.
  Our shared storage for the instances is on a nfs 4.1 export.  Using
  Ubuntu 16.04 LTS and Openstack Mitaka

  When we launch an instance, nova updates the mtime for the _base image to let the image cache manager know the image is actively used.  I think this was added here: https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=fb6ca3e7c8a38328d384cd41c061ded6623dac90
  Because of this, in our setup, we are seeing unexpected input/output errors:

  Stderr: u"/bin/touch: setting times of
  '/var/lib/nova/instances/_base/79e34519bacb47ad6f64c4baca4d33fd5c57d34d':
  Input/output error

  A full trace can be found here:
  http://paste.openstack.org/show/570161/

  This error particularly shows itself when launching multiple instances
  at once.

  Also, because of this error, the instances are rescheduled.  The assigned neutron ports, however, are not deleted.  This results in multiple ip's assigned to the instances, with only one of them UP.  This also results in attached floating ip's not working ..
  This is similar to https://bugs.launchpad.net/nova/+bug/1609526, nova should tell neutron, either to delete the unused port, or update it instead of creating a new one.

  Some more info on our environment:
  ----------------------------------
  Using libvirt + kvm, neutron with openvswitch L3 HA

  # dpkg -l | grep nova
  ii  nova-common                        2:13.0.0-0ubuntu2                   all          OpenStack Compute - common files
  ii  nova-compute                       2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                   2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
  ii  python-nova                        2:13.0.0-0ubuntu2                   all          OpenStack Compute Python libraries
  ii  python-novaclient                  2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7

  # dpkg -l |grep libvirt
  ii  libvirt-bin                        1.3.1-1ubuntu10.1                   amd64        programs for the libvirt library
  ii  libvirt0:amd64                     1.3.1-1ubuntu10.1                   amd64        library for interfacing with different virtualization systems
  ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
  ii  python-libvirt                     1.3.1-1ubuntu1                      amd64        libvirt Python bindings

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1621818/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next