yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

** Also affects: nova/mitaka
   Importance: Undecided
       Status: New

** Tags added: compute image-cache nfs

** Changed in: nova
       Status: New => Confirmed

** Summary changed:

- nova-compute unexpected input/output errors on starting instances
+ nova-compute unexpected input/output errors on starting instances (NFS + image-cache)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1621818

Title:
  nova-compute unexpected input/output errors on starting instances (NFS
  + image-cache)

Status in OpenStack Compute (nova):
  Confirmed
Status in OpenStack Compute (nova) mitaka series:
  New

Bug description:
  Our setup consists of multiple controllers and multiple hypervisors.
  Our shared storage for the instances is on a nfs 4.1 export.  Using
  Ubuntu 16.04 LTS and Openstack Mitaka

  When we launch an instance, nova updates the mtime for the _base image to let the image cache manager know the image is actively used.  I think this was added here: https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=fb6ca3e7c8a38328d384cd41c061ded6623dac90
  Because of this, in our setup, we are seeing unexpected input/output errors:

  Stderr: u"/bin/touch: setting times of
  '/var/lib/nova/instances/_base/79e34519bacb47ad6f64c4baca4d33fd5c57d34d':
  Input/output error

  A full trace can be found here:
  http://paste.openstack.org/show/570161/

  This error particularly shows itself when launching multiple instances
  at once.

  Also, because of this error, the instances are rescheduled.  The assigned neutron ports, however, are not deleted.  This results in multiple ip's assigned to the instances, with only one of them UP.  This also results in attached floating ip's not working ..
  This is similar to https://bugs.launchpad.net/nova/+bug/1609526, nova should tell neutron, either to delete the unused port, or update it instead of creating a new one.

  Some more info on our environment:
  ----------------------------------
  Using libvirt + kvm, neutron with openvswitch L3 HA

  # dpkg -l | grep nova
  ii  nova-common                        2:13.0.0-0ubuntu2                   all          OpenStack Compute - common files
  ii  nova-compute                       2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                   2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
  ii  python-nova                        2:13.0.0-0ubuntu2                   all          OpenStack Compute Python libraries
  ii  python-novaclient                  2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7

  # dpkg -l |grep libvirt
  ii  libvirt-bin                        1.3.1-1ubuntu10.1                   amd64        programs for the libvirt library
  ii  libvirt0:amd64                     1.3.1-1ubuntu10.1                   amd64        library for interfacing with different virtualization systems
  ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
  ii  python-libvirt                     1.3.1-1ubuntu1                      amd64        libvirt Python bindings

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1621818/+subscriptions