yahoo-eng-team team mailing list archive

[Adam Gandelman <1326183@xxxxxxxxxxxxxxxxxx>]

** Also affects: nova/icehouse
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1326183

Title:
  detach interface fails as instance info cache is corrupted

Status in OpenStack Compute (Nova):
  Fix Committed
Status in OpenStack Compute (nova) icehouse series:
  New

Bug description:
  
  Performing attach/detach interface on a VM sometimes results in an interface that can't be detached from the VM.
  I could triage it to the corrupted instance cache info due to non-atomic update of that information.
  Details on how to reproduce the bug are as follows. Since this is due to a race condition, the test can take quite a bit of time before it hits the bug.

  Steps to reproduce:

  1) Devstack with trunk with the following local.conf:
  disable_service n-net
  enable_service q-svc
  enable_service q-agt
  enable_service q-dhcp
  enable_service q-l3
  enable_service q-meta
  enable_service q-metering
  RECLONE=yes
  # and other options as set in the trunk's local

  2) Create few networks:
  $> neutron net-create testnet1
  $> neutron net-create testnet2
  $> neutron net-create testnet3
  $> neutron subnet-create testnet1 192.168.1.0/24
  $> neutron subnet-create testnet2 192.168.2.0/24
  $> neutron subnet-create testnet3 192.168.3.0/24

  2) Create a testvm in testnet1:
  $> nova boot --flavor m1.tiny --image cirros-0.3.2-x86_64-uec --nic net-id=`neutron net-list | grep testnet1 | cut -f 2 -d ' '` testvm

  3) Run the following shell script to attach and detach interfaces for this vm in the remaining two networks in a loop until we run into the issue at hand:
  --------
  #! /bin/bash
  c=10000
  netid1=`neutron net-list | grep testnet2 | cut -f 2 -d ' '`
  netid2=`neutron net-list | grep testnet3 | cut -f 2 -d ' '`
  while [ $c -gt 0 ]
  do
     echo "Round: " $c
     echo -n "Attaching two interfaces... "
     nova interface-attach --net-id $netid1 testvm
     nova interface-attach --net-id $netid2 testvm
     echo "Done"
     echo "Sleeping until both those show up in interfaces"
     waittime=0
     while [ $waittime -lt 60 ]
     do
         count=`nova interface-list testvm | wc -l`
         if [ $count -eq 7 ]
         then
             break
         fi
         sleep 2
         (( waittime+=2 ))
     done
     echo "Waited for " $waittime " seconds"
     echo "Detaching both... "
     nova interface-list testvm | grep $netid1 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
     nova interface-list testvm | grep $netid2 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
     echo "Done; check interfaces are gone in a minute."
     waittime=0
     while [ $waittime -lt 60 ]
     do
         count=`nova interface-list testvm | wc -l`
         echo "line count: " $count
         if [ $count -eq 5 ]
         then
             break
         fi
         sleep 2
         (( waittime+=2 ))
     done
     if [ $waittime -ge 60 ]
     then
        echo "bad case"
        exit 1
     fi
     echo "Interfaces are gone"
     ((  c-- ))
  done
  ---------

  Eventually the test will stop with a failure ("bad case") and the
  interface remaining either from testnet2 or testnet3 can not be
  detached at all.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1326183/+subscriptions