yahoo-eng-team team mailing list archive

[Praveen Yalagandula <ypraveen@xxxxxxxxx>]

Public bug reported:


Performing attach/detach interface on a VM sometimes results in an interface that can't be detached from the VM.
I could triage it to the corrupted instance cache info due to non-atomic update of that information.
Details on how to reproduce the bug are as follows. Since this is due to a race condition, the test can take quite a bit of time before it hits the bug.

Steps to reproduce:

1) Devstack with trunk with the following local.conf:
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service q-metering
RECLONE=yes
# and other options as set in the trunk's local

2) Create few networks:
$> neutron net-create testnet1
$> neutron net-create testnet2
$> neutron net-create testnet3
$> neutron subnet-create testnet1 192.168.1.0/24
$> neutron subnet-create testnet2 192.168.2.0/24
$> neutron subnet-create testnet3 192.168.3.0/24

2) Create a testvm in testnet1:
$> nova boot --flavor m1.tiny --image cirros-0.3.2-x86_64-uec --nic net-id=`neutron net-list | grep testnet1 | cut -f 2 -d ' '` testvm

3) Run the following shell script to attach and detach interfaces for this vm in the remaining two networks in a loop until we run into the issue at hand:
--------
#! /bin/bash
c=10000
netid1=`neutron net-list | grep testnet2 | cut -f 2 -d ' '`
netid2=`neutron net-list | grep testnet3 | cut -f 2 -d ' '`
while [ $c -gt 0 ]
do
   echo "Round: " $c
   echo -n "Attaching two interfaces... "
   nova interface-attach --net-id $netid1 testvm
   nova interface-attach --net-id $netid2 testvm
   echo "Done"
   echo "Sleeping until both those show up in interfaces"
   waittime=0
   while [ $waittime -lt 60 ]
   do
       count=`nova interface-list testvm | wc -l`
       if [ $count -eq 7 ]
       then
           break
       fi
       sleep 2
       (( waittime+=2 ))
   done
   echo "Waited for " $waittime " seconds"
   echo "Detaching both... "
   nova interface-list testvm | grep $netid1 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
   nova interface-list testvm | grep $netid2 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
   echo "Done; check interfaces are gone in a minute."
   waittime=0
   while [ $waittime -lt 60 ]
   do
       count=`nova interface-list testvm | wc -l`
       echo "line count: " $count
       if [ $count -eq 5 ]
       then
           break
       fi
       sleep 2
       (( waittime+=2 ))
   done
   if [ $waittime -ge 60 ]
   then
      echo "bad case"
      exit 1
   fi
   echo "Interfaces are gone"
   ((  c-- ))
done
---------

Eventually the test will stop with a failure ("bad case") and the
interface remaining either from testnet2 or testnet3 can not be detached
at all.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: api network neutron nova

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1326183

Title:
  detach interface fails as instance info cache is corrupted

Status in OpenStack Compute (Nova):
  New

Bug description:
  
  Performing attach/detach interface on a VM sometimes results in an interface that can't be detached from the VM.
  I could triage it to the corrupted instance cache info due to non-atomic update of that information.
  Details on how to reproduce the bug are as follows. Since this is due to a race condition, the test can take quite a bit of time before it hits the bug.

  Steps to reproduce:

  1) Devstack with trunk with the following local.conf:
  disable_service n-net
  enable_service q-svc
  enable_service q-agt
  enable_service q-dhcp
  enable_service q-l3
  enable_service q-meta
  enable_service q-metering
  RECLONE=yes
  # and other options as set in the trunk's local

  2) Create few networks:
  $> neutron net-create testnet1
  $> neutron net-create testnet2
  $> neutron net-create testnet3
  $> neutron subnet-create testnet1 192.168.1.0/24
  $> neutron subnet-create testnet2 192.168.2.0/24
  $> neutron subnet-create testnet3 192.168.3.0/24

  2) Create a testvm in testnet1:
  $> nova boot --flavor m1.tiny --image cirros-0.3.2-x86_64-uec --nic net-id=`neutron net-list | grep testnet1 | cut -f 2 -d ' '` testvm

  3) Run the following shell script to attach and detach interfaces for this vm in the remaining two networks in a loop until we run into the issue at hand:
  --------
  #! /bin/bash
  c=10000
  netid1=`neutron net-list | grep testnet2 | cut -f 2 -d ' '`
  netid2=`neutron net-list | grep testnet3 | cut -f 2 -d ' '`
  while [ $c -gt 0 ]
  do
     echo "Round: " $c
     echo -n "Attaching two interfaces... "
     nova interface-attach --net-id $netid1 testvm
     nova interface-attach --net-id $netid2 testvm
     echo "Done"
     echo "Sleeping until both those show up in interfaces"
     waittime=0
     while [ $waittime -lt 60 ]
     do
         count=`nova interface-list testvm | wc -l`
         if [ $count -eq 7 ]
         then
             break
         fi
         sleep 2
         (( waittime+=2 ))
     done
     echo "Waited for " $waittime " seconds"
     echo "Detaching both... "
     nova interface-list testvm | grep $netid1 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
     nova interface-list testvm | grep $netid2 | awk '{print "deleting ",$4; system("nova interface-detach testvm "$4 " ; sleep 2");}'
     echo "Done; check interfaces are gone in a minute."
     waittime=0
     while [ $waittime -lt 60 ]
     do
         count=`nova interface-list testvm | wc -l`
         echo "line count: " $count
         if [ $count -eq 5 ]
         then
             break
         fi
         sleep 2
         (( waittime+=2 ))
     done
     if [ $waittime -ge 60 ]
     then
        echo "bad case"
        exit 1
     fi
     echo "Interfaces are gone"
     ((  c-- ))
  done
  ---------

  Eventually the test will stop with a failure ("bad case") and the
  interface remaining either from testnet2 or testnet3 can not be
  detached at all.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1326183/+subscriptions