yahoo-eng-team team mailing list archive

[Alan Pevec <1444497@xxxxxxxxxxxxxxxxxx>]

** Also affects: nova/kilo
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1444497

Title:
  Instance doesn't get an address via DHCP (nova-network) because of
  issue with live migration

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) kilo series:
  New

Bug description:
  When instance is migrated to another compute node, it's dhcp lease is not removed from the first compute node even after instance termination.
  If a new instance got the same IP which was present in the previous instance created on the the first compute node where dhcp lease for this IP remains, then the dnsmasq refuse DHCP request of the IP address for a new instance with different MAC.

  Steps to reproduce:
          Scenario:
              1. Create cluster (CentOS, nova-network with Flat-DHCP , Ceph for images and volumes)
              2. Add 1 node with controller and ceph OSD roles
              3. Add 2 node with compute and ceph OSD roles
              4. Deploy the cluster

              5. Create a VM
              6. Wait until the VM got IP address via DHCP (in VM console log)
              7. Migrate the VM to another compute node.
              8. Terminate the VM.

              9. Repeat stages from 5 to 8 several times (in my case - 4..6 times was enough) until a new instance stops receiving IP address via DHCP.
              10. Check dnsmasq-dhcp.log (/var/log/daemon.log on the compute node) for messages like :
  =============================================
  2014-11-09T20:28:29.671344+00:00 warning: not using configured address 10.0.0.2 because it is leased to fa:16:3e:65:70:be

  This means that:
     I. An instance was created on the compute node-1 and got a dhcp lease:
  ==== nova-dhcpbridge.log
  2014-11-09 20:12:03.811 27360 DEBUG nova.dhcpbridge [-] Called 'add' for mac 'fa:16:3e:65:70:be' with ip '10.0.0.2' main /usr/lib/python2.6/site-packages/nova/cmd/dhcpbridge.py:135

    II. When the instance was migrating from compute node-1 to node-3, 'dhcp_release' was not performed on compute node-1, please check the time range in the logs : 2014-11-09 20:14:36-37
  ==== Running.log (node-1)
  2014-11-09T20:14:36.647588+00:00 debug: cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf conntrack -D -r 10.0.0.2
  ### But there is missing a command like: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.2 fa:16:3e:65:70:be

    III. On the compute node-3, DHCP lease was added and it was successfully removed when the instance was terminated:
  ==== Running.log (node-3)
  2014-11-09T20:15:17.250243+00:00 debug: cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.2 fa:16:3e:65:70:be

    IV. When an another instance got the same address '10.0.0.2' and was created on node-1, it didn't get IP address via DHCP:
  ==== Running.log (node-1)
  2014-11-09T20:28:29.671344+00:00 warning: not using configured address 10.0.0.2 because it is leased to fa:16:3e:65:70:be

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1444497/+subscriptions