yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

I believe that https://review.openstack.org/#/c/273042/4 largely solves
this issue

** Changed in: nova
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1445569

Title:
  No dhcp lease after shelve unshelve

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  This may be related to 1290635 but I am not familiar enough with
  Nova's dhcp and shelve implementations to know for sure. Also the
  behavior I am seeing seems to be slightly different.

  In the multinode nova-net job
  (http://logs.openstack.org/88/174288/1/check/check-tempest-dsvm-
  multinode-full/3e3be58/) during tempest test_shelve_instance test we
  see dhcp fail when the shelved instance is unshelved:

  http://logs.openstack.org/88/174288/1/check/check-tempest-dsvm-multinode-full/3e3be58/console.html#_2015-04-16_11_26_00_029
  2015-04-16 11:26:00.029 |     Starting network...
  2015-04-16 11:26:00.029 |     udhcpc (v1.20.1) started
  2015-04-16 11:26:00.029 |     Sending discover...
  2015-04-16 11:26:00.029 |     Sending discover...
  2015-04-16 11:26:00.029 |     Sending discover...
  2015-04-16 11:26:00.029 |     No lease, failing
  2015-04-16 11:26:00.029 |     WARN: /etc/rc3.d/S40-network failed
  2015-04-16 11:26:00.029 |     cirros-ds 'net' up at 187.20

  Looking at tempest logs we find that node's MAC address
  (fa:16:3e:fb:3e:3e):

  http://logs.openstack.org/88/174288/1/check/check-tempest-dsvm-multinode-full/3e3be58/console.html#_2015-04-16_11_25_59_976
  2015-04-16 11:25:59.976 |             Body: {"server": {"status": "ACTIVE", "updated": "2015-04-16T11:16:44Z", "hostId": "d0dc2083935df1bf05cadea5c75358ee9d9e0406887667ea4bb582de", "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:fb:3e:3e", "version": 4, "addr": "10.1.0.6", "OS-EXT-IPS:type": "fixed"}, {"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:fb:3e:3e", "version": 4, "addr": "172.24.5.6", "OS-EXT-IPS:type": "floating"}]}, "links": [{"href": "http://10.208.224.113:8774/v2/b7b633c0117148628342ab9162d7885e/servers/0e9a79cd-96d5-4fcd-a0db-994638967291";, "rel": "self"}, {"href": "http://10.208.224.113:8774/b7b633c0117148628342ab9162d7885e/servers/0e9a79cd-96d5-4fcd-a0db-994638967291";, "rel": "bookmark"}], "key_name": "TestShelveInstance-494463835", "image": {"id": "18e4f345-a147-4d0a-922c-46b72b9497e9", "links": [{"href": "http://10.208.224.113:8774/b7b633c0117148628342ab9162d7885e/images/18e4f345-a147-4d0a-922c-46b72b9497e9";, "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2015-04-16T11:16:44.000000", "flavor": {"id": "42", "links": [{"href": "http://10.208.224.113:8774/b7b633c0117148628342ab9162d7885e/flavors/42";, "rel": "bookmark"}]}, "id": "0e9a79cd-96d5-4fcd-a0db-994638967291", "security_groups": [{"name": "TestShelveInstance-909686184"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "a02a8bd6d7734cd1a7aebbfdb4a3eb16", "name": "TestShelveInstance-512771961", "created": "2015-04-16T11:15:34Z", "tenant_id": "b7b633c0117148628342ab9162d7885e", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}}

  According to the logs above MAC addr fa:16:3e:fb:3e:3e should get IP
  10.1.0.6 but syslog shows:

  http://logs.openstack.org/88/174288/1/check/check-tempest-dsvm-multinode-full/3e3be58/logs/10.176.200.184-subnode/syslog.txt.gz#_Apr_16_11_16_00
  devstack-trusty-2-node-rax-iad-2194251-1687 dnsmasq-dhcp[22310]: DHCPDISCOVER(br100) fa:16:3e:fb:3e:3e no address available

  I think this points at one of two problems. Either there is a race
  between booting a node and setting its dnsmasq config with nova-net or
  nova-net is never setting the dnsmasq config in the first place.

  If it is a race then this likely affects other operations. If it is
  never set at all that may be a shelve unshelve specific issue with
  restoring instance state.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1445569/+subscriptions