yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1348421] [NEW] VM Can't get DHCP IP

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ramy Allam <linux@xxxxxxxxxxxxx>
Date: Thu, 24 Jul 2014 23:34:37 -0000
Reply-to: Bug 1348421 <1348421@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hello,

I'm running OpenStack RDO with the following setup.

- 1x Controller Node ( Neutron - Nova -   glance - Horizon - Keystone - GRE +   FlatDHCPManager)
-  10x Compute Nodes (   Nova Compute  - KVM ) 
- OS : CentOS 6 - 64bit

Suddenly all vm's on the compute nodes can't get new IP address from the
controller node. It was working properly hours ago without any
modifications from me. I tried to restart neutron-dhcp-agent - qpidd and
dnsmasq with no luck.

This log from the controller node - **many errors**  
> http://paste.openstack.org/show/88016/

And this from the compute node
> http://paste.openstack.org/show/88017/

 Realtime compute node log after restarting vm network service
> http://paste.openstack.org/show/88039/

Installed packages on Controller Node :
# rpm -qa | grep -i openstack

    openstack-selinux-0.1.3-2.el6ost.noarch
    openstack-puppet-modules-2013.2-9.1.el6.noarch
    openstack-ceilometer-api-2013.2.3-2.el6.noarch
    openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch
    openstack-nova-scheduler-2013.2.3-1.el6.noarch
    openstack-ceilometer-common-2013.2.3-2.el6.noarch
    python-django-openstack-auth-1.1.2-1.el6.noarch
    openstack-ceilometer-central-2013.2.3-2.el6.noarch
    openstack-ceilometer-collector-2013.2.3-2.el6.noarch
    openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
    openstack-nova-common-2013.2.3-1.el6.noarch
    openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch
    openstack-glance-2013.2.3-2.el6.noarch
    openstack-nova-conductor-2013.2.3-1.el6.noarch
    openstack-nova-novncproxy-2013.2.3-1.el6.noarch
    openstack-nova-cert-2013.2.3-1.el6.noarch
    openstack-keystone-2013.2.3-3.el6.noarch
    openstack-neutron-2013.2.3-9.el6.noarch
    openstack-ceilometer-alarm-2013.2.3-2.el6.noarch
    openstack-dashboard-2013.2.3-1.el6.noarch
    openstack-nova-api-2013.2.3-1.el6.noarch
    openstack-nova-console-2013.2.3-1.el6.noarch
    openstack-utils-2013.2-2.el6.noarch

On Controller Node

# ps aux | grep -i dhcp
    nobody   12639  0.0  0.0  12884   864 ?        S    Jul19   0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7139b265-41 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/host --dhcp-optsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
    neutron  24884  0.0  0.1 273748 32140 ?        S    Jul24   0:00 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini

# ip netns
    qrouter-id
    qdhcp-id

# ip netns exec qdhcp-network-id ip a
    16: tap7139b265-41: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
        link/ether fa:16:3e:2e:18:35 brd ff:ff:ff:ff:ff:ff
        inet 10.0.0.3/24 brd 10.0.0.255 scope global tap7139b265-41
        inet6 fe80::f816:3eff:fe2e:1835/64 scope link 
           valid_lft forever preferred_lft forever
    17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    >> ip netns exec qdhcp-network-id ifconfig
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:4 errors:0 dropped:0 overruns:0 frame:0
              TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:2304 (2.2 KiB)  TX bytes:2304 (2.2 KiB)
    
    tap7139b265-41 Link encap:Ethernet  HWaddr FA:16:3E:2E:18:35  
              inet addr:10.0.0.3  Bcast:10.0.0.255  Mask:255.255.255.0
              inet6 addr: fe80::f816:3eff:fe2e:1835/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:91007 errors:0 dropped:0 overruns:0 frame:0
              TX packets:727 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:4633459 (4.4 MiB)  TX bytes:193767 (189.2 KiB)

Regards,

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1348421

Title:
  VM Can't get DHCP IP

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Hello,

  I'm running OpenStack RDO with the following setup.

  - 1x Controller Node ( Neutron - Nova -   glance - Horizon - Keystone - GRE +   FlatDHCPManager)
  -  10x Compute Nodes (   Nova Compute  - KVM ) 
  - OS : CentOS 6 - 64bit

  Suddenly all vm's on the compute nodes can't get new IP address from
  the controller node. It was working properly hours ago without any
  modifications from me. I tried to restart neutron-dhcp-agent - qpidd
  and dnsmasq with no luck.

  This log from the controller node - **many errors**  
  > http://paste.openstack.org/show/88016/

  And this from the compute node
  > http://paste.openstack.org/show/88017/

   Realtime compute node log after restarting vm network service
  > http://paste.openstack.org/show/88039/

  Installed packages on Controller Node :
  # rpm -qa | grep -i openstack

      openstack-selinux-0.1.3-2.el6ost.noarch
      openstack-puppet-modules-2013.2-9.1.el6.noarch
      openstack-ceilometer-api-2013.2.3-2.el6.noarch
      openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch
      openstack-nova-scheduler-2013.2.3-1.el6.noarch
      openstack-ceilometer-common-2013.2.3-2.el6.noarch
      python-django-openstack-auth-1.1.2-1.el6.noarch
      openstack-ceilometer-central-2013.2.3-2.el6.noarch
      openstack-ceilometer-collector-2013.2.3-2.el6.noarch
      openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
      openstack-nova-common-2013.2.3-1.el6.noarch
      openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch
      openstack-glance-2013.2.3-2.el6.noarch
      openstack-nova-conductor-2013.2.3-1.el6.noarch
      openstack-nova-novncproxy-2013.2.3-1.el6.noarch
      openstack-nova-cert-2013.2.3-1.el6.noarch
      openstack-keystone-2013.2.3-3.el6.noarch
      openstack-neutron-2013.2.3-9.el6.noarch
      openstack-ceilometer-alarm-2013.2.3-2.el6.noarch
      openstack-dashboard-2013.2.3-1.el6.noarch
      openstack-nova-api-2013.2.3-1.el6.noarch
      openstack-nova-console-2013.2.3-1.el6.noarch
      openstack-utils-2013.2-2.el6.noarch

  On Controller Node

  # ps aux | grep -i dhcp
      nobody   12639  0.0  0.0  12884   864 ?        S    Jul19   0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7139b265-41 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/host --dhcp-optsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
      neutron  24884  0.0  0.1 273748 32140 ?        S    Jul24   0:00 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini

  # ip netns
      qrouter-id
      qdhcp-id

  # ip netns exec qdhcp-network-id ip a
      16: tap7139b265-41: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
          link/ether fa:16:3e:2e:18:35 brd ff:ff:ff:ff:ff:ff
          inet 10.0.0.3/24 brd 10.0.0.255 scope global tap7139b265-41
          inet6 fe80::f816:3eff:fe2e:1835/64 scope link 
             valid_lft forever preferred_lft forever
      17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
          inet6 ::1/128 scope host 
             valid_lft forever preferred_lft forever
      >> ip netns exec qdhcp-network-id ifconfig
      lo        Link encap:Local Loopback  
                inet addr:127.0.0.1  Mask:255.0.0.0
                inet6 addr: ::1/128 Scope:Host
                UP LOOPBACK RUNNING  MTU:16436  Metric:1
                RX packets:4 errors:0 dropped:0 overruns:0 frame:0
                TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:0 
                RX bytes:2304 (2.2 KiB)  TX bytes:2304 (2.2 KiB)
      
      tap7139b265-41 Link encap:Ethernet  HWaddr FA:16:3E:2E:18:35  
                inet addr:10.0.0.3  Bcast:10.0.0.255  Mask:255.255.255.0
                inet6 addr: fe80::f816:3eff:fe2e:1835/64 Scope:Link
                UP BROADCAST RUNNING  MTU:1500  Metric:1
                RX packets:91007 errors:0 dropped:0 overruns:0 frame:0
                TX packets:727 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:0 
                RX bytes:4633459 (4.4 MiB)  TX bytes:193767 (189.2 KiB)

  Regards,

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1348421/+subscriptions
Follow ups

[Bug 1348421] Re: VM Can't get DHCP IP
From: Launchpad Bug Tracker, 2014-09-27
[Bug 1348421] [NEW] VM Can't get DHCP IP
From: Ramy Allam, 2014-07-24
References

[Bug 1348421] [NEW] VM Can't get DHCP IP
From: Ramy Allam, 2014-07-24