yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #21432
[Bug 1072014] Re: instance cannot get ip automatically under FlatDHCP mode
** Changed in: nova
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1072014
Title:
instance cannot get ip automatically under FlatDHCP mode
Status in OpenStack Compute (Nova):
Invalid
Bug description:
I installed the folsom-2012.2 nova/glance/keystone in a singe node and all things seemed right
except for that the instance can't get their private fixed IP via DHCP protocol under FlatDHCP network mode.
After I used vnc to get access to vm instance and use command `ip addr add %FIXED-IP%/%NETMASK% dev eth0` to configure
the ip address for instance manually, the network of the instance became to work.
(the image of the instances is debian-6.0.4-amd64-standard)
Then I tried to use tcpdump to capture the udp packets on br100(which nova-network is using), found that the dnsmasq(which acted as dhcp server)
didn't respond to the DHCPDISCOVER requests send by dhcp client inside the instance.
I also found similar problem reported by others, but in this case my server's kernel version is 3.2
DHCP broken for Openstack Nova instances since kernel v3.3
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1035172
Bellow are debug infos, can anyone give some help? thx
kernel version
hzzhanggy % uname -a
Linux DEV6 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux
dnsmasq version
hzzhanggy % sudo aptitude show dnsmasq
Package: dnsmasq
State: installed
Automatically installed: no
Version: 2.63-4
Priority: optional
Section: net
Maintainer: Simon Kelley <simon@xxxxxxxxxxxxxxxxx>
Architecture: all
Uncompressed Size: 39.9 k
Depends: netbase, dnsmasq-base (>= 2.63-4)
Suggests: resolvconf
Conflicts: resolvconf (< 1.15)
Description: Small caching DNS proxy and DHCP/TFTP server
Dnsmasq is a lightweight, easy to configure, DNS forwarder and DHCP server. It is designed to provide DNS and optionally, DHCP, to a small network. It can serve the names of local machines which are
not in the global DNS. The DHCP server integrates with the DNS server and allows machines with DHCP-allocated addresses to appear in the DNS with names configured either in each host or in a central
configuration file. Dnsmasq supports static and dynamic DHCP leases and BOOTP/TFTP for network booting of diskless machines.
network interface
hzzhanggy % ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet 169.254.169.254/32 scope link lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 5c:f3:fc:98:97:d8 brd ff:ff:ff:ff:ff:ff
inet xxx.xxx.xxx.6/24 brd xxx.xxx.xxx.255 scope global eth0
inet6 fe80::5ef3:fcff:fe98:97d8/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br100 state UP qlen 1000
link/ether 5c:f3:fc:98:97:da brd ff:ff:ff:ff:ff:ff
inet6 fe80::5ef3:fcff:fe98:97da/64 scope link
valid_lft forever preferred_lft forever
4: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 5e:f3:fc:9c:97:db brd ff:ff:ff:ff:ff:ff
78: br101: <BROADCAST,MULTICAST,PROMISC> mtu 1500 qdisc noqueue state DOWN
link/ether 2a:bc:e5:2f:4b:4c brd ff:ff:ff:ff:ff:ff
inet 10.120.33.1/25 brd 10.120.33.127 scope global br101
81: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 5c:f3:fc:98:97:da brd ff:ff:ff:ff:ff:ff
inet 10.120.33.1/25 brd 10.120.33.127 scope global br100
inet6 fe80::5ef3:fcff:fe98:97da/64 scope link
valid_lft forever preferred_lft forever
102: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br100 state UNKNOWN qlen 500
link/ether fe:16:3e:06:c4:71 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe06:c471/64 scope link
valid_lft forever preferred_lft forever
103: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br100 state UNKNOWN qlen 500
link/ether fe:16:3e:08:1d:0d brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe08:1d0d/64 scope link
valid_lft forever preferred_lft forever
dnsmasq process
hzzhanggy % ps aux|grep dnsmasq
nobody 24867 0.0 0.0 21360 952 ? SN 17:49 0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/home/hzzhanggy/lib/nova/networks/nova-br100.pid --listen-address=10.120.33.1 --except-interface=lo --dhcp-range=set:'private',10.120.33.2,static,120s --dhcp-lease-max=128 --dhcp-hostsfile=/home/hzzhanggy/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/local/bin/nova-dhcpbridge --leasefile-ro
root 24868 0.0 0.0 21332 420 ? SN 17:49 0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/home/hzzhanggy/lib/nova/networks/nova-br100.pid --listen-address=10.120.33.1 --except-interface=lo --dhcp-range=set:'private',10.120.33.2,static,120s --dhcp-lease-max=128 --dhcp-hostsfile=/home/hzzhanggy/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/local/bin/nova-dhcpbridge --leasefile-ro
the dnamasq listening port
hzzhanggy % sudo netstat -anpu|grep dns
udp 0 0 10.120.33.1:53 0.0.0.0:* 24867/dnsmasq
udp 0 0 0.0.0.0:67 0.0.0.0:* 24867/dnsmasq
the content of dhcp-hostsfile
hzzhanggy % cat ~/lib/nova/networks/nova-br100.conf
fa:16:3e:08:1d:0d,server-001.novalocal,10.120.33.44
fa:16:3e:06:c4:71,server-002.novalocal,10.120.33.45
run dhclient in instance with mac "fa:16:3e:08:1d:0d", while running tcpdump to capture udp packets on br100 on openstack server,
just DHCPDISCOVER packets captured.
root # dhclient eth0
hzzhanggy % sudo tcpdump -v -i br100 udp
22:01:03.347651 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:08:1d:0d (oui Unknown), length 300, xid 0xd5a2dd48, Flags [none]
Client-Ethernet-Address fa:16:3e:08:1d:0d (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Parameter-Request Option 55, length 13:
Subnet-Mask, BR, Time-Zone, Default-Gateway
Domain-Name, Domain-Name-Server, Option 119, Hostname
Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
NTP
22:01:05.351754 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:08:1d:0d (oui Unknown), length 300, xid 0xa59d1867, secs 8, Flags [none]
Client-Ethernet-Address fa:16:3e:08:1d:0d (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Parameter-Request Option 55, length 13:
Subnet-Mask, BR, Time-Zone, Default-Gateway
Domain-Name, Domain-Name-Server, Option 119, Hostname
Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
NTP
22:01:05.443905 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:08:1d:0d (oui Unknown), length 300, xid 0x56275753, Flags [none]
Client-Ethernet-Address fa:16:3e:08:1d:0d (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Parameter-Request Option 55, length 13:
Subnet-Mask, BR, Time-Zone, Default-Gateway
Domain-Name, Domain-Name-Server, Option 119, Hostname
Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
NTP
22:01:06.346870 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:08:1d:0d (oui Unknown), length 300, xid 0xd5a2dd48, secs 3, Flags [none]
Client-Ethernet-Address fa:16:3e:08:1d:0d (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Parameter-Request Option 55, length 13:
Subnet-Mask, BR, Time-Zone, Default-Gateway
Domain-Name, Domain-Name-Server, Option 119, Hostname
Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
NTP
trace iptables, the dhcp request matched an ACCEPT rule
hzzhanggy % dmesg|grep "fa:16:3e:08:1d:0d"|tail -n 10
[1481865.887860] TRACE: filter:nova-network-local:return:1 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887874] TRACE: filter:nova-filter-top:rule:2 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887888] TRACE: filter:nova-compute-local:return:3 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887903] TRACE: filter:nova-filter-top:rule:3 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887916] TRACE: filter:nova-api-local:return:1 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887931] TRACE: filter:nova-filter-top:return:4 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887943] TRACE: filter:FORWARD:rule:2 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887958] TRACE: filter:nova-network-FORWARD:rule:1 IN=br100 OUT=br100 PHYSIN=vnet0 PHYSOUT=eth1 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887980] TRACE: filter:INPUT:rule:1 IN=br100 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[1481865.887995] TRACE: filter:nova-network-INPUT:rule:1 IN=br100 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:fa:16:3e:08:1d:0d:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
hzzhanggy % sudo iptables-save|grep nova-network-INPUT
:nova-network-INPUT - [0:0]
-A INPUT -j nova-network-INPUT
-A nova-network-INPUT -i br100 -p udp -m udp --dport 67 -j ACCEPT
-A nova-network-INPUT -i br100 -p tcp -m tcp --dport 67 -j ACCEPT
-A nova-network-INPUT -i br100 -p udp -m udp --dport 53 -j ACCEPT
-A nova-network-INPUT -i br100 -p tcp -m tcp --dport 53 -j ACCEPT
strace dnsmasq, found that the dnsmasq wait at the "select" operation, seemded not data reveiced
hzzhanggy % sudo strace -p 24867
time(NULL) = 1351256984
fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
recvmsg(4, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000400}, msg_iov(1)=[{"\200\0\0\0\31\0\0\0\0\0\0\0\0\0\0\0\n\200\0\0\377\0\0\1\0\2\0\0\10\0\17\0"..., 384}], msg_controllen=0, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 128
recvmsg(4, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000400}, msg_iov(1)=[{"\200\0\0\0\31\0\0\0\0\0\0\0\0\0\0\0\n\200\0\0\377\0\0\1\0\2\0\0\10\0\17\0"..., 384}], msg_controllen=0, msg_flags=0}, 0) = 128
fcntl(4, F_SETFL, O_RDWR) = 0
select(8, [3 4 5 6 7], [], [], NULL
lsof result of dnsmasq
hzzhanggy % sudo lsof -p 24867
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dnsmasq 24867 nobody cwd DIR 8,1 4096 2 /
dnsmasq 24867 nobody rtd DIR 8,1 4096 2 /
dnsmasq 24867 nobody txt REG 8,1 253592 81988 /usr/sbin/dnsmasq
dnsmasq 24867 nobody mem REG 8,1 14768 123450 /lib/x86_64-linux-gnu/libdl-2.13.so
dnsmasq 24867 nobody mem REG 8,1 31744 123456 /lib/x86_64-linux-gnu/librt-2.13.so
dnsmasq 24867 nobody mem REG 8,1 131107 123454 /lib/x86_64-linux-gnu/libpthread-2.13.so
dnsmasq 24867 nobody mem REG 8,1 1583120 123435 /lib/x86_64-linux-gnu/libc-2.13.so
dnsmasq 24867 nobody mem REG 8,1 211376 123191 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
dnsmasq 24867 nobody mem REG 8,1 24712 82068 /usr/lib/libnfnetlink.so.0.2.0
dnsmasq 24867 nobody mem REG 8,1 96520 123284 /usr/lib/x86_64-linux-gnu/libnetfilter_conntrack.so.3.3.0
dnsmasq 24867 nobody mem REG 8,1 286488 123894 /lib/x86_64-linux-gnu/libdbus-1.so.3.7.2
dnsmasq 24867 nobody mem REG 8,1 136936 123458 /lib/x86_64-linux-gnu/ld-2.13.so
dnsmasq 24867 nobody mem REG 8,6 217016 851971 /var/cache/nscd/group
dnsmasq 24867 nobody mem REG 8,6 217016 851970 /var/cache/nscd/passwd
dnsmasq 24867 nobody mem REG 8,1 1534672 92791 /usr/lib/locale/locale-archive
dnsmasq 24867 nobody mem REG 8,1 26066 123643 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
dnsmasq 24867 nobody 0u CHR 1,3 0t0 1028 /dev/null
dnsmasq 24867 nobody 1u CHR 1,3 0t0 1028 /dev/null
dnsmasq 24867 nobody 2u CHR 1,3 0t0 1028 /dev/null
dnsmasq 24867 nobody 3u IPv4 387210551 0t0 UDP *:bootps
dnsmasq 24867 nobody 4u netlink 0t0 387210552 ROUTE
dnsmasq 24867 nobody 5u IPv4 387210563 0t0 UDP 10.120.33.1:domain
dnsmasq 24867 nobody 6u IPv4 387210564 0t0 TCP 10.120.33.1:domain (LISTEN)
dnsmasq 24867 nobody 7r FIFO 0,8 0t0 387208289 pipe
dnsmasq 24867 nobody 8w FIFO 0,8 0t0 387208289 pipe
dnsmasq 24867 nobody 9u unix 0xffff880665f44c40 0t0 387210157 socket
dnsmasq 24867 nobody 12w FIFO 0,8 0t0 387210158 pipe
but when I kill the dnsmasq process and use nc to listen on udp 67 port, it definately could received something.
hzzhanggy % _ nc -l -u -p 67|xxd
0000000: 0101 0600 c12f 2d0f 0014 0000 0000 0000 ...../-.........
0000010: 0000 0000 0000 0000 0000 0000 fa16 3e75 ..............>u
0000020: 38c3 0000 0000 0000 0000 0000 0000 0000 8...............
0000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000e0: 0000 0000 0000 0000 0000 0000 6382 5363 ............c.Sc
00000f0: 3501 0137 0d01 1c02 030f 0677 0c2c 2f1a 5..7.......w.,/.
0000100: 792a ff00 0000 0000 0000 0000 0000 0000 y*..............
0000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1072014/+subscriptions
