openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #12591
inter vm communication issue
Hi all,
Can I request some help in resolving a vlan networking issue we are encountering in the final stages of our openstack installation?
We have installed a multi host vlan network configuration on 3 hosts all running ubuntu 12.04 (openstack essex ).
One of these hosts is a "public" host running the compute and network services, the other 2 hosts are on a private vlan and are running compute and network as well as all other components of the openstack installation.
All physical hosts have 2 nic's in a bond (for redundancy) configured with an ip in the 10.0.0.0/24 range as a private network.
The vm networks we have created are in the 192.168.0.0/16 range and the appropriate vlan tagged networks have been created on the switch.
All openstack components are running fine as we can create, run and live migrate instances with no issues. All vm's can contact all physical hosts in the 10.0.0.0/24 range as well as the outside word using a proxy running on the 10.0.0.254 ip.
The problem arrises when we try to communicate in between vm's running on different hosts:
- name resolution is not working for vm's running on different physical hosts ( I suppose dns should work, no? )
- all packages of communication performed using the ip of the vm directly ( ping, ssh, ...) are arriving on the bridge interface of the physical host running the vm we are tying to reach, but the vm itself is not picking up or responding to the requests...
The weird thing is, when we start 2 vm's on the same physical host, name resolution and networking are working fine. When we then live-migrate one of the vm's to a new physical host, the networking will continue to work for a varying amount of time after the live migration has completed! A variable amount of the packages start getting lost until we end up with no communication being possible in between the virtual machines. ( after new dhcp lease? arp table getting flushed?... )
As no errors are appearing in any of the nova logs (all on verbose...) or in the syslog (from the dnsmasq) I really have no clue as to what might be causing this issue... or is it a bug?
My feeling is the per physical host vm gateway is not performing as it should and not routing the packages correctly in between physical hosts but I have no idea on how to check this other than capture the packages on the bridge interface and observe the requests not getting answered...
Another option is the problem residing with the 2 physical interfaces in the network bond... but wireshark is showing all packages are arriving on the bridge interface where the vm we are trying to reach is residing so this seems unlikely?
I have included the nova.conf the ifconfig and the iptables (+nat) of one of the physical hosts in this mail but can provide any other output if this might be helpful.
Kind regards,
Bram
###################
# /etc/nova/nova.conf
###################
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
##--force_dhcp_release
##--iscsi_helper=tgtadm
--libvirt_use_virtio_for_bridges
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose
--ec2_private_dns_show_ip
--auth_strategy=keystone
--rabbit_host=10.0.0.100
--nova_url=http://10.0.0.100:8774/v1.1/
--floating_range=999.999.999.0/24
--fixed_range=192.168.0.0/16
--routing_source_ip=10.0.0.103
--sql_connection=postgresql://clouddbadmin:password@10.0.0.100/nova
--glance_api_servers=10.0.0.100:9292
--image_service=nova.image.glance.GlanceImageService
--network_manager=nova.network.manager.VlanManager
--vlan_interface=bond0
--public_interface=eth0
--multi-host=true
###################
# ifconfig
###################
bond0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a
inet addr:10.0.0.103 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::be30:5bff:fedd:c8a/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:1400289 errors:0 dropped:67725 overruns:0 frame:0
TX packets:2414277 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1288957456 (1.2 GB) TX bytes:3217320483 (3.2 GB)
br1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::182b:5aff:feda:38f3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:488 (488.0 B) TX bytes:4940 (4.9 KB)
br1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab
inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::5014:d5ff:fe05:93dd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4200 errors:0 dropped:15 overruns:0 frame:0
TX packets:5024 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:433834 (433.8 KB) TX bytes:20260632 (20.2 MB)
eth0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:86
inet addr:999.999.999.58 Bcast:999.999.999.255 Mask:255.255.255.0
inet6 addr: fe80::be30:5bff:fedd:c86/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:38664 errors:0 dropped:246 overruns:0 frame:0
TX packets:27311 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5127536 (5.1 MB) TX bytes:28006322 (28.0 MB)
Interrupt:36 Memory:d6000000-d6012800
eth1 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:88
inet addr:157.193.229.69 Bcast:157.193.229.255 Mask:255.255.255.0
inet6 addr: fe80::be30:5bff:fedd:c88/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:21745 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2593490 (2.5 MB) TX bytes:1312 (1.3 KB)
Interrupt:48 Memory:d8000000-d8012800
eth2 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:322566 errors:0 dropped:2 overruns:0 frame:0
TX packets:1132927 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:171375115 (171.3 MB) TX bytes:1563837296 (1.5 GB)
Interrupt:32 Memory:da000000-da012800
eth3 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8c
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1077723 errors:0 dropped:67478 overruns:0 frame:0
TX packets:1281350 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1117582341 (1.1 GB) TX bytes:1653483187 (1.6 GB)
Interrupt:42 Memory:dc000000-dc012800
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:342519 errors:0 dropped:0 overruns:0 frame:0
TX packets:342519 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3762417359 (3.7 GB) TX bytes:3762417359 (3.7 GB)
virbr0 Link encap:Ethernet HWaddr ce:c0:87:1e:39:52
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
vlan1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f
inet6 addr: fe80::f816:3eff:fe50:1f3f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:9 errors:0 dropped:0 overruns:0 frame:0
TX packets:116 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:534 (534.0 B) TX bytes:7756 (7.7 KB)
vlan1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab
inet6 addr: fe80::f816:3eff:fe1e:4aab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:482 errors:0 dropped:0 overruns:0 frame:0
TX packets:497 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:34886 (34.8 KB) TX bytes:50938 (50.9 KB)
vnet2 Link encap:Ethernet HWaddr fe:16:3e:6c:af:bc
inet6 addr: fe80::fc16:3eff:fe6c:afbc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:383 errors:0 dropped:0 overruns:0 frame:0
TX packets:280 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:84937 (84.9 KB) TX bytes:39749 (39.7 KB)
###################
# sudo iptables -L
###################
Chain INPUT (policy ACCEPT)
target prot opt source destination
nova-compute-INPUT all -- anywhere anywhere
nova-network-INPUT all -- anywhere anywhere
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
Chain FORWARD (policy ACCEPT)
target prot opt source destination
nova-filter-top all -- anywhere anywhere
nova-compute-FORWARD all -- anywhere anywhere
nova-network-FORWARD all -- anywhere anywhere
ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
nova-filter-top all -- anywhere anywhere
nova-compute-OUTPUT all -- anywhere anywhere
nova-network-OUTPUT all -- anywhere anywhere
Chain nova-compute-FORWARD (1 references)
target prot opt source destination
Chain nova-compute-INPUT (1 references)
target prot opt source destination
Chain nova-compute-OUTPUT (1 references)
target prot opt source destination
Chain nova-compute-inst-97 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere state INVALID
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
nova-compute-provider all -- anywhere anywhere
ACCEPT udp -- 192.168.0.4 anywhere udp spt:bootps dpt:bootpc
ACCEPT all -- 192.168.0.0/24 anywhere
ACCEPT icmp -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
nova-compute-sg-fallback all -- anywhere anywhere
Chain nova-compute-local (1 references)
target prot opt source destination
nova-compute-inst-97 all -- anywhere 192.168.0.40
Chain nova-compute-provider (1 references)
target prot opt source destination
Chain nova-compute-sg-fallback (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
Chain nova-filter-top (2 references)
target prot opt source destination
nova-compute-local all -- anywhere anywhere
nova-network-local all -- anywhere anywhere
Chain nova-network-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT udp -- anywhere 192.168.1.2 udp dpt:openvpn
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT udp -- anywhere 192.168.0.2 udp dpt:openvpn
Chain nova-network-INPUT (1 references)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
Chain nova-network-OUTPUT (1 references)
target prot opt source destination
Chain nova-network-local (1 references)
target prot opt source destination
###################
# sudo iptables -L -t nat
###################
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
nova-compute-PREROUTING all -- anywhere anywhere
nova-network-PREROUTING all -- anywhere anywhere
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
nova-compute-OUTPUT all -- anywhere anywhere
nova-network-OUTPUT all -- anywhere anywhere
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
nova-compute-POSTROUTING all -- anywhere anywhere
nova-network-POSTROUTING all -- anywhere anywhere
nova-postrouting-bottom all -- anywhere anywhere
Chain nova-compute-OUTPUT (1 references)
target prot opt source destination
Chain nova-compute-POSTROUTING (1 references)
target prot opt source destination
Chain nova-compute-PREROUTING (1 references)
target prot opt source destination
Chain nova-compute-float-snat (1 references)
target prot opt source destination
Chain nova-compute-snat (1 references)
target prot opt source destination
nova-compute-float-snat all -- anywhere anywhere
Chain nova-network-OUTPUT (1 references)
target prot opt source destination
DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.1.2:1194
DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.0.2:1194
Chain nova-network-POSTROUTING (1 references)
target prot opt source destination
ACCEPT all -- 192.168.0.0/16 999.999.999.58
ACCEPT all -- 192.168.0.0/16 10.128.0.0/24
ACCEPT all -- 192.168.0.0/16 192.168.0.0/16 ! ctstate DNAT
Chain nova-network-PREROUTING (1 references)
target prot opt source destination
DNAT tcp -- anywhere 169.254.169.254 tcp dpt:http to:999.999.999.58:8775
DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.1.2:1194
DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.0.2:1194
Chain nova-network-float-snat (1 references)
target prot opt source destination
Chain nova-network-snat (1 references)
target prot opt source destination
nova-network-float-snat all -- anywhere anywhere
SNAT all -- 192.168.0.0/16 anywhere to:10.0.0.103
Chain nova-postrouting-bottom (1 references)
target prot opt source destination
nova-compute-snat all -- anywhere anywhere
nova-network-snat all -- anywhere anywhere
Follow ups
