openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #12609
Re: inter vm communication issue
Thanx Vish,
On the name resolution: would you consider this a bug (I can file one if you would like) or a feature? Could this be fixed by changing the /usr/bin/nova-dhcpbridge script to load all mac, hostname, ip combinations for the database instead of just the physical hosts one? Or would this create other issues?
Security rules are setup correctly I guess, as all traffic to and from vm's running on the same host is not experiencing any issues.
nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp | -1 | -1 | 0.0.0.0/0 | |
| tcp | 22 | 22 | 0.0.0.0/0 | |
+-------------+-----------+---------+-----------+--------------+
The bonding might indeed be an issue, we are currently running a adaptive load balancing bond, thus the physical traffic can jump for one physical interface to the other at any time...
I will try an disable the bonds and get back to you ass soon as I have done that.
Kind regards
bram
On 1-jun-2012, at 09:04, Vishvananda Ishaya wrote:
> Ideas inline.
>
> Vish
>
> On May 31, 2012, at 1:41 PM, Bram De Wilde wrote:
>
>> Hi all,
>>
>> Can I request some help in resolving a vlan networking issue we are encountering in the final stages of our openstack installation?
>>
>> We have installed a multi host vlan network configuration on 3 hosts all running ubuntu 12.04 (openstack essex ).
>>
>> One of these hosts is a "public" host running the compute and network services, the other 2 hosts are on a private vlan and are running compute and network as well as all other components of the openstack installation.
>> All physical hosts have 2 nic's in a bond (for redundancy) configured with an ip in the 10.0.0.0/24 range as a private network.
>>
>> The vm networks we have created are in the 192.168.0.0/16 range and the appropriate vlan tagged networks have been created on the switch.
>>
>> All openstack components are running fine as we can create, run and live migrate instances with no issues. All vm's can contact all physical hosts in the 10.0.0.0/24 range as well as the outside word using a proxy running on the 10.0.0.254 ip.
>>
>> The problem arrises when we try to communicate in between vm's running on different hosts:
>> - name resolution is not working for vm's running on different physical hosts ( I suppose dns should work, no? )
>
> This is expected in multihost mode. The copy of dnsmasq that runs on each host only knows about its own vms. You will need to set up a shared dns if you really need this to work.
>
>> - all packages of communication performed using the ip of the vm directly ( ping, ssh, ...) are arriving on the bridge interface of the physical host running the vm we are tying to reach, but the vm itself is not picking up or responding to the requests...
>
> Have you set up security group rules to allow the traffic? That is the only reason I can think that packets wouldn't be getting into the vnet if it is showing up on the bridge. There is also a possiblity that bonding + bridging + vlans has some sort of an issue.
>
>>
>> The weird thing is, when we start 2 vm's on the same physical host, name resolution and networking are working fine. When we then live-migrate one of the vm's to a new physical host, the networking will continue to work for a varying amount of time after the live migration has completed! A variable amount of the packages start getting lost until we end up with no communication being possible in between the virtual machines. ( after new dhcp lease? arp table getting flushed?... )
>>
>> As no errors are appearing in any of the nova logs (all on verbose...) or in the syslog (from the dnsmasq) I really have no clue as to what might be causing this issue... or is it a bug?
>>
>> My feeling is the per physical host vm gateway is not performing as it should and not routing the packages correctly in between physical hosts but I have no idea on how to check this other than capture the packages on the bridge interface and observe the requests not getting answered...
>> Another option is the problem residing with the 2 physical interfaces in the network bond... but wireshark is showing all packages are arriving on the bridge interface where the vm we are trying to reach is residing so this seems unlikely?
>>
>> I have included the nova.conf the ifconfig and the iptables (+nat) of one of the physical hosts in this mail but can provide any other output if this might be helpful.
>>
>> Kind regards,
>> Bram
>>
>> ###################
>> # /etc/nova/nova.conf
>> ###################
>>
>> --dhcpbridge_flagfile=/etc/nova/nova.conf
>> --dhcpbridge=/usr/bin/nova-dhcpbridge
>> --logdir=/var/log/nova
>> --state_path=/var/lib/nova
>> --lock_path=/var/lock/nova
>> ##--force_dhcp_release
>> ##--iscsi_helper=tgtadm
>> --libvirt_use_virtio_for_bridges
>> --connection_type=libvirt
>> --root_helper=sudo nova-rootwrap
>> --verbose
>> --ec2_private_dns_show_ip
>> --auth_strategy=keystone
>> --rabbit_host=10.0.0.100
>> --nova_url=http://10.0.0.100:8774/v1.1/
>> --floating_range=999.999.999.0/24
>> --fixed_range=192.168.0.0/16
>> --routing_source_ip=10.0.0.103
>> --sql_connection=postgresql://clouddbadmin:password@10.0.0.100/nova
>> --glance_api_servers=10.0.0.100:9292
>> --image_service=nova.image.glance.GlanceImageService
>> --network_manager=nova.network.manager.VlanManager
>> --vlan_interface=bond0
>> --public_interface=eth0
>> --multi-host=true
>>
>> ###################
>> # ifconfig
>> ###################
>>
>> bond0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a
>> inet addr:10.0.0.103 Bcast:10.0.0.255 Mask:255.255.255.0
>> inet6 addr: fe80::be30:5bff:fedd:c8a/64 Scope:Link
>> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
>> RX packets:1400289 errors:0 dropped:67725 overruns:0 frame:0
>> TX packets:2414277 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:1288957456 (1.2 GB) TX bytes:3217320483 (3.2 GB)
>>
>> br1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f
>> inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
>> inet6 addr: fe80::182b:5aff:feda:38f3/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:8 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:488 (488.0 B) TX bytes:4940 (4.9 KB)
>>
>> br1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab
>> inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0
>> inet6 addr: fe80::5014:d5ff:fe05:93dd/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:4200 errors:0 dropped:15 overruns:0 frame:0
>> TX packets:5024 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:433834 (433.8 KB) TX bytes:20260632 (20.2 MB)
>>
>> eth0 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:86
>> inet addr:999.999.999.58 Bcast:999.999.999.255 Mask:255.255.255.0
>> inet6 addr: fe80::be30:5bff:fedd:c86/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:38664 errors:0 dropped:246 overruns:0 frame:0
>> TX packets:27311 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:5127536 (5.1 MB) TX bytes:28006322 (28.0 MB)
>> Interrupt:36 Memory:d6000000-d6012800
>>
>> eth1 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:88
>> inet addr:157.193.229.69 Bcast:157.193.229.255 Mask:255.255.255.0
>> inet6 addr: fe80::be30:5bff:fedd:c88/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:21745 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:2593490 (2.5 MB) TX bytes:1312 (1.3 KB)
>> Interrupt:48 Memory:d8000000-d8012800
>>
>> eth2 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8a
>> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
>> RX packets:322566 errors:0 dropped:2 overruns:0 frame:0
>> TX packets:1132927 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:171375115 (171.3 MB) TX bytes:1563837296 (1.5 GB)
>> Interrupt:32 Memory:da000000-da012800
>>
>> eth3 Link encap:Ethernet HWaddr bc:30:5b:dd:0c:8c
>> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
>> RX packets:1077723 errors:0 dropped:67478 overruns:0 frame:0
>> TX packets:1281350 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:1117582341 (1.1 GB) TX bytes:1653483187 (1.6 GB)
>> Interrupt:42 Memory:dc000000-dc012800
>>
>> lo Link encap:Local Loopback
>> inet addr:127.0.0.1 Mask:255.0.0.0
>> inet6 addr: ::1/128 Scope:Host
>> UP LOOPBACK RUNNING MTU:16436 Metric:1
>> RX packets:342519 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:342519 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:3762417359 (3.7 GB) TX bytes:3762417359 (3.7 GB)
>>
>> virbr0 Link encap:Ethernet HWaddr ce:c0:87:1e:39:52
>> inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
>> UP BROADCAST MULTICAST MTU:1500 Metric:1
>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
>>
>> vlan1997 Link encap:Ethernet HWaddr fa:16:3e:50:1f:3f
>> inet6 addr: fe80::f816:3eff:fe50:1f3f/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:9 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:116 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:534 (534.0 B) TX bytes:7756 (7.7 KB)
>>
>> vlan1998 Link encap:Ethernet HWaddr fa:16:3e:1e:4a:ab
>> inet6 addr: fe80::f816:3eff:fe1e:4aab/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:482 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:497 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:34886 (34.8 KB) TX bytes:50938 (50.9 KB)
>>
>> vnet2 Link encap:Ethernet HWaddr fe:16:3e:6c:af:bc
>> inet6 addr: fe80::fc16:3eff:fe6c:afbc/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:383 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:280 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:500
>> RX bytes:84937 (84.9 KB) TX bytes:39749 (39.7 KB)
>>
>>
>> ###################
>> # sudo iptables -L
>> ###################
>>
>> Chain INPUT (policy ACCEPT)
>> target prot opt source destination
>> nova-compute-INPUT all -- anywhere anywhere
>> nova-network-INPUT all -- anywhere anywhere
>> ACCEPT udp -- anywhere anywhere udp dpt:domain
>> ACCEPT tcp -- anywhere anywhere tcp dpt:domain
>> ACCEPT udp -- anywhere anywhere udp dpt:bootps
>> ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
>>
>> Chain FORWARD (policy ACCEPT)
>> target prot opt source destination
>> nova-filter-top all -- anywhere anywhere
>> nova-compute-FORWARD all -- anywhere anywhere
>> nova-network-FORWARD all -- anywhere anywhere
>> ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED
>> ACCEPT all -- 192.168.122.0/24 anywhere
>> ACCEPT all -- anywhere anywhere
>> REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
>> REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
>>
>> Chain OUTPUT (policy ACCEPT)
>> target prot opt source destination
>> nova-filter-top all -- anywhere anywhere
>> nova-compute-OUTPUT all -- anywhere anywhere
>> nova-network-OUTPUT all -- anywhere anywhere
>>
>> Chain nova-compute-FORWARD (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-INPUT (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-OUTPUT (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-inst-97 (1 references)
>> target prot opt source destination
>> DROP all -- anywhere anywhere state INVALID
>> ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
>> nova-compute-provider all -- anywhere anywhere
>> ACCEPT udp -- 192.168.0.4 anywhere udp spt:bootps dpt:bootpc
>> ACCEPT all -- 192.168.0.0/24 anywhere
>> ACCEPT icmp -- anywhere anywhere
>> ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
>> nova-compute-sg-fallback all -- anywhere anywhere
>>
>> Chain nova-compute-local (1 references)
>> target prot opt source destination
>> nova-compute-inst-97 all -- anywhere 192.168.0.40
>>
>> Chain nova-compute-provider (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-sg-fallback (1 references)
>> target prot opt source destination
>> DROP all -- anywhere anywhere
>>
>> Chain nova-filter-top (2 references)
>> target prot opt source destination
>> nova-compute-local all -- anywhere anywhere
>> nova-network-local all -- anywhere anywhere
>>
>> Chain nova-network-FORWARD (1 references)
>> target prot opt source destination
>> ACCEPT all -- anywhere anywhere
>> ACCEPT all -- anywhere anywhere
>> ACCEPT udp -- anywhere 192.168.1.2 udp dpt:openvpn
>> ACCEPT all -- anywhere anywhere
>> ACCEPT all -- anywhere anywhere
>> ACCEPT udp -- anywhere 192.168.0.2 udp dpt:openvpn
>>
>> Chain nova-network-INPUT (1 references)
>> target prot opt source destination
>> ACCEPT udp -- anywhere anywhere udp dpt:bootps
>> ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
>> ACCEPT udp -- anywhere anywhere udp dpt:domain
>> ACCEPT tcp -- anywhere anywhere tcp dpt:domain
>> ACCEPT udp -- anywhere anywhere udp dpt:bootps
>> ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
>> ACCEPT udp -- anywhere anywhere udp dpt:domain
>> ACCEPT tcp -- anywhere anywhere tcp dpt:domain
>>
>> Chain nova-network-OUTPUT (1 references)
>> target prot opt source destination
>>
>> Chain nova-network-local (1 references)
>> target prot opt source destination
>>
>> ###################
>> # sudo iptables -L -t nat
>> ###################
>>
>> Chain PREROUTING (policy ACCEPT)
>> target prot opt source destination
>> nova-compute-PREROUTING all -- anywhere anywhere
>> nova-network-PREROUTING all -- anywhere anywhere
>>
>> Chain INPUT (policy ACCEPT)
>> target prot opt source destination
>>
>> Chain OUTPUT (policy ACCEPT)
>> target prot opt source destination
>> nova-compute-OUTPUT all -- anywhere anywhere
>> nova-network-OUTPUT all -- anywhere anywhere
>>
>> Chain POSTROUTING (policy ACCEPT)
>> target prot opt source destination
>> nova-compute-POSTROUTING all -- anywhere anywhere
>> nova-network-POSTROUTING all -- anywhere anywhere
>> nova-postrouting-bottom all -- anywhere anywhere
>>
>> Chain nova-compute-OUTPUT (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-POSTROUTING (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-PREROUTING (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-float-snat (1 references)
>> target prot opt source destination
>>
>> Chain nova-compute-snat (1 references)
>> target prot opt source destination
>> nova-compute-float-snat all -- anywhere anywhere
>>
>> Chain nova-network-OUTPUT (1 references)
>> target prot opt source destination
>> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.1.2:1194
>> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.0.2:1194
>>
>> Chain nova-network-POSTROUTING (1 references)
>> target prot opt source destination
>> ACCEPT all -- 192.168.0.0/16 999.999.999.58
>> ACCEPT all -- 192.168.0.0/16 10.128.0.0/24
>> ACCEPT all -- 192.168.0.0/16 192.168.0.0/16 ! ctstate DNAT
>>
>> Chain nova-network-PREROUTING (1 references)
>> target prot opt source destination
>> DNAT tcp -- anywhere 169.254.169.254 tcp dpt:http to:999.999.999.58:8775
>> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.1.2:1194
>> DNAT udp -- anywhere 999.999.999.58 udp dpt:1000 to:192.168.0.2:1194
>>
>> Chain nova-network-float-snat (1 references)
>> target prot opt source destination
>>
>> Chain nova-network-snat (1 references)
>> target prot opt source destination
>> nova-network-float-snat all -- anywhere anywhere
>> SNAT all -- 192.168.0.0/16 anywhere to:10.0.0.103
>>
>> Chain nova-postrouting-bottom (1 references)
>> target prot opt source destination
>> nova-compute-snat all -- anywhere anywhere
>> nova-network-snat all -- anywhere anywhere
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>
Follow ups
References
