openstack team mailing list archive

Thread
Date
Networking issues in Essex

To: openstack maillist <openstack@xxxxxxxxxxxxxxxxxxx>
From: Michael Chapman <michael.chapman@xxxxxxxxxx>
Date: Thu, 12 Jul 2012 14:56:57 +1000
Sender: woppin@xxxxxxxxx
Hi all, I'm hoping I could get some assistance figuring out my networking
problems with a small Essex test cluster. I have a small Diablo cluster
running without any problems but have hit a wall in deploying Essex.

I can launch VMs without issue and access them from the compute host, but
from there I can't access anything except the host, DNS services, and other
VMs.

I have separate machines running keystone, glance, postgresql, rabbit-mq
and nova-api. They're all on the .os domain with 172.22.1.X IPs

I have one machine running nova-compute, nova-network and nova-api, with a
public address 192.43.239.175 and also an IP on the 172.22.1.X subnet in
the .os domain. It has the following nova/conf:

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--force_dhcp_release
--iscsi_helper=tgtadm
--libvirt_use_virtio_for_bridges
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose
--ec2_private_dns_show_ip

--network_manager=nova.network.manager.FlatDHCPManager
--rabbit_host=os-amqp.os
--sql_connection=postgresql://[user]:[password]@os-sql.os/nova
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=os-glance.os:9292
--auth_strategy=keystone
--scheduler_driver=nova.scheduler.simple.SimpleScheduler
--keystone_ec2_url=http://os-key.os:5000/v2.0/ec2tokens

--api_paste_config=/etc/nova/api-paste.ini

--my_ip=192.43.239.175
--flat_interface=eth0
--public_interface=eth1
--multi_host=True
--routing_source_ip=192.43.239.175
--network_host=192.43.239.175

--dmz_cidr=$my_ip

--ec2_host=192.43.239.175
--ec2_dmz_host=192.43.239.175

I believe I'm seeing a natting issue of some sort - my VMs cannot ping
external IPs, though DNS seems to work.
ubuntu@monday:~$ ping www.google.com
PING www.l.google.com (74.125.237.148) 56(84) bytes of data.
<AWKWARD SILENCE>

When I do a tcpdump on the compute host things seem fairly normal, even
though nothing is getting back to the VM

root@ncios1:~# tcpdump icmp -i br100
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes
14:35:28.046416 IP 10.0.0.8 > syd01s13-in-f20.1e100.net: ICMP echo request,
id 5002, seq 9, length 64
14:35:28.051477 IP syd01s13-in-f20.1e100.net > 10.0.0.8: ICMP echo reply,
id 5002, seq 9, length 64
14:35:29.054505 IP 10.0.0.8 > syd01s13-in-f20.1e100.net: ICMP echo request,
id 5002, seq 10, length 64
14:35:29.059556 IP syd01s13-in-f20.1e100.net > 10.0.0.8: ICMP echo reply,
id 5002, seq 10, length 64

I've pored over the iptables nat rules and can't see anything amiss apart
from the masquerades that are automatically added: (I've cut out some empty
chains for brevity)

root@ncios1:~# iptables -L -t nat -v
Chain PREROUTING (policy ACCEPT 22 packets, 2153 bytes)
 pkts bytes target     prot opt in     out     source
destination
   22  2153 nova-network-PREROUTING  all  --  any    any     anywhere
      anywhere
   22  2153 nova-compute-PREROUTING  all  --  any    any     anywhere
      anywhere
   22  2153 nova-api-PREROUTING  all  --  any    any     anywhere
  anywhere

Chain INPUT (policy ACCEPT 12 packets, 1573 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 31 packets, 2021 bytes)
 pkts bytes target     prot opt in     out     source
destination
   31  2021 nova-network-OUTPUT  all  --  any    any     anywhere
  anywhere
   31  2021 nova-compute-OUTPUT  all  --  any    any     anywhere
  anywhere
   31  2021 nova-api-OUTPUT  all  --  any    any     anywhere
anywhere

Chain POSTROUTING (policy ACCEPT 30 packets, 1961 bytes)
 pkts bytes target     prot opt in     out     source
destination
   31  2021 nova-network-POSTROUTING  all  --  any    any     anywhere
        anywhere
   30  1961 nova-compute-POSTROUTING  all  --  any    any     anywhere
        anywhere
   30  1961 nova-api-POSTROUTING  all  --  any    any     anywhere
    anywhere
   30  1961 nova-postrouting-bottom  all  --  any    any     anywhere
      anywhere
    0     0 MASQUERADE  tcp  --  any    any     192.168.122.0/24    !
192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  udp  --  any    any     192.168.122.0/24    !
192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  all  --  any    any     192.168.122.0/24    !
192.168.122.0/24

Chain nova-api-snat (1 references)
 pkts bytes target     prot opt in     out     source
destination
   30  1961 nova-api-float-snat  all  --  any    any     anywhere
  anywhere

Chain nova-compute-snat (1 references)
 pkts bytes target     prot opt in     out     source
destination
   30  1961 nova-compute-float-snat  all  --  any    any     anywhere
      anywhere

Chain nova-network-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source
destination
    0     0 ACCEPT     all  --  any    any     10.0.0.0/8
nri5.nci.org.au
    0     0 ACCEPT     all  --  any    any     10.0.0.0/8
nri5.nci.org.au
    1    60 ACCEPT     all  --  any    any     10.0.0.0/8
10.0.0.0/8           ! ctstate DNAT

Chain nova-network-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source
destination
    0     0 DNAT       tcp  --  any    any     anywhere
169.254.169.254      tcp dpt:http to:192.43.239.175:8775

Chain nova-network-snat (1 references)
 pkts bytes target     prot opt in     out     source
destination
   30  1961 nova-network-float-snat  all  --  any    any     anywhere
      anywhere
    0     0 SNAT       all  --  any    any     10.0.0.0/8
anywhere             to:192.43.239.175

Chain nova-postrouting-bottom (1 references)
 pkts bytes target     prot opt in     out     source
destination
   30  1961 nova-network-snat  all  --  any    any     anywhere
anywhere
   30  1961 nova-compute-snat  all  --  any    any     anywhere
anywhere
   30  1961 nova-api-snat  all  --  any    any     anywhere
anywhere

and the ACCEPT icmp rule seems to be there in filter for the security group
as well, though it's not being triggered for some reason:

Chain nova-compute-inst-6 (1 references)
 pkts bytes target     prot opt in     out     source
destination
    0     0 DROP       all  --  any    any     anywhere
anywhere             state INVALID
   39  6545 ACCEPT     all  --  any    any     anywhere
anywhere             state RELATED,ESTABLISHED
    1    60 nova-compute-provider  all  --  any    any     anywhere
    anywhere
    0     0 ACCEPT     udp  --  any    any     10.0.0.3
anywhere             udp spt:bootps dpt:bootpc
    1    60 ACCEPT     all  --  any    any     10.0.0.0/24
 anywhere
    0     0 ACCEPT     icmp --  any    any     anywhere
anywhere
    0     0 ACCEPT     tcp  --  any    any     anywhere
anywhere             tcp dpt:ssh
    0     0 nova-compute-sg-fallback  all  --  any    any     anywhere
        anywhere

I've tried changing the routing source IP between using the private
172.22.1.X IP and the public one but it doesn't seem to change anything. I
tried without that config option at all and also without the network host
flag and not much seems to change.

Any help would be much appreciated.



-- 
Michael Chapman
*Cloud Computing Services*
ANU Supercomputer Facility
Room 318, Leonard Huxley Building (#56), Mills Road
The Australian National University
Canberra ACT 0200 Australia
Tel: *+61 2 6125 7106*
Web: http://nci.org.au
Follow ups

Re: Networking issues in Essex
From: Jonathan Proulx, 2012-07-12