openstack team mailing list archive

Thread
Date
Re: Networking issues in Essex

To: Jonathan Proulx <jon@xxxxxxxxxxxxx>
From: Michael Chapman <michael.chapman@xxxxxxxxxx>
Date: Fri, 13 Jul 2012 15:41:45 +1000
Cc: openstack maillist <openstack@xxxxxxxxxxxxxxxxxxx>
In-reply-to: <20120712121510.GB7393@csail.mit.edu>
Sender: woppin@xxxxxxxxx
Thanks for the tip, unfortunately the interfaces are already up.

 - Michael

On Thu, Jul 12, 2012 at 10:15 PM, Jonathan Proulx <jon@xxxxxxxxxxxxx> wrote:

>
> I've only deployed openstack for the first time a couple weeks ago,
> but FWIW...
>
> I had similar symptoms on my Essex test deployment (on Ubuntu 12.04)
> turned out my problem was taht while the br100 bridge was up and
> configured the underlying eth1 physical interface was down so the bits
> went nowhere.  'ifconfig eth1 up' fixed all, followed ofcoures by
> fixing in /etc/network/interfaces as well so this happens on it's own
> in future.
>
> -Jon
>
> On Thu, Jul 12, 2012 at 02:56:57PM +1000, Michael Chapman wrote:
> :Hi all, I'm hoping I could get some assistance figuring out my networking
> :problems with a small Essex test cluster. I have a small Diablo cluster
> :running without any problems but have hit a wall in deploying Essex.
> :
> :I can launch VMs without issue and access them from the compute host, but
> :from there I can't access anything except the host, DNS services, and
> other
> :VMs.
> :
> :I have separate machines running keystone, glance, postgresql, rabbit-mq
> :and nova-api. They're all on the .os domain with 172.22.1.X IPs
> :
> :I have one machine running nova-compute, nova-network and nova-api, with a
> :public address 192.43.239.175 and also an IP on the 172.22.1.X subnet in
> :the .os domain. It has the following nova/conf:
> :
> :--dhcpbridge_flagfile=/etc/nova/nova.conf
> :--dhcpbridge=/usr/bin/nova-dhcpbridge
> :--logdir=/var/log/nova
> :--state_path=/var/lib/nova
> :--lock_path=/var/lock/nova
> :--force_dhcp_release
> :--iscsi_helper=tgtadm
> :--libvirt_use_virtio_for_bridges
> :--connection_type=libvirt
> :--root_helper=sudo nova-rootwrap
> :--verbose
> :--ec2_private_dns_show_ip
> :
> :--network_manager=nova.network.manager.FlatDHCPManager
> :--rabbit_host=os-amqp.os
> :--sql_connection=postgresql://[user]:[password]@os-sql.os/nova
> :--image_service=nova.image.glance.GlanceImageService
> :--glance_api_servers=os-glance.os:9292
> :--auth_strategy=keystone
> :--scheduler_driver=nova.scheduler.simple.SimpleScheduler
> :--keystone_ec2_url=http://os-key.os:5000/v2.0/ec2tokens
> :
> :--api_paste_config=/etc/nova/api-paste.ini
> :
> :--my_ip=192.43.239.175
> :--flat_interface=eth0
> :--public_interface=eth1
> :--multi_host=True
> :--routing_source_ip=192.43.239.175
> :--network_host=192.43.239.175
> :
> :--dmz_cidr=$my_ip
> :
> :--ec2_host=192.43.239.175
> :--ec2_dmz_host=192.43.239.175
> :
> :I believe I'm seeing a natting issue of some sort - my VMs cannot ping
> :external IPs, though DNS seems to work.
> :ubuntu@monday:~$ ping www.google.com
> :PING www.l.google.com (74.125.237.148) 56(84) bytes of data.
> :<AWKWARD SILENCE>
> :
> :When I do a tcpdump on the compute host things seem fairly normal, even
> :though nothing is getting back to the VM
> :
> :root@ncios1:~# tcpdump icmp -i br100
> :tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> :listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes
> :14:35:28.046416 IP 10.0.0.8 > syd01s13-in-f20.1e100.net: ICMP echo
> request,
> :id 5002, seq 9, length 64
> :14:35:28.051477 IP syd01s13-in-f20.1e100.net > 10.0.0.8: ICMP echo reply,
> :id 5002, seq 9, length 64
> :14:35:29.054505 IP 10.0.0.8 > syd01s13-in-f20.1e100.net: ICMP echo
> request,
> :id 5002, seq 10, length 64
> :14:35:29.059556 IP syd01s13-in-f20.1e100.net > 10.0.0.8: ICMP echo reply,
> :id 5002, seq 10, length 64
> :
> :I've pored over the iptables nat rules and can't see anything amiss apart
> :from the masquerades that are automatically added: (I've cut out some
> empty
> :chains for brevity)
> :
> :root@ncios1:~# iptables -L -t nat -v
> :Chain PREROUTING (policy ACCEPT 22 packets, 2153 bytes)
> : pkts bytes target     prot opt in     out     source
> :destination
> :   22  2153 nova-network-PREROUTING  all  --  any    any     anywhere
> :      anywhere
> :   22  2153 nova-compute-PREROUTING  all  --  any    any     anywhere
> :      anywhere
> :   22  2153 nova-api-PREROUTING  all  --  any    any     anywhere
> :  anywhere
> :
> :Chain INPUT (policy ACCEPT 12 packets, 1573 bytes)
> : pkts bytes target     prot opt in     out     source
> :destination
> :
> :Chain OUTPUT (policy ACCEPT 31 packets, 2021 bytes)
> : pkts bytes target     prot opt in     out     source
> :destination
> :   31  2021 nova-network-OUTPUT  all  --  any    any     anywhere
> :  anywhere
> :   31  2021 nova-compute-OUTPUT  all  --  any    any     anywhere
> :  anywhere
> :   31  2021 nova-api-OUTPUT  all  --  any    any     anywhere
> :anywhere
> :
> :Chain POSTROUTING (policy ACCEPT 30 packets, 1961 bytes)
> : pkts bytes target     prot opt in     out     source
> :destination
> :   31  2021 nova-network-POSTROUTING  all  --  any    any     anywhere
> :        anywhere
> :   30  1961 nova-compute-POSTROUTING  all  --  any    any     anywhere
> :        anywhere
> :   30  1961 nova-api-POSTROUTING  all  --  any    any     anywhere
> :    anywhere
> :   30  1961 nova-postrouting-bottom  all  --  any    any     anywhere
> :      anywhere
> :    0     0 MASQUERADE  tcp  --  any    any     192.168.122.0/24    !
> :192.168.122.0/24     masq ports: 1024-65535
> :    0     0 MASQUERADE  udp  --  any    any     192.168.122.0/24    !
> :192.168.122.0/24     masq ports: 1024-65535
> :    0     0 MASQUERADE  all  --  any    any     192.168.122.0/24    !
> :192.168.122.0/24
> :
> :Chain nova-api-snat (1 references)
> : pkts bytes target     prot opt in     out     source
> :destination
> :   30  1961 nova-api-float-snat  all  --  any    any     anywhere
> :  anywhere
> :
> :Chain nova-compute-snat (1 references)
> : pkts bytes target     prot opt in     out     source
> :destination
> :   30  1961 nova-compute-float-snat  all  --  any    any     anywhere
> :      anywhere
> :
> :Chain nova-network-POSTROUTING (1 references)
> : pkts bytes target     prot opt in     out     source
> :destination
> :    0     0 ACCEPT     all  --  any    any     10.0.0.0/8
> :nri5.nci.org.au
> :    0     0 ACCEPT     all  --  any    any     10.0.0.0/8
> :nri5.nci.org.au
> :    1    60 ACCEPT     all  --  any    any     10.0.0.0/8
> :10.0.0.0/8           ! ctstate DNAT
> :
> :Chain nova-network-PREROUTING (1 references)
> : pkts bytes target     prot opt in     out     source
> :destination
> :    0     0 DNAT       tcp  --  any    any     anywhere
> :169.254.169.254      tcp dpt:http to:192.43.239.175:8775
> :
> :Chain nova-network-snat (1 references)
> : pkts bytes target     prot opt in     out     source
> :destination
> :   30  1961 nova-network-float-snat  all  --  any    any     anywhere
> :      anywhere
> :    0     0 SNAT       all  --  any    any     10.0.0.0/8
> :anywhere             to:192.43.239.175
> :
> :Chain nova-postrouting-bottom (1 references)
> : pkts bytes target     prot opt in     out     source
> :destination
> :   30  1961 nova-network-snat  all  --  any    any     anywhere
> :anywhere
> :   30  1961 nova-compute-snat  all  --  any    any     anywhere
> :anywhere
> :   30  1961 nova-api-snat  all  --  any    any     anywhere
> :anywhere
> :
> :and the ACCEPT icmp rule seems to be there in filter for the security
> group
> :as well, though it's not being triggered for some reason:
> :
> :Chain nova-compute-inst-6 (1 references)
> : pkts bytes target     prot opt in     out     source
> :destination
> :    0     0 DROP       all  --  any    any     anywhere
> :anywhere             state INVALID
> :   39  6545 ACCEPT     all  --  any    any     anywhere
> :anywhere             state RELATED,ESTABLISHED
> :    1    60 nova-compute-provider  all  --  any    any     anywhere
> :    anywhere
> :    0     0 ACCEPT     udp  --  any    any     10.0.0.3
> :anywhere             udp spt:bootps dpt:bootpc
> :    1    60 ACCEPT     all  --  any    any     10.0.0.0/24
> : anywhere
> :    0     0 ACCEPT     icmp --  any    any     anywhere
> :anywhere
> :    0     0 ACCEPT     tcp  --  any    any     anywhere
> :anywhere             tcp dpt:ssh
> :    0     0 nova-compute-sg-fallback  all  --  any    any     anywhere
> :        anywhere
> :
> :I've tried changing the routing source IP between using the private
> :172.22.1.X IP and the public one but it doesn't seem to change anything. I
> :tried without that config option at all and also without the network host
> :flag and not much seems to change.
> :
> :Any help would be much appreciated.
> :
> :
> :
> :--
> :Michael Chapman
> :*Cloud Computing Services*
> :ANU Supercomputer Facility
> :Room 318, Leonard Huxley Building (#56), Mills Road
> :The Australian National University
> :Canberra ACT 0200 Australia
> :Tel: *+61 2 6125 7106*
> :Web: http://nci.org.au
>
> :_______________________________________________
> :Mailing list: https://launchpad.net/~openstack
> :Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> :Unsubscribe : https://launchpad.net/~openstack
> :More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Michael Chapman
*Cloud Computing Services*
ANU Supercomputer Facility
Room 318, Leonard Huxley Building (#56), Mills Road
The Australian National University
Canberra ACT 0200 Australia
Tel: *+61 2 6125 7106*
Web: http://nci.org.au
References

Networking issues in Essex
From: Michael Chapman, 2012-07-12
Re: Networking issues in Essex
From: Jonathan Proulx, 2012-07-12