← Back to team overview

openstack team mailing list archive

Re: Networking issues in Essex

 

I've only deployed openstack for the first time a couple weeks ago,
but FWIW...

I had similar symptoms on my Essex test deployment (on Ubuntu 12.04)
turned out my problem was taht while the br100 bridge was up and
configured the underlying eth1 physical interface was down so the bits
went nowhere.  'ifconfig eth1 up' fixed all, followed ofcoures by
fixing in /etc/network/interfaces as well so this happens on it's own
in future.

-Jon

On Thu, Jul 12, 2012 at 02:56:57PM +1000, Michael Chapman wrote:
:Hi all, I'm hoping I could get some assistance figuring out my networking
:problems with a small Essex test cluster. I have a small Diablo cluster
:running without any problems but have hit a wall in deploying Essex.
:
:I can launch VMs without issue and access them from the compute host, but
:from there I can't access anything except the host, DNS services, and other
:VMs.
:
:I have separate machines running keystone, glance, postgresql, rabbit-mq
:and nova-api. They're all on the .os domain with 172.22.1.X IPs
:
:I have one machine running nova-compute, nova-network and nova-api, with a
:public address 192.43.239.175 and also an IP on the 172.22.1.X subnet in
:the .os domain. It has the following nova/conf:
:
:--dhcpbridge_flagfile=/etc/nova/nova.conf
:--dhcpbridge=/usr/bin/nova-dhcpbridge
:--logdir=/var/log/nova
:--state_path=/var/lib/nova
:--lock_path=/var/lock/nova
:--force_dhcp_release
:--iscsi_helper=tgtadm
:--libvirt_use_virtio_for_bridges
:--connection_type=libvirt
:--root_helper=sudo nova-rootwrap
:--verbose
:--ec2_private_dns_show_ip
:
:--network_manager=nova.network.manager.FlatDHCPManager
:--rabbit_host=os-amqp.os
:--sql_connection=postgresql://[user]:[password]@os-sql.os/nova
:--image_service=nova.image.glance.GlanceImageService
:--glance_api_servers=os-glance.os:9292
:--auth_strategy=keystone
:--scheduler_driver=nova.scheduler.simple.SimpleScheduler
:--keystone_ec2_url=http://os-key.os:5000/v2.0/ec2tokens
:
:--api_paste_config=/etc/nova/api-paste.ini
:
:--my_ip=192.43.239.175
:--flat_interface=eth0
:--public_interface=eth1
:--multi_host=True
:--routing_source_ip=192.43.239.175
:--network_host=192.43.239.175
:
:--dmz_cidr=$my_ip
:
:--ec2_host=192.43.239.175
:--ec2_dmz_host=192.43.239.175
:
:I believe I'm seeing a natting issue of some sort - my VMs cannot ping
:external IPs, though DNS seems to work.
:ubuntu@monday:~$ ping www.google.com
:PING www.l.google.com (74.125.237.148) 56(84) bytes of data.
:<AWKWARD SILENCE>
:
:When I do a tcpdump on the compute host things seem fairly normal, even
:though nothing is getting back to the VM
:
:root@ncios1:~# tcpdump icmp -i br100
:tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
:listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes
:14:35:28.046416 IP 10.0.0.8 > syd01s13-in-f20.1e100.net: ICMP echo request,
:id 5002, seq 9, length 64
:14:35:28.051477 IP syd01s13-in-f20.1e100.net > 10.0.0.8: ICMP echo reply,
:id 5002, seq 9, length 64
:14:35:29.054505 IP 10.0.0.8 > syd01s13-in-f20.1e100.net: ICMP echo request,
:id 5002, seq 10, length 64
:14:35:29.059556 IP syd01s13-in-f20.1e100.net > 10.0.0.8: ICMP echo reply,
:id 5002, seq 10, length 64
:
:I've pored over the iptables nat rules and can't see anything amiss apart
:from the masquerades that are automatically added: (I've cut out some empty
:chains for brevity)
:
:root@ncios1:~# iptables -L -t nat -v
:Chain PREROUTING (policy ACCEPT 22 packets, 2153 bytes)
: pkts bytes target     prot opt in     out     source
:destination
:   22  2153 nova-network-PREROUTING  all  --  any    any     anywhere
:      anywhere
:   22  2153 nova-compute-PREROUTING  all  --  any    any     anywhere
:      anywhere
:   22  2153 nova-api-PREROUTING  all  --  any    any     anywhere
:  anywhere
:
:Chain INPUT (policy ACCEPT 12 packets, 1573 bytes)
: pkts bytes target     prot opt in     out     source
:destination
:
:Chain OUTPUT (policy ACCEPT 31 packets, 2021 bytes)
: pkts bytes target     prot opt in     out     source
:destination
:   31  2021 nova-network-OUTPUT  all  --  any    any     anywhere
:  anywhere
:   31  2021 nova-compute-OUTPUT  all  --  any    any     anywhere
:  anywhere
:   31  2021 nova-api-OUTPUT  all  --  any    any     anywhere
:anywhere
:
:Chain POSTROUTING (policy ACCEPT 30 packets, 1961 bytes)
: pkts bytes target     prot opt in     out     source
:destination
:   31  2021 nova-network-POSTROUTING  all  --  any    any     anywhere
:        anywhere
:   30  1961 nova-compute-POSTROUTING  all  --  any    any     anywhere
:        anywhere
:   30  1961 nova-api-POSTROUTING  all  --  any    any     anywhere
:    anywhere
:   30  1961 nova-postrouting-bottom  all  --  any    any     anywhere
:      anywhere
:    0     0 MASQUERADE  tcp  --  any    any     192.168.122.0/24    !
:192.168.122.0/24     masq ports: 1024-65535
:    0     0 MASQUERADE  udp  --  any    any     192.168.122.0/24    !
:192.168.122.0/24     masq ports: 1024-65535
:    0     0 MASQUERADE  all  --  any    any     192.168.122.0/24    !
:192.168.122.0/24
:
:Chain nova-api-snat (1 references)
: pkts bytes target     prot opt in     out     source
:destination
:   30  1961 nova-api-float-snat  all  --  any    any     anywhere
:  anywhere
:
:Chain nova-compute-snat (1 references)
: pkts bytes target     prot opt in     out     source
:destination
:   30  1961 nova-compute-float-snat  all  --  any    any     anywhere
:      anywhere
:
:Chain nova-network-POSTROUTING (1 references)
: pkts bytes target     prot opt in     out     source
:destination
:    0     0 ACCEPT     all  --  any    any     10.0.0.0/8
:nri5.nci.org.au
:    0     0 ACCEPT     all  --  any    any     10.0.0.0/8
:nri5.nci.org.au
:    1    60 ACCEPT     all  --  any    any     10.0.0.0/8
:10.0.0.0/8           ! ctstate DNAT
:
:Chain nova-network-PREROUTING (1 references)
: pkts bytes target     prot opt in     out     source
:destination
:    0     0 DNAT       tcp  --  any    any     anywhere
:169.254.169.254      tcp dpt:http to:192.43.239.175:8775
:
:Chain nova-network-snat (1 references)
: pkts bytes target     prot opt in     out     source
:destination
:   30  1961 nova-network-float-snat  all  --  any    any     anywhere
:      anywhere
:    0     0 SNAT       all  --  any    any     10.0.0.0/8
:anywhere             to:192.43.239.175
:
:Chain nova-postrouting-bottom (1 references)
: pkts bytes target     prot opt in     out     source
:destination
:   30  1961 nova-network-snat  all  --  any    any     anywhere
:anywhere
:   30  1961 nova-compute-snat  all  --  any    any     anywhere
:anywhere
:   30  1961 nova-api-snat  all  --  any    any     anywhere
:anywhere
:
:and the ACCEPT icmp rule seems to be there in filter for the security group
:as well, though it's not being triggered for some reason:
:
:Chain nova-compute-inst-6 (1 references)
: pkts bytes target     prot opt in     out     source
:destination
:    0     0 DROP       all  --  any    any     anywhere
:anywhere             state INVALID
:   39  6545 ACCEPT     all  --  any    any     anywhere
:anywhere             state RELATED,ESTABLISHED
:    1    60 nova-compute-provider  all  --  any    any     anywhere
:    anywhere
:    0     0 ACCEPT     udp  --  any    any     10.0.0.3
:anywhere             udp spt:bootps dpt:bootpc
:    1    60 ACCEPT     all  --  any    any     10.0.0.0/24
: anywhere
:    0     0 ACCEPT     icmp --  any    any     anywhere
:anywhere
:    0     0 ACCEPT     tcp  --  any    any     anywhere
:anywhere             tcp dpt:ssh
:    0     0 nova-compute-sg-fallback  all  --  any    any     anywhere
:        anywhere
:
:I've tried changing the routing source IP between using the private
:172.22.1.X IP and the public one but it doesn't seem to change anything. I
:tried without that config option at all and also without the network host
:flag and not much seems to change.
:
:Any help would be much appreciated.
:
:
:
:-- 
:Michael Chapman
:*Cloud Computing Services*
:ANU Supercomputer Facility
:Room 318, Leonard Huxley Building (#56), Mills Road
:The Australian National University
:Canberra ACT 0200 Australia
:Tel: *+61 2 6125 7106*
:Web: http://nci.org.au

:_______________________________________________
:Mailing list: https://launchpad.net/~openstack
:Post to     : openstack@xxxxxxxxxxxxxxxxxxx
:Unsubscribe : https://launchpad.net/~openstack
:More help   : https://help.launchpad.net/ListHelp



Follow ups

References