openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #21643
Re: VM guest can't access outside world.
Hi Jeff,
Thanks for looking into this but the masquerade still not working. I have more
information and hope you will be able to help.
I have a single bare metal with everything installed ( Nova-compute, network
node, controller, etc... )
There four NIC on that box
NIC em1 connect to external network with IP 10.38.5.251
NIC em3 connect to internal network with no IP configured
em2 and em4 are disabled
After everything is configured ( adding router, net, sub-net ,etc.. ) and
running, I run ifconfig and found out em1's has no more ip but a bridge has
created
brq7f248f20-a6 Link encap:Ethernet HWaddr 00:21:9B:95:99:7A
inet addr:10.38.15.251 Bcast:10.38.255.255 Mask:255.255.0.0
em1 Link encap:Ethernet HWaddr 00:21:9B:95:99:7A
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
I think this is how the quantum/linuxbridge work.
I also create a floatingIP range ( 10.38.17.1-254 ). Then I saw a virtual NIC
is created with IP 10.38.17.1 which I believe is the router IP for the NAT
qg-0503ddc6-1d Link encap:Ethernet HWaddr 8E:57:D6:DA:2B:AA
inet addr:10.38.17.1 Bcast:10.38.17.255 Mask:255.255.255.0
Now I run tcpdump on the openstack box ( ie 10.38.5.251 ) and the target machine
( 10.38.1.2 ). Then ping 10.38.1.2 from my VM ( 192.168.151.4 ). I saw the
packet did arrive to 10.38.1.2 but with ip address 192.168.151.4. I supposed to
see 10.38.17.1 right?
20:52:43.492160 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq
5, length 64
20:52:43.492170 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 5,
length 64
20:52:44.492597 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq
6, length 64
20:52:44.492608 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 6,
length 64
20:52:45.492894 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq
7, length 64
20:52:45.492906 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 7,
length 64
20:52:46.493183 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq
8, length 64
20:52:46.493193 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 8,
length 64
I also think it is the IP masquerade rule, but it didn't work. I tried all
three interface ( em1, brq7f248f20-a6 and qg-0503ddc6-1d ) but none of them
work. For some reason SNAT didn't seem to happen..
Here is the iptables status
service iptables status
Table: nat
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
2 quantum-l3-agent-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
2 quantum-l3-agent-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
3 quantum-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0
4 nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
5 nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0
6 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
7 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
8 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
2 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-api-OUTPUT (1 references)
num target prot opt source destination
Chain nova-api-POSTROUTING (1 references)
num target prot opt source destination
Chain nova-api-PREROUTING (1 references)
num target prot opt source destination
Chain nova-api-float-snat (1 references)
num target prot opt source destination
Chain nova-api-snat (1 references)
num target prot opt source destination
1 nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-compute-OUTPUT (1 references)
num target prot opt source destination
Chain nova-compute-POSTROUTING (1 references)
num target prot opt source destination
Chain nova-compute-PREROUTING (1 references)
num target prot opt source destination
Chain nova-compute-float-snat (1 references)
num target prot opt source destination
Chain nova-compute-snat (1 references)
num target prot opt source destination
1 nova-compute-float-snat all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-postrouting-bottom (1 references)
num target prot opt source destination
1 nova-compute-snat all -- 0.0.0.0/0 0.0.0.0/0
2 nova-api-snat all -- 0.0.0.0/0 0.0.0.0/0
Chain quantum-l3-agent-OUTPUT (1 references)
num target prot opt source destination
Chain quantum-l3-agent-POSTROUTING (1 references)
num target prot opt source destination
1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ! ctstate DNAT
Chain quantum-l3-agent-PREROUTING (1 references)
num target prot opt source destination
Chain quantum-l3-agent-float-snat (1 references)
num target prot opt source destination
Chain quantum-l3-agent-snat (1 references)
num target prot opt source destination
1 quantum-l3-agent-float-snat all -- 0.0.0.0/0 0.0.0.0/0
2 SNAT all -- 192.168.151.0/24 0.0.0.0/0 to:10.38.17.1
Chain quantum-postrouting-bottom (1 references)
num target prot opt source destination
1 quantum-l3-agent-snat all -- 0.0.0.0/0 0.0.0.0/0
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 nova-compute-INPUT all -- 0.0.0.0/0 0.0.0.0/0
2 quantum-l3-agent-INPUT all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-INPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0
2 nova-compute-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0
4 quantum-l3-agent-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
5 nova-api-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0
2 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0
4 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
5 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-api-FORWARD (1 references)
num target prot opt source destination
Chain nova-api-INPUT (1 references)
num target prot opt source destination
1 ACCEPT tcp -- 0.0.0.0/0 10.38.15.251 tcp dpt:8775
Chain nova-api-OUTPUT (1 references)
num target prot opt source destination
Chain nova-api-local (1 references)
num target prot opt source destination
Chain nova-compute-FORWARD (1 references)
num target prot opt source destination
Chain nova-compute-INPUT (1 references)
num target prot opt source destination
Chain nova-compute-OUTPUT (1 references)
num target prot opt source destination
Chain nova-compute-inst-24 (1 references)
num target prot opt source destination
1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0
4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp spt:67
dpt:68
5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0
6 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-compute-inst-25 (1 references)
num target prot opt source destination
1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0
4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp spt:67
dpt:68
5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0
6 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
7 ACCEPT icmp -- 192.168.151.4 0.0.0.0/0
8 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
9 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-compute-local (1 references)
num target prot opt source destination
1 nova-compute-inst-24 all -- 0.0.0.0/0 192.168.151.3
2 nova-compute-inst-25 all -- 0.0.0.0/0 192.168.151.4
Chain nova-compute-provider (2 references)
num target prot opt source destination
Chain nova-compute-sg-fallback (2 references)
num target prot opt source destination
1 DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-filter-top (2 references)
num target prot opt source destination
1 nova-compute-local all -- 0.0.0.0/0 0.0.0.0/0
2 nova-api-local all -- 0.0.0.0/0 0.0.0.0/0
Chain quantum-filter-top (2 references)
num target prot opt source destination
1 quantum-l3-agent-local all -- 0.0.0.0/0 0.0.0.0/0
Chain quantum-l3-agent-FORWARD (1 references)
num target prot opt source destination
Chain quantum-l3-agent-INPUT (1 references)
num target prot opt source destination
Chain quantum-l3-agent-OUTPUT (1 references)
num target prot opt source destination
Chain quantum-l3-agent-local (1 references)
num target prot opt source destination
----- Original Message ----
From: Jeff Peeler <jpeeler@xxxxxxxxxx>
To: Barrow Kwan <barrowkwan@xxxxxxxxx>
Sent: Wed, March 6, 2013 10:48:00 AM
Subject: Re: [Openstack] VM guest can't access outside world.
I'm not replying to the list because I don't know for sure what to tell
you. If this does work for you, feel free to CC the list.
What I mean is by whichever interface uses 10.38.1.2, add a rule to
iptables like:
iptables --table nat -A POSTROUTING -o <above interface> -j MASQUERADE
You're right that this may be a bug in quantum. I haven't really dug too
deeply to confirm that or not yet. If the above works for you, you may
also have to explicitly set the dns nameserver for the subnet in quantum
to something that makes sense.
Jeff
On Mon, Mar 04, 2013 at 11:37:41AM -0800, Barrow Kwan wrote:
> Hi,
> Thanks Jeff. this is what I got from tcpdump. The target (10.38.1.2 ) didn't
> seem to reply. might be the target ( 10.38.1.2 ) didn't know how to route the
>
> packet to 192.168.151.3? could that be SNAT issue? or like you said it needs
>IP
>
> masquerading rule. might be a bug in Quantum?
>
>
> Barrow
>
>
>
>
> tcpdump: WARNING: em1: no IPv4 address assigned
> tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 65535
>bytes
> 11:31:02.825150 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP
>(1),
>
>
> length 84)
> 192.168.151.3 > 10.38.1.2: ICMP echo request, id 11910, seq 133, length 64
> 11:31:03.825338 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP
>(1),
>
>
> length 84)
> 192.168.151.3 > 10.38.1.2: ICMP echo request, id 11910, seq 134, length 64
> 2 packets captured
> 3 packets received by filter
> 0 packets dropped by kernel
>
>
>
>
> ----- Original Message ----
> From: Jeff Peeler <jpeeler@xxxxxxxxxx>
> To: openstack@xxxxxxxxxxxxxxxxxxx
> Sent: Mon, March 4, 2013 7:39:03 AM
> Subject: Re: [Openstack] VM guest can't access outside world.
>
> On Wed, Feb 27, 2013 at 12:38:45PM -0800, Barrow Kwan wrote:
> > [root@optst01 quantum]# service iptables status
> > Table: nat
> > Chain PREROUTING (policy ACCEPT)
> > num target prot opt source destination
> > 1 nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
>
>
>
>
> > 2 quantum-l3-agent-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
>
>
>
>
> >
> > 3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain POSTROUTING (policy ACCEPT)
> > num target prot opt source destination
> > 1 nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
>
> >
> > 2 quantum-l3-agent-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
>
>
> >
> >
> >
> > 3 quantum-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0
>
>
> >
> >
> >
> > 4 nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
> > 5 nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0
>
>
>
>
> >
> > Chain OUTPUT (policy ACCEPT)
> > num target prot opt source destination
> > 1 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
> > 2 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
>
>
>
>
> > 3 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain nova-api-OUTPUT (1 references)
> > num target prot opt source destination
> >
> > Chain nova-api-POSTROUTING (1 references)
> > num target prot opt source destination
> >
> > Chain nova-api-PREROUTING (1 references)
> > num target prot opt source destination
> >
> > Chain nova-api-float-snat (1 references)
> > num target prot opt source destination
> >
> > Chain nova-api-snat (1 references)
> > num target prot opt source destination
> > 1 nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain nova-compute-OUTPUT (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-POSTROUTING (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-PREROUTING (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-float-snat (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-snat (1 references)
> > num target prot opt source destination
> > 1 nova-compute-float-snat all -- 0.0.0.0/0 0.0.0.0/0
>
>
>
>
> >
> > Chain nova-postrouting-bottom (1 references)
> > num target prot opt source destination
> > 1 nova-compute-snat all -- 0.0.0.0/0 0.0.0.0/0
> > 2 nova-api-snat all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain quantum-l3-agent-OUTPUT (1 references)
> > num target prot opt source destination
> >
> > Chain quantum-l3-agent-POSTROUTING (1 references)
> > num target prot opt source destination
> > 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ! ctstate
> > DNAT
> >
> > Chain quantum-l3-agent-PREROUTING (1 references)
> > num target prot opt source destination
> >
> > Chain quantum-l3-agent-float-snat (1 references)
> > num target prot opt source destination
> >
> > Chain quantum-l3-agent-snat (1 references)
> > num target prot opt source destination
> > 1 quantum-l3-agent-float-snat all -- 0.0.0.0/0 0.0.0.0/0
>
>
>
>
> >
> > 2 SNAT all -- 192.168.151.0/24 0.0.0.0/0
>to:10.38.17.1
>
> >
> >
> >
> > Chain quantum-postrouting-bottom (1 references)
> > num target prot opt source destination
> > 1 quantum-l3-agent-snat all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Table: filter
> > Chain INPUT (policy ACCEPT)
> > num target prot opt source destination
> > 1 nova-compute-INPUT all -- 0.0.0.0/0 0.0.0.0/0
> > 2 quantum-l3-agent-INPUT all -- 0.0.0.0/0 0.0.0.0/0
>
> > 3 nova-api-INPUT all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain FORWARD (policy ACCEPT)
> > num target prot opt source destination
> > 1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0
> > 2 nova-compute-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
> > 3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0
> > 4 quantum-l3-agent-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
>
> >
> > 5 nova-api-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain OUTPUT (policy ACCEPT)
> > num target prot opt source destination
> > 1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0
> > 2 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
> > 3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0
> > 4 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
>
>
>
>
> > 5 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain nova-api-FORWARD (1 references)
> > num target prot opt source destination
> >
> > Chain nova-api-INPUT (1 references)
> > num target prot opt source destination
> > 1 ACCEPT tcp -- 0.0.0.0/0 10.38.15.251 tcp
>dpt:8775
>
>
>
> >
> > Chain nova-api-OUTPUT (1 references)
> > num target prot opt source destination
> >
> > Chain nova-api-local (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-FORWARD (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-INPUT (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-OUTPUT (1 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-inst-20 (1 references)
> > num target prot opt source destination
> > 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state
>INVALID
>
> >
> >
> > 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> > RELATED,ESTABLISHED
> > 3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0
> > 4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp spt:67
> > dpt:68
> > 5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0
> > 6 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
> > 7 ACCEPT icmp -- 192.168.151.3 0.0.0.0/0
> > 8 ACCEPT icmp -- 192.168.151.4 0.0.0.0/0
> > 9 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
> > 10 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
>
> >
> >
> > Chain nova-compute-inst-21 (1 references)
> > num target prot opt source destination
> > 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state
>INVALID
>
> >
> >
> > 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> > RELATED,ESTABLISHED
> > 3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0
> > 4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp spt:67
> > dpt:68
> > 5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0
> > 6 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
> > 7 ACCEPT icmp -- 192.168.151.3 0.0.0.0/0
> > 8 ACCEPT icmp -- 192.168.151.4 0.0.0.0/0
> > 9 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
> > 10 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
>
> >
> >
> > Chain nova-compute-local (1 references)
> > num target prot opt source destination
> > 1 nova-compute-inst-20 all -- 0.0.0.0/0 192.168.151.3
> > 2 nova-compute-inst-21 all -- 0.0.0.0/0 192.168.151.4
> >
> > Chain nova-compute-provider (2 references)
> > num target prot opt source destination
> >
> > Chain nova-compute-sg-fallback (2 references)
> > num target prot opt source destination
> > 1 DROP all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain nova-filter-top (2 references)
> > num target prot opt source destination
> > 1 nova-compute-local all -- 0.0.0.0/0 0.0.0.0/0
> > 2 nova-api-local all -- 0.0.0.0/0 0.0.0.0/0
> >
> > Chain quantum-filter-top (2 references)
> > num target prot opt source destination
> > 1 quantum-l3-agent-local all -- 0.0.0.0/0 0.0.0.0/0
>
> >
> > Chain quantum-l3-agent-FORWARD (1 references)
> > num target prot opt source destination
> >
> > Chain quantum-l3-agent-INPUT (1 references)
> > num target prot opt source destination
> >
> > Chain quantum-l3-agent-OUTPUT (1 references)
> > num target prot opt source destination
> >
> > Chain quantum-l3-agent-local (1 references)
> > num target prot opt source destination
>
> Have you tried running tcpdump on the public interface to see how far
> the packets are getting? Maybe something like: tcpdump -n -c2 icmp -i em1,
> then try pinging from the VM. It could be that you're attempting to send
> unroutable packets, in which case an IP masquerading rule needs adding.
>
> Jeff
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
Follow ups
References
