openstack team mailing list archive

[Jorge Luiz Corrêa <correajl@xxxxxxxxx>]

Some considerations ... I don't know if nova remove rules when an instance is terminated. But it seems that some rules are missing. For example, we can se a lot of address in the NAT table, from .4 to .9 ... But, in the FILTER table, we can see rules just to hosts .7 and .18.

Chain nova-compute-local (1 references)
target     prot opt source               destination         
nova-compute-inst-996  all  --  0.0.0.0/0            10.0.1.7            
nova-compute-inst-1011  all  --  0.0.0.0/0            10.0.1.1

The packets are going but not returning. 

Another thing... the names that VMs query to DNS are wrong. See in tcpdump output:

10.0.1.4.52463 > 10.0.1.1.domain: 28545+ A? google.com.novalocal. (38)

The normal would be google.com. and not google.com.novalocal. This is related to 

domain novalocal
search novalocal 

entries in resolv.conf. I've had the same problem. If I change the resolv.conf it returns to this default after some time. 

Now, WE need help... lol !

:)


On Oct 7, 2011, at 4:43 PM, Sharif Islam wrote:

> 
> Thanks Jorge.
> 
> On 10/07/2011 02:30 PM, Jorge Luiz Correa wrote:
>> It seems that configs are OK. 
> 
> 
> Yes, that's what baffling me. I am pretty sure it was working before. I
> applied some redhat update and rebooted the cluster couple weeks ago.
> 
> 
> 
>> 
>> If you use dig from the controller, could resolv names? I'm asking
>> because can be case that packets arrive from VMs to controller but
>> couldn't go to Internet. 
> 
>> From the controller, it is fine:
> 
> # dig @10.0.1.1 google.com
> 
> ; <<>> DiG 9.7.3-P1-RedHat-9.7.3-2.el6_1.P1.1 <<>> @10.0.1.1 google.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18002
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 4, ADDITIONAL: 4
> 
> ;; QUESTION SECTION:
> ;google.com.			IN	A
> 
> ;; ANSWER SECTION:
> google.com.		263	IN	A	72.14.204.99
> google.com.		263	IN	A	72.14.204.103
> google.com.		263	IN	A	72.14.204.104
> google.com.		263	IN	A	72.14.204.105
> google.com.		263	IN	A	72.14.204.147
> 
> ;; AUTHORITY SECTION:
> google.com.		84809	IN	NS	ns2.google.com.
> google.com.		84809	IN	NS	ns3.google.com.
> google.com.		84809	IN	NS	ns4.google.com.
> google.com.		84809	IN	NS	ns1.google.com.
> 
> ;; ADDITIONAL SECTION:
> ns1.google.com.		160584	IN	A	216.239.32.10
> ns2.google.com.		159501	IN	A	216.239.34.10
> ns3.google.com.		159500	IN	A	216.239.36.10
> ns4.google.com.		159497	IN	A	216.239.38.10
> 
> ;; Query time: 1 msec
> ;; SERVER: 10.0.1.1#53(10.0.1.1)
> ;; WHEN: Fri Oct  7 14:44:10 2011
> ;; MSG SIZE  rcvd: 244
> 
> 
> 
> 
> 
> 
>> 
>> Another thing you can check. Although the resolv.conf of VMs are set
>> with 10.0.1.1, there are a lot of iptables rules. I was using Cactus and
>> I noticed that. If you type nova-manage network list you will see the
>> networks and you can see a DNS collumn. The default was 8.8.4.4 but when
>> I started instances this values changed to 10.0.2.1 or something like
>> that! My concern is about what address nova uses to create rules!! Maybe
>> all services are OK but a wrong iptables rule is dropping packets!
>> 
>> iptables -n -L
> 
> http://paste.openstack.org/show/2646/
> 
> 
> 
>> iptables -n -L -t nat
> 
> http://paste.openstack.org/show/2647/
> 
>> 
>> Check if you have some rule permitting udp 53 to be forward/accepted
>> (ie, not dropped). 
> 
> Looks ok to me:
> 
> ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:53
> ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:53
> 
> 
> 
>> 
>> As a debug option, you can run tcpdump on the controller interface and
>> see what are happening with the packets.
>> 
>> tcpdump -n -i <interface> port 53
> 
> #  tcpdump -n -i eth0 port 53
> tcpdump: WARNING: eth0: no IPv4 address assigned
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 15:42:41.459072 IP 10.0.1.4.46200 > 10.0.1.1.domain: 46894+ A?
> google.com. (28)
> 15:42:41.459423 IP 10.0.1.4.49593 > 10.0.1.1.domain: 46894+ A?
> google.com. (28)
> 15:42:41.459748 IP 10.0.1.4.32779 > 10.0.1.1.domain: 28545+ A?
> google.com.novalocal. (38)
> 15:42:41.460029 IP 10.0.1.4.52463 > 10.0.1.1.domain: 28545+ A?
> google.com.novalocal. (38)
> 
> 
> This is when I pinged google.com from the vm. So iptables blocking
> something?
> 
> --sharif
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp