openstack team mailing list archive

Thread
Date
Re: dns issue?

To: openstack@xxxxxxxxxxxxxxxxxxx
From: Sharif Islam <islamsh@xxxxxxxxxxx>
Date: Mon, 10 Oct 2011 16:07:39 -0400
In-reply-to: <715E0492-33BF-44FC-ABF4-E559726D8747@gmail.com>
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110921 Thunderbird/3.1.15
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am still stuck with this problem: my instances can ping only with ip
addresses. I deleted nova-network and created it again and then fired up
an instance. But still resolv.conf shows nameserver 10.0.1.1. I only
have one instance running now, this is not the route looks (a.b.c.d is
my public ip. v.x.y.z  is my nova network host):

# iptables -L -t nat|grep 10.0
DNAT       all  --  anywhere             a.b.c.d to:10.0.1.2
ACCEPT     all  --  10.0.0.0/8           10.128.0.0/24
ACCEPT     all  --  10.0.0.0/8           10.0.0.0/8
DNAT       all  --  anywhere             a.b.c.d to:10.0.1.2
SNAT       all  --  10.0.1.2             anywhere            to:a.b.c.d
SNAT       all  --  10.0.0.0/8           anywhere            to:v.x.y.z


mysql> select * from networks \G
*************************** 1. row ***************************
         created_at: 2011-10-10 18:32:06
         updated_at: 2011-10-10 18:32:54
         deleted_at: NULL
            deleted: 0
                 id: 24
           injected: 0
               cidr: 10.0.1.0/24
            netmask: 255.255.255.0
             bridge: br100
            gateway: 10.0.1.1
          broadcast: 10.0.1.255
                dns: 8.8.4.4
               vlan: NULL
 vpn_public_address: NULL
    vpn_public_port: NULL
vpn_private_address: NULL
         dhcp_start: 10.0.1.2
         project_id: NULL
               host: i26
            cidr_v6: NULL
         gateway_v6: NULL
              label: public
         netmask_v6: NULL
1 row in set (0.00 sec)

root@i-00000499:~# cat /etc/resolv.conf
domain novalocal
search novalocal
nameserver 10.0.1.1

How does nova assign the nameserver to the VMs? Can I change it someone
so it has 8.8.4.4?


- --sharif





On 10/07/2011 05:55 PM, Jorge Luiz Corrêa wrote:
> Some considerations ... I don't know if nova remove rules when an instance is terminated. But it seems that some rules are missing. For example, we can se a lot of address in the NAT table, from .4 to .9 ... But, in the FILTER table, we can see rules just to hosts .7 and .18.
> 
> Chain nova-compute-local (1 references)
> target     prot opt source               destination         
> nova-compute-inst-996  all  --  0.0.0.0/0            10.0.1.7            
> nova-compute-inst-1011  all  --  0.0.0.0/0            10.0.1.1
> 
> The packets are going but not returning. 
> 
> Another thing... the names that VMs query to DNS are wrong. See in tcpdump output:
> 
> 10.0.1.4.52463 > 10.0.1.1.domain: 28545+ A? google.com.novalocal. (38)
> 
> The normal would be google.com. and not google.com.novalocal. This is related to 
> 
> domain novalocal
> search novalocal 
> 
> entries in resolv.conf. I've had the same problem. If I change the resolv.conf it returns to this default after some time. 
> 
> Now, WE need help... lol !
> 
> :)
> 
> 
> On Oct 7, 2011, at 4:43 PM, Sharif Islam wrote:
> 
>>
>> Thanks Jorge.
>>
>> On 10/07/2011 02:30 PM, Jorge Luiz Correa wrote:
>>> It seems that configs are OK. 
>>
>>
>> Yes, that's what baffling me. I am pretty sure it was working before. I
>> applied some redhat update and rebooted the cluster couple weeks ago.
>>
>>
>>
>>>
>>> If you use dig from the controller, could resolv names? I'm asking
>>> because can be case that packets arrive from VMs to controller but
>>> couldn't go to Internet. 
>>
>>> From the controller, it is fine:
>>
>> # dig @10.0.1.1 google.com
>>
>> ; <<>> DiG 9.7.3-P1-RedHat-9.7.3-2.el6_1.P1.1 <<>> @10.0.1.1 google.com
>> ; (1 server found)
>> ;; global options: +cmd
>> ;; Got answer:
>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18002
>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 4, ADDITIONAL: 4
>>
>> ;; QUESTION SECTION:
>> ;google.com.			IN	A
>>
>> ;; ANSWER SECTION:
>> google.com.		263	IN	A	72.14.204.99
>> google.com.		263	IN	A	72.14.204.103
>> google.com.		263	IN	A	72.14.204.104
>> google.com.		263	IN	A	72.14.204.105
>> google.com.		263	IN	A	72.14.204.147
>>
>> ;; AUTHORITY SECTION:
>> google.com.		84809	IN	NS	ns2.google.com.
>> google.com.		84809	IN	NS	ns3.google.com.
>> google.com.		84809	IN	NS	ns4.google.com.
>> google.com.		84809	IN	NS	ns1.google.com.
>>
>> ;; ADDITIONAL SECTION:
>> ns1.google.com.		160584	IN	A	216.239.32.10
>> ns2.google.com.		159501	IN	A	216.239.34.10
>> ns3.google.com.		159500	IN	A	216.239.36.10
>> ns4.google.com.		159497	IN	A	216.239.38.10
>>
>> ;; Query time: 1 msec
>> ;; SERVER: 10.0.1.1#53(10.0.1.1)
>> ;; WHEN: Fri Oct  7 14:44:10 2011
>> ;; MSG SIZE  rcvd: 244
>>
>>
>>
>>
>>
>>
>>>
>>> Another thing you can check. Although the resolv.conf of VMs are set
>>> with 10.0.1.1, there are a lot of iptables rules. I was using Cactus and
>>> I noticed that. If you type nova-manage network list you will see the
>>> networks and you can see a DNS collumn. The default was 8.8.4.4 but when
>>> I started instances this values changed to 10.0.2.1 or something like
>>> that! My concern is about what address nova uses to create rules!! Maybe
>>> all services are OK but a wrong iptables rule is dropping packets!
>>>
>>> iptables -n -L
>>
>> http://paste.openstack.org/show/2646/
>>
>>
>>
>>> iptables -n -L -t nat
>>
>> http://paste.openstack.org/show/2647/
>>
>>>
>>> Check if you have some rule permitting udp 53 to be forward/accepted
>>> (ie, not dropped). 
>>
>> Looks ok to me:
>>
>> ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:53
>> ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:53
>>
>>
>>
>>>
>>> As a debug option, you can run tcpdump on the controller interface and
>>> see what are happening with the packets.
>>>
>>> tcpdump -n -i <interface> port 53
>>
>> #  tcpdump -n -i eth0 port 53
>> tcpdump: WARNING: eth0: no IPv4 address assigned
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 15:42:41.459072 IP 10.0.1.4.46200 > 10.0.1.1.domain: 46894+ A?
>> google.com. (28)
>> 15:42:41.459423 IP 10.0.1.4.49593 > 10.0.1.1.domain: 46894+ A?
>> google.com. (28)
>> 15:42:41.459748 IP 10.0.1.4.32779 > 10.0.1.1.domain: 28545+ A?
>> google.com.novalocal. (38)
>> 15:42:41.460029 IP 10.0.1.4.52463 > 10.0.1.1.domain: 28545+ A?
>> google.com.novalocal. (38)
>>
>>
>> This is when I pinged google.com from the vm. So iptables blocking
>> something?
>>
>> --sharif
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


- -- 
Sharif Islam
Senior Systems Analyst/Programmer
FutureGrid (http://futuregrid.org)
Pervasive Technology Institute, Indiana University Bloomington
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOk1CLAAoJEACffes9SivFkRMH/3JCE5P+MucQHeJKzaxxw/v7
8sD3LLRIBnpczLVDlCF9nXkpnmj4UxI3iMPWNIcsIvaPOKPR0omlRF2sQ0S27P9G
9fzR95KivQPzcrv8dVS6WqDzBgHHKeHIj/9xj0UInoSwaZnF1JGz0uRJNraxRSLg
EqXw7wCxNLFbAKUesT00gpsMzQ9kzJtWKuaIzSs15tUllTeLsBRXbcAdCbMJ3Q7j
wkNLpGBI88A9HdBSRyukrnOAvryihzaE1gpeziQNMZAUOP7eceDOwN7yrLuuvTYK
XhwoRcgCQZ1XYEd+A5ABZIYcCExQe3rLNgXqUBKj+To8kbWopJTcGm1i7jCxMfQ=
=5ZPb
-----END PGP SIGNATURE-----
Follow ups

Re: dns issue?
From: Jason Kölker, 2011-10-10
References

dns issue?
From: Sharif Islam, 2011-10-07
Re: dns issue?
From: Jorge Luiz Correa, 2011-10-07
Re: dns issue?
From: Sharif Islam, 2011-10-07
Re: dns issue?
From: Jorge Luiz Correa, 2011-10-07
Re: dns issue?
From: Sharif Islam, 2011-10-07
Re: dns issue?
From: Jorge Luiz Corrêa, 2011-10-07