openstack team mailing list archive

[Thomas Kärgel <kaergel@xxxxxxxxxxxxx>]

Hi Marco,

can you also reproduce it?

If so i would suggest to give dnsmasq 2.66 a try. Since i'm not able to
reproduce this behavior on the environment i'm woking on it will take
weeks to get to an really conclusive result.

Best regards
thomas


Am 22.05.2013 09:55, schrieb Marco Colombo:
> Hi All,
> i have the same problem with Grizzly, Ubuntu 12.04 and Dnsmasq 2.59
> "killall dnsmasq && service quantum-dhcp-agent restart" fixes the
> problem temporarily
> 
> Best Regards,
> Marco
> 
> 
> 
> 
> 2013/5/22 Thomas Kärgel <kaergel@xxxxxxxxxxxxx
> <mailto:kaergel@xxxxxxxxxxxxx>>
> 
>     Hi Johanna,
> 
>     sure, I'll report my results, but this can take a while. The last time
>     this issue occurred on my environment is May 10th. It occurs quite
>     sporadic and i don't know how to manually reproduce this issue yet (i
>     have to be very careful with my experiments since the environment is
>     running productive already). Maybe you should also give dnsmasq 2.66 a
>     try, because you wrote that you can manually reproduce this behavior.
> 
> 
>     Best Regards,
>     Thomas
> 
> 
>     Am 20.05.2013 12:25, schrieb Heinonen, Johanna (NSN - FI/Espoo):
>     > Hi,
>     >
>     > You are right. Manually doing kill -HUP <PID> has just the effect
>     you mentioned. In the syslog you can see
>     >
>     > May 20 13:09:41 grizzly-236 dnsmasq[6170]: cleared cache
>     > May 20 13:09:41 grizzly-236 dnsmasq-dhcp[6170]: read
>     /var/lib/quantum/dhcp/d5879bbb-ada6-4323-a7b1-87b5db244513/host
>     > May 20 13:09:41 grizzly-236 dnsmasq-dhcp[6170]: read
>     /var/lib/quantum/dhcp/d5879bbb-ada6-4323-a7b1-87b5db244513/opts
>     >
>     >
>     > But the problem stays. Only 'service quantum-dhcp-agent restart'
>     fixes the problem.
>     > If you try the newer version of the qnsmasq, I'd be interested to
>     hear the results.
>     >
>     > BR
>     > Johanna
>     >
>     >
>     >
>     > -----Original Message-----
>     > From: ext Thomas Kärgel [mailto:kaergel@xxxxxxxxxxxxx
>     <mailto:kaergel@xxxxxxxxxxxxx>]
>     > Sent: Monday, May 20, 2013 11:37 AM
>     > To: Heinonen, Johanna (NSN - FI/Espoo)
>     > Cc: openstack@xxxxxxxxxxxxxxxxxxx
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     > Subject: Re: [Openstack] DHCP problem in grizzly
>     >
>     > Hi,
>     >
>     > thx for the interesting info, Johanna: dnsmasq version is also 2.59 on
>     > my environment. Can you can confirm that manually sigHUPing all
>     running
>     > dnsmasq processes has no effect?
>     > (dnsmasq claims to have reread the configs in syslog, but still
>     refuses
>     > to deliver the new addresses.)
>     > Maybe updating dnsmasq to a more recent release would solve our
>     problem.
>     > Current stable is 2.66. I'll give it a try when i get back to office
>     > tomorrow.
>     >
>     > Best regards
>     >
>     > Thomas
>     >
>     >
>     > Am 20.05.2013 10:21, schrieb Heinonen, Johanna (NSN - FI/Espoo):
>     >> Hi Thomas,
>     >>
>     >> I am using Ubuntu12.04 and the dnsmasq version is 2.59
>     >>
>     >> BR
>     >> Johanna
>     >>
>     >>
>     >> -----Original Message-----
>     >> From: Openstack [mailto:openstack-bounces+johanna.heinonen
>     <mailto:openstack-bounces%2Bjohanna.heinonen>=nsn.com@xxxxxxxxxxxxxxxxxxx
>     <mailto:nsn.com@xxxxxxxxxxxxxxxxxxx>] On Behalf Of ext Thomas Kärgel
>     >> Sent: Monday, May 20, 2013 10:09 AM
>     >> To: openstack@xxxxxxxxxxxxxxxxxxx
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     >> Subject: Re: [Openstack] DHCP problem in grizzly
>     >>
>     >> Hi Johanna,
>     >>
>     >> I'm facing the same behavior on a Folsom-installation on SLES11SP2. I
>     >> noticed that new hosts have correct entries in the dnsmasq
>     config-files.
>     >> The dnsmasq processes get a HUP signal by DHCP-Agent, but simply
>     refuse
>     >> to deliver the new address. Instead dnsmasq logs claim "no address
>     >> available". Exactly like in your description.
>     >> What distrubtion and exact dnsmasq-versions are in use on your
>     environment?
>     >> I assume dnsmasq is not rereading its configs correctly on signal
>     HUP.
>     >> dnsmasq logs claim it has reread configs, but it still does not
>     deliver
>     >> the new adresses in host-file.
>     >> Manually trying to sigHUP dnsmasq had no effect. The only way to
>     get out
>     >> of this state seems to be restarting DHCP-agent.
>     >>
>     >> Best regards
>     >> Thomas
>     >>
>     >> Am 20.05.2013 08:51, schrieb Heinonen, Johanna (NSN - FI/Espoo):
>     >>> Hi,
>     >>>
>     >>> I have installed grizzly with quantum and ovs-plugin. It seems that
>     >>> grizzly allocates the third address of each subnet for dhcp. (In
>     folsom
>     >>> it was the second address). This means that the VMs will get
>     addresses
>     >>> .2, .4, .5, .
>     >>>
>     >>> In my setup the first VM always boots fine and gets the address
>     x.x.x.2.
>     >>> This can be seen in the syslog:
>     >>>
>     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>     >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:2d:0d:e0
>     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>     >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>     DHCPACK(tapdbcef145-f5)
>     >>> 10.20.30.2 fa:16:3e:2d:0d:e0 10-20-30-2
>     >>>
>     >>> The problem comes when I start the second VM. Nova shows that the
>     >>> x.x.x.4 is allocated
>     >>>
>     >>> root@grizzly-236:~# nova list
>     >>>
>     +--------------------------------------+----------+--------+------------------------+
>     >>> | ID                                   | Name     | Status |
>     >>> Networks               |
>     >>>
>     +--------------------------------------+----------+--------+------------------------+
>     >>> | c112ccbb-5039-4d05-b414-b53a1eafa2d8 | q-test   | ACTIVE |
>     >>> tenant1-net=10.20.30.2 |
>     >>> | 4f26975c-995d-403d-88b0-e7bbf189baad | q-test-2 | ACTIVE |
>     >>> tenant1-net=10.20.30.4 |
>     >>>
>     +--------------------------------------+----------+--------+------------------------+
>     >>>
>     >>> But from syslog I see that the answer to the DHCPDISCOVER is "no
>     address
>     >>> available"
>     >>>
>     >>> May 20 08:33:34 grizzly-236 dnsmasq-dhcp[2190]:
>     >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:fc:1f:9a no address
>     >>> available
>     >>> May 20 08:33:52 grizzly-236 dnsmasq-dhcp[2190]:
>     >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:fc:1f:9a no address
>     >>> available
>     >>>
>     >>> When I restart the quantum-dhcp-server the problem disappears.
>     This can
>     >>> be seen from the syslog:
>     >>>
>     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: started, version 2.59
>     >>> cachesize 150
>     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: compile time options:
>     IPv6
>     >>> GNU-getopt DBus i18n DHCP TFTP conntrack IDN
>     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: warning: no upstream
>     servers
>     >>> configured
>     >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: DHCP, static
>     leases only
>     >>> on 10.20.30.0, lease time 2m
>     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: cleared cache
>     >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: read
>     >>> /var/lib/quantum/dhcp/e6f27330-be41-478c-b4d2-49ed4ce0af00/host
>     >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: read
>     >>> /var/lib/quantum/dhcp/e6f27330-be41-478c-b4d2-49ed4ce0af00/opts
>     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>     DHCPNAK(tapdbcef145-f5)
>     >>> 10.20.30.2 fa:16:3e:2d:0d:e0 lease not found
>     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>     >>> DHCPDISCOVER(tapdbcef145-f5) fa:16:3e:2d:0d:e0
>     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>     >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>     DHCPACK(tapdbcef145-f5)
>     >>> 10.20.30.2 fa:16:3e:2d:0d:e0 10-20-30-2
>     >>>
>     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>     >>> DHCPDISCOVER(tapdbcef145-f5) fa:16:3e:fc:1f:9a*
>     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>     >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.4 fa:16:3e:fc:1f:9a*
>     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.4 fa:16:3e:fc:1f:9a*
>     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>     DHCPACK(tapdbcef145-f5)
>     >>> 10.20.30.4 fa:16:3e:fc:1f:9a 10-20-30-4*
>     >>>
>     >>>
>     >>> What could be the problem? Have you seen similar behavior? If
>     yes, how
>     >>> did you fix this?
>     >>>
>     >>> Best regards,
>     >>> Johanna
>     >>>
>     >>>
>     >>>
>     >>> _______________________________________________
>     >>> Mailing list: https://launchpad.net/~openstack
>     >>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     >>> Unsubscribe : https://launchpad.net/~openstack
>     >>> More help   : https://help.launchpad.net/ListHelp
>     >>>
>     >>
>     >>
>     >
>     >
> 
> 
>     --
>     Thomas Kärgel
>     Linux Consultant
>     Mail: kaergel@xxxxxxxxxxxxx <mailto:kaergel@xxxxxxxxxxxxx>
>     B1 Systems GmbH
>     Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
>     GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
> 
> 
>     _______________________________________________
>     Mailing list: https://launchpad.net/~openstack
>     Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     Unsubscribe : https://launchpad.net/~openstack
>     More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
> -- 
> Marco Colombo


-- 
Thomas Kärgel
Linux Consultant
Tel.: +49 172 2037945
Mail: kaergel@xxxxxxxxxxxxx
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

Attachment: signature.asc
Description: OpenPGP digital signature