← Back to team overview

openstack team mailing list archive

Re: DHCP problem in grizzly

 

Hi,

the bug report is open and can be viewed at:
http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2013q2/007212.html

I used some of your logs in order to add more details to  my description.
Hopefully someone can help us solving this issue.

Best regards
thomas

Am 22.05.2013 17:05, schrieb Thomas Kärgel:
> Hi Marco,
> 
> thanks for the interesting information. I think we can be sure now that
> this is a bug in dnsmasq. I'll go ahead and report this bug to the
> dnsmasq-developers tomorrow morning. Hopefully they can fix this issue soon.
> I'll report the details when the bugreport is open. Maybe they want more
> infos from us...
> 
> Best regards
> Thomas
> 
> Am 22.05.2013 14:33, schrieb Marco Colombo:
>> Hi Thomas,
>> i have upgrade dnsmasq to version 2.66, but the problem persist.
>> If you need more log/information, i'm able to reproduce the problem.
>>
>> Thanks
>> Best Regards
>>
>>
>> 2013/5/22 Thomas Kärgel <kaergel@xxxxxxxxxxxxx
>> <mailto:kaergel@xxxxxxxxxxxxx>>
>>
>>     Hi Marco,
>>
>>     can you also reproduce it?
>>
>>     If so i would suggest to give dnsmasq 2.66 a try. Since i'm not able to
>>     reproduce this behavior on the environment i'm woking on it will take
>>     weeks to get to an really conclusive result.
>>
>>     Best regards
>>     thomas
>>
>>
>>     Am 22.05.2013 09:55, schrieb Marco Colombo:
>>     > Hi All,
>>     > i have the same problem with Grizzly, Ubuntu 12.04 and Dnsmasq 2.59
>>     > "killall dnsmasq && service quantum-dhcp-agent restart" fixes the
>>     > problem temporarily
>>     >
>>     > Best Regards,
>>     > Marco
>>     >
>>     >
>>     >
>>     >
>>     > 2013/5/22 Thomas Kärgel <kaergel@xxxxxxxxxxxxx
>>     <mailto:kaergel@xxxxxxxxxxxxx>
>>     > <mailto:kaergel@xxxxxxxxxxxxx <mailto:kaergel@xxxxxxxxxxxxx>>>
>>     >
>>     >     Hi Johanna,
>>     >
>>     >     sure, I'll report my results, but this can take a while. The
>>     last time
>>     >     this issue occurred on my environment is May 10th. It occurs quite
>>     >     sporadic and i don't know how to manually reproduce this issue
>>     yet (i
>>     >     have to be very careful with my experiments since the
>>     environment is
>>     >     running productive already). Maybe you should also give
>>     dnsmasq 2.66 a
>>     >     try, because you wrote that you can manually reproduce this
>>     behavior.
>>     >
>>     >
>>     >     Best Regards,
>>     >     Thomas
>>     >
>>     >
>>     >     Am 20.05.2013 12:25, schrieb Heinonen, Johanna (NSN - FI/Espoo):
>>     >     > Hi,
>>     >     >
>>     >     > You are right. Manually doing kill -HUP <PID> has just the
>>     effect
>>     >     you mentioned. In the syslog you can see
>>     >     >
>>     >     > May 20 13:09:41 grizzly-236 dnsmasq[6170]: cleared cache
>>     >     > May 20 13:09:41 grizzly-236 dnsmasq-dhcp[6170]: read
>>     >     /var/lib/quantum/dhcp/d5879bbb-ada6-4323-a7b1-87b5db244513/host
>>     >     > May 20 13:09:41 grizzly-236 dnsmasq-dhcp[6170]: read
>>     >     /var/lib/quantum/dhcp/d5879bbb-ada6-4323-a7b1-87b5db244513/opts
>>     >     >
>>     >     >
>>     >     > But the problem stays. Only 'service quantum-dhcp-agent restart'
>>     >     fixes the problem.
>>     >     > If you try the newer version of the qnsmasq, I'd be
>>     interested to
>>     >     hear the results.
>>     >     >
>>     >     > BR
>>     >     > Johanna
>>     >     >
>>     >     >
>>     >     >
>>     >     > -----Original Message-----
>>     >     > From: ext Thomas Kärgel [mailto:kaergel@xxxxxxxxxxxxx
>>     <mailto:kaergel@xxxxxxxxxxxxx>
>>     >     <mailto:kaergel@xxxxxxxxxxxxx <mailto:kaergel@xxxxxxxxxxxxx>>]
>>     >     > Sent: Monday, May 20, 2013 11:37 AM
>>     >     > To: Heinonen, Johanna (NSN - FI/Espoo)
>>     >     > Cc: openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>>     >     > Subject: Re: [Openstack] DHCP problem in grizzly
>>     >     >
>>     >     > Hi,
>>     >     >
>>     >     > thx for the interesting info, Johanna: dnsmasq version is
>>     also 2.59 on
>>     >     > my environment. Can you can confirm that manually sigHUPing all
>>     >     running
>>     >     > dnsmasq processes has no effect?
>>     >     > (dnsmasq claims to have reread the configs in syslog, but still
>>     >     refuses
>>     >     > to deliver the new addresses.)
>>     >     > Maybe updating dnsmasq to a more recent release would solve our
>>     >     problem.
>>     >     > Current stable is 2.66. I'll give it a try when i get back
>>     to office
>>     >     > tomorrow.
>>     >     >
>>     >     > Best regards
>>     >     >
>>     >     > Thomas
>>     >     >
>>     >     >
>>     >     > Am 20.05.2013 10:21, schrieb Heinonen, Johanna (NSN - FI/Espoo):
>>     >     >> Hi Thomas,
>>     >     >>
>>     >     >> I am using Ubuntu12.04 and the dnsmasq version is 2.59
>>     >     >>
>>     >     >> BR
>>     >     >> Johanna
>>     >     >>
>>     >     >>
>>     >     >> -----Original Message-----
>>     >     >> From: Openstack [mailto:openstack-bounces+johanna.heinonen
>>     <mailto:openstack-bounces%2Bjohanna.heinonen>
>>     >     <mailto:openstack-bounces%2Bjohanna.heinonen
>>     <mailto:openstack-bounces%252Bjohanna.heinonen>>=nsn.com@xxxxxxxxxxxxxxxxxxx
>>     <mailto:nsn.com@xxxxxxxxxxxxxxxxxxx>
>>     >     <mailto:nsn.com@xxxxxxxxxxxxxxxxxxx
>>     <mailto:nsn.com@xxxxxxxxxxxxxxxxxxx>>] On Behalf Of ext Thomas Kärgel
>>     >     >> Sent: Monday, May 20, 2013 10:09 AM
>>     >     >> To: openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>>     >     >> Subject: Re: [Openstack] DHCP problem in grizzly
>>     >     >>
>>     >     >> Hi Johanna,
>>     >     >>
>>     >     >> I'm facing the same behavior on a Folsom-installation on
>>     SLES11SP2. I
>>     >     >> noticed that new hosts have correct entries in the dnsmasq
>>     >     config-files.
>>     >     >> The dnsmasq processes get a HUP signal by DHCP-Agent, but
>>     simply
>>     >     refuse
>>     >     >> to deliver the new address. Instead dnsmasq logs claim "no
>>     address
>>     >     >> available". Exactly like in your description.
>>     >     >> What distrubtion and exact dnsmasq-versions are in use on your
>>     >     environment?
>>     >     >> I assume dnsmasq is not rereading its configs correctly on
>>     signal
>>     >     HUP.
>>     >     >> dnsmasq logs claim it has reread configs, but it still does not
>>     >     deliver
>>     >     >> the new adresses in host-file.
>>     >     >> Manually trying to sigHUP dnsmasq had no effect. The only
>>     way to
>>     >     get out
>>     >     >> of this state seems to be restarting DHCP-agent.
>>     >     >>
>>     >     >> Best regards
>>     >     >> Thomas
>>     >     >>
>>     >     >> Am 20.05.2013 08:51, schrieb Heinonen, Johanna (NSN -
>>     FI/Espoo):
>>     >     >>> Hi,
>>     >     >>>
>>     >     >>> I have installed grizzly with quantum and ovs-plugin. It
>>     seems that
>>     >     >>> grizzly allocates the third address of each subnet for
>>     dhcp. (In
>>     >     folsom
>>     >     >>> it was the second address). This means that the VMs will get
>>     >     addresses
>>     >     >>> .2, .4, .5, .
>>     >     >>>
>>     >     >>> In my setup the first VM always boots fine and gets the
>>     address
>>     >     x.x.x.2.
>>     >     >>> This can be seen in the syslog:
>>     >     >>>
>>     >     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>>     >     >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:2d:0d:e0
>>     >     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>>     >     >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>>     >     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>>     >     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>>     >     >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>>     >     DHCPACK(tapdbcef145-f5)
>>     >     >>> 10.20.30.2 fa:16:3e:2d:0d:e0 10-20-30-2
>>     >     >>>
>>     >     >>> The problem comes when I start the second VM. Nova shows
>>     that the
>>     >     >>> x.x.x.4 is allocated
>>     >     >>>
>>     >     >>> root@grizzly-236:~# nova list
>>     >     >>>
>>     >    
>>     +--------------------------------------+----------+--------+------------------------+
>>     >     >>> | ID                                   | Name     | Status |
>>     >     >>> Networks               |
>>     >     >>>
>>     >    
>>     +--------------------------------------+----------+--------+------------------------+
>>     >     >>> | c112ccbb-5039-4d05-b414-b53a1eafa2d8 | q-test   | ACTIVE |
>>     >     >>> tenant1-net=10.20.30.2 |
>>     >     >>> | 4f26975c-995d-403d-88b0-e7bbf189baad | q-test-2 | ACTIVE |
>>     >     >>> tenant1-net=10.20.30.4 |
>>     >     >>>
>>     >    
>>     +--------------------------------------+----------+--------+------------------------+
>>     >     >>>
>>     >     >>> But from syslog I see that the answer to the DHCPDISCOVER
>>     is "no
>>     >     address
>>     >     >>> available"
>>     >     >>>
>>     >     >>> May 20 08:33:34 grizzly-236 dnsmasq-dhcp[2190]:
>>     >     >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:fc:1f:9a
>>     no address
>>     >     >>> available
>>     >     >>> May 20 08:33:52 grizzly-236 dnsmasq-dhcp[2190]:
>>     >     >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:fc:1f:9a
>>     no address
>>     >     >>> available
>>     >     >>>
>>     >     >>> When I restart the quantum-dhcp-server the problem disappears.
>>     >     This can
>>     >     >>> be seen from the syslog:
>>     >     >>>
>>     >     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: started,
>>     version 2.59
>>     >     >>> cachesize 150
>>     >     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: compile time
>>     options:
>>     >     IPv6
>>     >     >>> GNU-getopt DBus i18n DHCP TFTP conntrack IDN
>>     >     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: warning: no
>>     upstream
>>     >     servers
>>     >     >>> configured
>>     >     >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: DHCP, static
>>     >     leases only
>>     >     >>> on 10.20.30.0, lease time 2m
>>     >     >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: cleared cache
>>     >     >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: read
>>     >     >>>
>>     /var/lib/quantum/dhcp/e6f27330-be41-478c-b4d2-49ed4ce0af00/host
>>     >     >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: read
>>     >     >>>
>>     /var/lib/quantum/dhcp/e6f27330-be41-478c-b4d2-49ed4ce0af00/opts
>>     >     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>>     >     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     DHCPNAK(tapdbcef145-f5)
>>     >     >>> 10.20.30.2 fa:16:3e:2d:0d:e0 lease not found
>>     >     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     >>> DHCPDISCOVER(tapdbcef145-f5) fa:16:3e:2d:0d:e0
>>     >     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>>     >     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>>     >     >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     DHCPACK(tapdbcef145-f5)
>>     >     >>> 10.20.30.2 fa:16:3e:2d:0d:e0 10-20-30-2
>>     >     >>>
>>     >     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     >>> DHCPDISCOVER(tapdbcef145-f5) fa:16:3e:fc:1f:9a*
>>     >     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.4 fa:16:3e:fc:1f:9a*
>>     >     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.4 fa:16:3e:fc:1f:9a*
>>     >     >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>>     >     DHCPACK(tapdbcef145-f5)
>>     >     >>> 10.20.30.4 fa:16:3e:fc:1f:9a 10-20-30-4*
>>     >     >>>
>>     >     >>>
>>     >     >>> What could be the problem? Have you seen similar behavior? If
>>     >     yes, how
>>     >     >>> did you fix this?
>>     >     >>>
>>     >     >>> Best regards,
>>     >     >>> Johanna
>>     >     >>>
>>     >     >>>
>>     >     >>>
>>     >     >>> _______________________________________________
>>     >     >>> Mailing list: https://launchpad.net/~openstack
>>     >     >>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>>     >     >>> Unsubscribe : https://launchpad.net/~openstack
>>     >     >>> More help   : https://help.launchpad.net/ListHelp
>>     >     >>>
>>     >     >>
>>     >     >>
>>     >     >
>>     >     >
>>     >
>>     >
>>     >     --
>>     >     Thomas Kärgel
>>     >     Linux Consultant
>>     >     Mail: kaergel@xxxxxxxxxxxxx <mailto:kaergel@xxxxxxxxxxxxx>
>>     <mailto:kaergel@xxxxxxxxxxxxx <mailto:kaergel@xxxxxxxxxxxxx>>
>>     >     B1 Systems GmbH
>>     >     Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
>>     >     GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG:
>>     Ingolstadt,HRB 3537
>>     >
>>     >
>>     >     _______________________________________________
>>     >     Mailing list: https://launchpad.net/~openstack
>>     >     Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>>     >     Unsubscribe : https://launchpad.net/~openstack
>>     >     More help   : https://help.launchpad.net/ListHelp
>>     >
>>     >
>>     >
>>     >
>>     > --
>>     > Marco Colombo
>>
>>
>>     --
>>     Thomas Kärgel
>>     Linux Consultant
>>     Tel.: +49 172 2037945
>>     Mail: kaergel@xxxxxxxxxxxxx <mailto:kaergel@xxxxxxxxxxxxx>
>>     B1 Systems GmbH
>>     Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
>>     GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
>>
>>
>>
>>
>> -- 
>> Marco Colombo
> 
> 
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


-- 
Thomas Kärgel
Linux Consultant
Mail: kaergel@xxxxxxxxxxxxx
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

Attachment: signature.asc
Description: OpenPGP digital signature


Follow ups

References