openstack team mailing list archive

[Sam Su <susltd.su@xxxxxxxxx>]

That's great, thank you for your efforts. Can you make a backport for essex?  

Sent from my iPhone

On Aug 24, 2012, at 7:15 PM, heut2008 <heut2008@xxxxxxxxx> wrote:

> I have fixed it here  https://review.openstack.org/#/c/11925/
> 
> 2012/8/25 Sam Su <susltd.su@xxxxxxxxx>:
>> Hi,
>> 
>> I also reported this bug:
>> https://bugs.launchpad.net/nova/+bug/1040255
>> 
>> If someone can combine you guys solution and get a perfect way to fix this
>> bug, that will be great.
>> 
>> BRs,
>> Sam
>> 
>> 
>> On Thu, Aug 23, 2012 at 9:27 PM, heut2008 <heut2008@xxxxxxxxx> wrote:
>>> 
>>> this bug has been filed here  https://bugs.launchpad.net/nova/+bug/1040537
>>> 
>>> 2012/8/24 Vishvananda Ishaya <vishvananda@xxxxxxxxx>:
>>>> +1 to this. Evan, can you report a bug (if one hasn't been reported yet)
>>>> and
>>>> propose the fix? Or else I can find someone else to propose it.
>>>> 
>>>> Vish
>>>> 
>>>> On Aug 23, 2012, at 1:38 PM, Evan Callicoat <diopter@xxxxxxxxx> wrote:
>>>> 
>>>> Hello all!
>>>> 
>>>> I'm the original author of the hairpin patch, and things have changed a
>>>> little bit in Essex and Folsom from the original Diablo target. I
>>>> believe I
>>>> can shed some light on what should be done here to solve the issue in
>>>> either
>>>> case.
>>>> 
>>>> ---
>>>> For Essex (stable/essex), in nova/virt/libvirt/connection.py:
>>>> ---
>>>> 
>>>> Currently _enable_hairpin() is only being called from spawn(). However,
>>>> spawn() is not the only place that vifs (veth#) get added to a bridge
>>>> (which
>>>> is when we need to enable hairpin_mode on them). The more relevant
>>>> function
>>>> is _create_new_domain(), which is called from spawn() and other places.
>>>> Without changing the information that gets passed to
>>>> _create_new_domain()
>>>> (which is just 'xml' from to_xml()), we can easily rewrite the first 2
>>>> lines
>>>> in _enable_hairpin(), as follows:
>>>> 
>>>> def _enable_hairpin(self, xml):
>>>>    interfaces = self.get_interfaces(xml['name'])
>>>> 
>>>> Then, we can move the self._enable_hairpin(instance) call from spawn()
>>>> up
>>>> into _create_new_domain(), and pass it xml as follows:
>>>> 
>>>> [...]
>>>> self._enable_hairpin(xml)
>>>> return domain
>>>> 
>>>> This will run the hairpin code every time a domain gets created, which
>>>> is
>>>> also when the domain's vif(s) gets inserted into the bridge with the
>>>> default
>>>> of hairpin_mode=0.
>>>> 
>>>> ---
>>>> For Folsom (trunk), in nova/virt/libvirt/driver.py:
>>>> ---
>>>> 
>>>> There've been a lot more changes made here, but the same strategy as
>>>> above
>>>> should work. Here, _create_new_domain() has been split into
>>>> _create_domain()
>>>> and _create_domain_and_network(), and _enable_hairpin() was moved from
>>>> spawn() to _create_domain_and_network(), which seems like it'd be the
>>>> right
>>>> thing to do, but doesn't quite cover all of the cases of vif
>>>> reinsertion,
>>>> since _create_domain() is the only function which actually creates the
>>>> domain (_create_domain_and_network() just calls it after doing some
>>>> pre-work). The solution here is likewise fairly simple; make the same 2
>>>> changes to _enable_hairpin():
>>>> 
>>>> def _enable_hairpin(self, xml):
>>>>    interfaces = self.get_interfaces(xml['name'])
>>>> 
>>>> And move it from _create_domain_and_network() to _create_domain(), like
>>>> before:
>>>> 
>>>> [...]
>>>> self._enable_hairpin(xml)
>>>> return domain
>>>> 
>>>> I haven't yet tested this on my Essex clusters and I don't have a Folsom
>>>> cluster handy at present, but the change is simple and makes sense.
>>>> Looking
>>>> at to_xml() and _prepare_xml_info(), it appears that the 'xml' variable
>>>> _create_[new_]domain() gets is just a python dictionary, and xml['name']
>>>> =
>>>> instance['name'], exactly what _enable_hairpin() was using the
>>>> 'instance'
>>>> variable for previously.
>>>> 
>>>> Let me know if this works, or doesn't work, or doesn't make sense, or if
>>>> you
>>>> need an address to send gifts, etc. Hope it's solved!
>>>> 
>>>> -Evan
>>>> 
>>>> On Thu, Aug 23, 2012 at 11:20 AM, Sam Su <susltd.su@xxxxxxxxx> wrote:
>>>>> 
>>>>> Hi Oleg,
>>>>> 
>>>>> Thank you for your investigation. Good lucky!
>>>>> 
>>>>> Can you let me know if find how to fix the bug?
>>>>> 
>>>>> Thanks,
>>>>> Sam
>>>>> 
>>>>> On Wed, Aug 22, 2012 at 12:50 PM, Oleg Gelbukh <ogelbukh@xxxxxxxxxxxx>
>>>>> wrote:
>>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> Is it possible that, during snapshotting, libvirt just tears down
>>>>>> virtual
>>>>>> interface at some point, and then re-creates it, with hairpin_mode
>>>>>> disabled
>>>>>> again?
>>>>>> This bugfix [https://bugs.launchpad.net/nova/+bug/933640] implies that
>>>>>> fix works on spawn of instance. This means that upon resume after
>>>>>> snapshot,
>>>>>> hairpin is not restored. May be if we insert the _enable_hairpin()
>>>>>> call in
>>>>>> snapshot procedure, it helps.
>>>>>> We're currently investigating this issue in one of our environments,
>>>>>> hope
>>>>>> to come up with answer by tomorrow.
>>>>>> 
>>>>>> --
>>>>>> Best regards,
>>>>>> Oleg
>>>>>> 
>>>>>> On Wed, Aug 22, 2012 at 11:29 PM, Sam Su <susltd.su@xxxxxxxxx> wrote:
>>>>>>> 
>>>>>>> My friend has found a way to enable ping itself, when this problem
>>>>>>> happened. But not found why this happen.
>>>>>>> sudo echo "1" >
>>>>>>> /sys/class/net/br1000/brif/<virtual-interface-name>/hairpin_mode
>>>>>>> 
>>>>>>> I file a ticket to report this problem:
>>>>>>> https://bugs.launchpad.net/nova/+bug/1040255
>>>>>>> 
>>>>>>> hopefully someone can find why this happen and solve it.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Sam
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jul 20, 2012 at 3:50 PM, Gabriel Hurley
>>>>>>> <Gabriel.Hurley@xxxxxxxxxx> wrote:
>>>>>>>> 
>>>>>>>> I ran into some similar issues with the _enable_hairpin() call. The
>>>>>>>> call is allowed to fail silently and (in my case) was failing. I
>>>>>>>> couldn’t
>>>>>>>> for the life of me figure out why, though, and since I’m really not
>>>>>>>> a
>>>>>>>> networking person I didn’t trace it along too far.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Just thought I’d share my similar pain.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -          Gabriel
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> From:
>>>>>>>> openstack-bounces+gabriel.hurley=nebula.com@xxxxxxxxxxxxxxxxxxx
>>>>>>>> 
>>>>>>>> [mailto:openstack-bounces+gabriel.hurley=nebula.com@xxxxxxxxxxxxxxxxxxx] On
>>>>>>>> Behalf Of Sam Su
>>>>>>>> Sent: Thursday, July 19, 2012 11:50 AM
>>>>>>>> To: Brian Haley
>>>>>>>> Cc: openstack
>>>>>>>> Subject: Re: [Openstack] VM can't ping self floating IP after a
>>>>>>>> snapshot is taken
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thank you for your support.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I checked the file  nova/virt/libvirt/connection.py, the sentence
>>>>>>>> self._enable_hairpin(instance) is already added to the function
>>>>>>>> _hard_reboot().
>>>>>>>> 
>>>>>>>> It looks like there are some difference between taking snapshot and
>>>>>>>> reboot instance. I tried to figure out how to fix this bug but
>>>>>>>> failed.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> It will be much appreciated if anyone can give some hints.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Sam
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, Jul 19, 2012 at 8:37 AM, Brian Haley <brian.haley@xxxxxx>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> On 07/17/2012 05:56 PM, Sam Su wrote:
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> Just This always happens in Essex release. After I take a snapshot
>>>>>>>>> of
>>>>>>>>> my VM ( I
>>>>>>>>> tried Ubuntu 12.04 or CentOS 5.8), VM can't ping its self floating
>>>>>>>>> IP; before I
>>>>>>>>> take a snapshot though, VM can ping its self floating IP.
>>>>>>>>> 
>>>>>>>>> This looks closely related to
>>>>>>>>> https://bugs.launchpad.net/nova/+bug/933640, but
>>>>>>>>> still a little different. In 933640, it sounds like VM can't ping
>>>>>>>>> its
>>>>>>>>> self
>>>>>>>>> floating IP regardless whether we take a snapshot or not.
>>>>>>>>> 
>>>>>>>>> Any suggestion to make an easy fix? And what is the root cause of
>>>>>>>>> the
>>>>>>>>> problem?
>>>>>>>> 
>>>>>>>> It might be because there's a missing _enable_hairpin() call in the
>>>>>>>> reboot()
>>>>>>>> function.  Try something like this...
>>>>>>>> 
>>>>>>>> nova/virt/libvirt/connection.py, _hard_reboot():
>>>>>>>> 
>>>>>>>>             self._create_new_domain(xml)
>>>>>>>> +            self._enable_hairpin(instance)
>>>>>>>>             self.firewall_driver.apply_instance_filter(instance,
>>>>>>>> network_info)
>>>>>>>> 
>>>>>>>> At least that's what I remember doing myself recently when testing
>>>>>>>> after a
>>>>>>>> reboot, don't know about snapshot.
>>>>>>>> 
>>>>>>>> Folsom has changed enough that something different would need to be
>>>>>>>> done there.
>>>>>>>> 
>>>>>>>> -Brian
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>> 
>>> 
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>> 
>>