openstack team mailing list archive

[Salvatore Orlando <sorlando@xxxxxxxxxx>]

Thanks Gary.

I have looked independently at this issue and came to the same conclusion.
How do you reckon we should go to fix this?
I think we agree on the diagnosis, but I reckon the severity is bigger than
just a usability issue.

This might affect anyone developing diagnostic/reporting/billing tools on
quantum API.
For instance, if I was a subscriber I would refuse to pay my service
provider for usage of floating IP, as I would tell them: "look, they've
been down all the time!".
Luckily, not all people are cheeky and dishonest like me.

Salvatore


On 20 June 2013 13:03, Gary Kotton <gkotton@xxxxxxxxxx> wrote:

>  On 06/20/2013 11:59 AM, Gary Kotton wrote:
>
> On 06/19/2013 07:00 PM, Samuel Winchenbach wrote:
>
>  Here are ALL the warnings and errors I see when restarting nova and
> quantum:  http://pastie.org/pastes/8059837/text
> Here are ALL the warnings and errors I see when creating both networks
> (public and internal), the router that bridges them, assigning the public
> network as the gateway, adding the internal interface to it, launching a
> VM, allocating a floating IP, and assigning it to the VM:
> http://pastie.org/pastes/8059850/text
>
>  I am not sure about the nbd stuff, but I think think that is just
> because the qemu disk image is not resized to the disk size yet.   The ONLY
> warning that I think might effect me is:
>
>  Jun 19 11:53:32 test1 dnsmasq[28913]: warning: no upstream servers
> configured
>
>  Any idea if that could be causing my DOWN ports issue?
>
>
> Sorry to chime in pretty late. This is a bug. In certain plugins when a
> port is created the status of the port is set to DOWN. This port is set to
> UP when the plugin receives an indication that the port is actually UP (for
> example when a agent detects that a tap device has been created). This is
> not treated well when dealing with gateway ports and flotaing IP's.
>
>
> Please note that this is just a usability issue. The port is actually up
> and forwarding traffic.
>
>
> I'll take care of this.
> Thanks
> Gary
>
>
>  Thanks,
> Sam
>
>
> On Wed, Jun 19, 2013 at 11:03 AM, Samuel Winchenbach <swinchen@xxxxxxxxx>wrote:
>
>> I
>> did that, and I went one step further... I deleted and recreated ALL the
>> databases to ensure that I would be starting from scratch.  I also deleted
>> all bridges, stopped openvswitch, deleted conf.db, started openvswitch and
>> recreated all the bridges.
>>
>>   This solved the reference to "test1-int" but the ports are still stuck
>> in the down state :(
>>
>>   Sam
>>
>>
>> On Tue, Jun 18, 2013 at 4:39 PM, Filipe Manco <filipe.manco@xxxxxxxxx>wrote:
>>
>>> Stop quantum agents (not quantum-api) and openvswitch service. Delete
>>> /etc/openvswitch/conf.db and delte all agents using quantum
>>> agent-delete <id>. Start openvswitch service and then quantum agents.
>>> If ports still down check quantum logs mainly quantum l3-agent. If you
>>> don't find anything interesting delete the networks and recreate them.
>>>
>>>  If you still have references to "test1-int" check the quantum database ovs_tunnel_endpoints
>>> table and manually remove any reference.
>>>
>>>  Filipe Manco
>>> http://about.me/fmanco
>>>
>>>
>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>
>>>>  Hmmm I used both of those commands, but no matter what I do I can not
>>>> remove references to "test1-int" in /etc/openvswitch/conf.db
>>>>
>>>>  Should I just manually replace those with the IP?  Delete the file?
>>>>
>>>>
>>>> On Tue, Jun 18, 2013 at 1:14 PM, Filipe Manco <filipe.manco@xxxxxxxxx>wrote:
>>>>
>>>>> Honestly I'm not sure because I've always used IPs. But according to
>>>>> the logs it looks so. After changing configurations you should probably run
>>>>> quantum-netns-cleanup and quantum-ovs-cleanup before starting the services.
>>>>>
>>>>>  Filipe Manco
>>>>> http://about.me/fmanco
>>>>>
>>>>>
>>>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>>
>>>>>>  I think I may be onto something:
>>>>>> http://pastie.org/pastes/8056137/text
>>>>>>
>>>>>>  from syslog
>>>>>> Jun 18 12:57:26 test1 ovs-vsctl: 00001|vsctl|INFO|Called as
>>>>>> /usr/bin/ovs-vsctl -- --may-exist add-port br-int qvo3eb6d144-07 -- set
>>>>>> Interface qvo3eb6d144-07
>>>>>> external-ids:iface-id=3eb6d144-077e-42cf-ad2e-57c50aa00399
>>>>>> external-ids:iface-status=active
>>>>>> external-ids:attached-mac=fa:16:3e:92:31:1e
>>>>>> external-ids:vm-uuid=add44e48-6f42-4ede-a646-f29e74ccc02d
>>>>>>  Jun 18 12:57:26 test1 ovs-vswitchd:
>>>>>> 03753|socket_util|ERR|"test1-int" is not a valid IP address
>>>>>>  Jun 18 12:57:26 test1 ovs-vswitchd: 03755|netdev_vport|ERR|gre-1:
>>>>>> gre type requires valid 'remote_ip' argument
>>>>>> Jun 18 12:57:30 test1 ovs-vsctl: 00001|vsctl|INFO|Called as
>>>>>> /usr/bin/ovs-vsctl --timeout=2 set Port qvo3eb6d144-07 tag=1
>>>>>>  Jun 18 12:57:30 test1 ovs-vswitchd: 03768|netdev_vport|ERR|gre-1:
>>>>>> gre type requires valid 'remote_ip' argument
>>>>>>
>>>>>>
>>>>>>  Looks like you might not be able to use entries from /etc/hosts in
>>>>>> the config files?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 18, 2013 at 10:42 AM, Samuel Winchenbach <
>>>>>> swinchen@xxxxxxxxx> wrote:
>>>>>>
>>>>>>>  I have three agents running (Open vSwitch agent, DHCP agent, and
>>>>>>> L3 agent): http://pastie.org/pastes/8055658/text
>>>>>>>  The agents listed on test3 are there because ubuntu starts them
>>>>>>> automatically.  L3 agent will never run on test3 because it doesn't even
>>>>>>> have an external interface.   Right now I am just trying to limit it to one
>>>>>>> node.
>>>>>>>
>>>>>>>  Here is my l3_agent.ini:  http://pastie.org/pastes/8055674/text
>>>>>>> Here are a list of bridges (eth1 is my external interface)
>>>>>>> http://pastie.org/pastes/8055678/text
>>>>>>>
>>>>>>>  Thanks again for all your help!
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 18, 2013 at 10:32 AM, Filipe Manco <
>>>>>>> filipe.manco@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> What is the status of quantum agent-list? I see on your node test3
>>>>>>>> the agents are down and you don't have openvswitch agent.
>>>>>>>> I would check for the logs of the l3 agent? Have you configured the
>>>>>>>> external network id on the l3 agent config file?
>>>>>>>>
>>>>>>>>  Filipe Manco
>>>>>>>> http://about.me/fmanco
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>>>>>
>>>>>>>>>  Hi Filipe,
>>>>>>>>>
>>>>>>>>>  Thanks for the response.  I already had the
>>>>>>>>> /etc/sudoers.d/quantum_sudoers file.   On a whim I added "root_helper =
>>>>>>>>> sudo quantum-rootwrap /etc/quantum/rootwrap.conf" to
>>>>>>>>> /etc/quantum/dhcp_agent.ini and that took care of that problem.
>>>>>>>>>
>>>>>>>>>  I managed to remove the libvirt errors by disabling apparmor.
>>>>>>>>>
>>>>>>>>>  All the ports on my public network are still listed as "DOWN"
>>>>>>>>> I have managed to remove all of the errors and warnings from quantum but
>>>>>>>>> those ports will still not come up.   I really am lost.
>>>>>>>>>
>>>>>>>>>  Thanks again for the post, I am not sure what to try next :/
>>>>>>>>>
>>>>>>>>>  Sam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  On Tue, Jun 18, 2013 at 10:13 AM, Filipe Manco <
>>>>>>>>> filipe.manco@xxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> From what I can see in the logs you must create the file
>>>>>>>>>> /etc/sudoers.d/quantum_sudoers with the following contents:
>>>>>>>>>>
>>>>>>>>>>  Defaults:quantum !requiretty
>>>>>>>>>> quantum ALL = (root) NOPASSWD: /usr/bin/quantum-rootwrap
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  About the libvirt error edit the file /etc/libvirt/qemu.confand add the following:
>>>>>>>>>>
>>>>>>>>>>  cgroup_device_acl = [
>>>>>>>>>>     "/dev/null", "/dev/full", "/dev/zero",
>>>>>>>>>>     "/dev/random", "/dev/urandom",
>>>>>>>>>>     "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
>>>>>>>>>>     "/dev/rtc","/dev/hpet" , "/dev/net/tun"
>>>>>>>>>> ]
>>>>>>>>>>
>>>>>>>>>>  Probably this won't fix all of your issues. The logs ofI don't
>>>>>>>>>> the l3 agent will be helpful.
>>>>>>>>>>
>>>>>>>>>>  Filipe Manco
>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>>>>>>>
>>>>>>>>>>>   I may have found the cause of my problem, but I am unsure of
>>>>>>>>>>> the solution.  In my libvirt log file I found many error
>>>>>>>>>>> messages similar to this:
>>>>>>>>>>>
>>>>>>>>>>>  2013-06-18 13:12:19.812+0000: 8353: warning : virAuditSend:135
>>>>>>>>>>> : Failed to send audit message virt=kvm resrc=net reason=open
>>>>>>>>>>> vm="instance-00000033" uuid=bca8a09e-46aa-408b-81cd-2432068361c1
>>>>>>>>>>> net=FA:16:3E:71:7F:68 path="/dev/net/tun" rdev=0A:C8: Operation not
>>>>>>>>>>> permitted
>>>>>>>>>>>
>>>>>>>>>>>  Sam
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  On Mon, Jun 17, 2013 at 8:52 PM, Samuel Winchenbach <
>>>>>>>>>>> swinchen@xxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>   Here is a bunch more information from quantum:
>>>>>>>>>>>> http://pastie.org/pastes/8053820/text
>>>>>>>>>>>>
>>>>>>>>>>>>  If anyone has any ideas I would really appreciate it.  Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 17, 2013 at 5:07 PM, Samuel Winchenbach <
>>>>>>>>>>>> swinchen@xxxxxxxxx> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have been stuck on a problem for a couple of days now.  I am
>>>>>>>>>>>>> using Grizzly on Ubuntu 12.04 LTS.  I can launch vms, create networks,
>>>>>>>>>>>>> subnets, routers, etc.  The problem is quantum reports that all fors on the
>>>>>>>>>>>>> public network are "DOWN"  for example:
>>>>>>>>>>>>> http://pastie.org/pastes/8053283/text
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does anyone have any hints or tips on what might be causing
>>>>>>>>>>>>> this, or the errors listed below in the quantum logs?  Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> quantum configuration: http://pastie.org/pastes/8053100/text
>>>>>>>>>>>>>
>>>>>>>>>>>>> nova configuration: http://pastie.org/pastes/8043800/text
>>>>>>>>>>>>>
>>>>>>>>>>>>> quantum logs: http://pastie.org/pastes/8053269/text (this is
>>>>>>>>>>>>> everything logged during the creation the networks, launching of vm
>>>>>>>>>>>>> instances, allocating the floating ip and assigning it to a VM)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  _______________________________________________
>>>>>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>