openstack team mailing list archive

[Samuel Winchenbach <swinchen@xxxxxxxxx>]

Thanks, I feel much less crazy now.   I had some other problems with my
public network that gmi (from IRC) was able to help me with but even after
fixing that the ports were listed as down even though it appeared to be
fine.  I thought I was losing my mind.....  well that is still a distinct
possibility.

Sam



On Thu, Jun 20, 2013 at 9:40 AM, Salvatore Orlando <sorlando@xxxxxxxxxx>wrote:

> Thanks Gary.
>
> I have looked independently at this issue and came to the same conclusion.
> How do you reckon we should go to fix this?
> I think we agree on the diagnosis, but I reckon the severity is bigger
> than just a usability issue.
>
> This might affect anyone developing diagnostic/reporting/billing tools on
> quantum API.
> For instance, if I was a subscriber I would refuse to pay my service
> provider for usage of floating IP, as I would tell them: "look, they've
> been down all the time!".
> Luckily, not all people are cheeky and dishonest like me.
>
> Salvatore
>
>
> On 20 June 2013 13:03, Gary Kotton <gkotton@xxxxxxxxxx> wrote:
>
>>  On 06/20/2013 11:59 AM, Gary Kotton wrote:
>>
>> On 06/19/2013 07:00 PM, Samuel Winchenbach wrote:
>>
>>  Here are ALL the warnings and errors I see when restarting nova and
>> quantum:  http://pastie.org/pastes/8059837/text
>> Here are ALL the warnings and errors I see when creating both networks
>> (public and internal), the router that bridges them, assigning the public
>> network as the gateway, adding the internal interface to it, launching a
>> VM, allocating a floating IP, and assigning it to the VM:
>> http://pastie.org/pastes/8059850/text
>>
>>  I am not sure about the nbd stuff, but I think think that is just
>> because the qemu disk image is not resized to the disk size yet.   The ONLY
>> warning that I think might effect me is:
>>
>>  Jun 19 11:53:32 test1 dnsmasq[28913]: warning: no upstream servers
>> configured
>>
>>  Any idea if that could be causing my DOWN ports issue?
>>
>>
>> Sorry to chime in pretty late. This is a bug. In certain plugins when a
>> port is created the status of the port is set to DOWN. This port is set to
>> UP when the plugin receives an indication that the port is actually UP (for
>> example when a agent detects that a tap device has been created). This is
>> not treated well when dealing with gateway ports and flotaing IP's.
>>
>>
>> Please note that this is just a usability issue. The port is actually up
>> and forwarding traffic.
>>
>>
>> I'll take care of this.
>> Thanks
>> Gary
>>
>>
>>  Thanks,
>> Sam
>>
>>
>> On Wed, Jun 19, 2013 at 11:03 AM, Samuel Winchenbach <swinchen@xxxxxxxxx>wrote:
>>
>>> I
>>> did that, and I went one step further... I deleted and recreated ALL the
>>> databases to ensure that I would be starting from scratch.  I also deleted
>>> all bridges, stopped openvswitch, deleted conf.db, started openvswitch and
>>> recreated all the bridges.
>>>
>>>   This solved the reference to "test1-int" but the ports are still
>>> stuck in the down state :(
>>>
>>>   Sam
>>>
>>>
>>> On Tue, Jun 18, 2013 at 4:39 PM, Filipe Manco <filipe.manco@xxxxxxxxx>wrote:
>>>
>>>> Stop quantum agents (not quantum-api) and openvswitch service. Delete
>>>> /etc/openvswitch/conf.db and delte all agents using quantum
>>>> agent-delete <id>. Start openvswitch service and then quantum agents.
>>>> If ports still down check quantum logs mainly quantum l3-agent. If you
>>>> don't find anything interesting delete the networks and recreate them.
>>>>
>>>>  If you still have references to "test1-int" check the quantum
>>>> database ovs_tunnel_endpoints table and manually remove any reference.
>>>>
>>>>  Filipe Manco
>>>> http://about.me/fmanco
>>>>
>>>>
>>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>
>>>>>  Hmmm I used both of those commands, but no matter what I do I can
>>>>> not remove references to "test1-int" in /etc/openvswitch/conf.db
>>>>>
>>>>>  Should I just manually replace those with the IP?  Delete the file?
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 18, 2013 at 1:14 PM, Filipe Manco <filipe.manco@xxxxxxxxx>wrote:
>>>>>
>>>>>> Honestly I'm not sure because I've always used IPs. But according to
>>>>>> the logs it looks so. After changing configurations you should probably run
>>>>>> quantum-netns-cleanup and quantum-ovs-cleanup before starting the services.
>>>>>>
>>>>>>  Filipe Manco
>>>>>> http://about.me/fmanco
>>>>>>
>>>>>>
>>>>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>>>
>>>>>>>  I think I may be onto something:
>>>>>>> http://pastie.org/pastes/8056137/text
>>>>>>>
>>>>>>>  from syslog
>>>>>>> Jun 18 12:57:26 test1 ovs-vsctl: 00001|vsctl|INFO|Called as
>>>>>>> /usr/bin/ovs-vsctl -- --may-exist add-port br-int qvo3eb6d144-07 -- set
>>>>>>> Interface qvo3eb6d144-07
>>>>>>> external-ids:iface-id=3eb6d144-077e-42cf-ad2e-57c50aa00399
>>>>>>> external-ids:iface-status=active
>>>>>>> external-ids:attached-mac=fa:16:3e:92:31:1e
>>>>>>> external-ids:vm-uuid=add44e48-6f42-4ede-a646-f29e74ccc02d
>>>>>>>  Jun 18 12:57:26 test1 ovs-vswitchd:
>>>>>>> 03753|socket_util|ERR|"test1-int" is not a valid IP address
>>>>>>>  Jun 18 12:57:26 test1 ovs-vswitchd: 03755|netdev_vport|ERR|gre-1:
>>>>>>> gre type requires valid 'remote_ip' argument
>>>>>>> Jun 18 12:57:30 test1 ovs-vsctl: 00001|vsctl|INFO|Called as
>>>>>>> /usr/bin/ovs-vsctl --timeout=2 set Port qvo3eb6d144-07 tag=1
>>>>>>>  Jun 18 12:57:30 test1 ovs-vswitchd: 03768|netdev_vport|ERR|gre-1:
>>>>>>> gre type requires valid 'remote_ip' argument
>>>>>>>
>>>>>>>
>>>>>>>  Looks like you might not be able to use entries from /etc/hosts in
>>>>>>> the config files?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 18, 2013 at 10:42 AM, Samuel Winchenbach <
>>>>>>> swinchen@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>>  I have three agents running (Open vSwitch agent, DHCP agent, and
>>>>>>>> L3 agent): http://pastie.org/pastes/8055658/text
>>>>>>>>  The agents listed on test3 are there because ubuntu starts them
>>>>>>>> automatically.  L3 agent will never run on test3 because it doesn't even
>>>>>>>> have an external interface.   Right now I am just trying to limit it to one
>>>>>>>> node.
>>>>>>>>
>>>>>>>>  Here is my l3_agent.ini:  http://pastie.org/pastes/8055674/text
>>>>>>>> Here are a list of bridges (eth1 is my external interface)
>>>>>>>> http://pastie.org/pastes/8055678/text
>>>>>>>>
>>>>>>>>  Thanks again for all your help!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 18, 2013 at 10:32 AM, Filipe Manco <
>>>>>>>> filipe.manco@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> What is the status of quantum agent-list? I see on your node
>>>>>>>>> test3 the agents are down and you don't have openvswitch agent.
>>>>>>>>> I would check for the logs of the l3 agent? Have you configured
>>>>>>>>> the external network id on the l3 agent config file?
>>>>>>>>>
>>>>>>>>>  Filipe Manco
>>>>>>>>> http://about.me/fmanco
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>>>>>>
>>>>>>>>>>  Hi Filipe,
>>>>>>>>>>
>>>>>>>>>>  Thanks for the response.  I already had the
>>>>>>>>>> /etc/sudoers.d/quantum_sudoers file.   On a whim I added "root_helper =
>>>>>>>>>> sudo quantum-rootwrap /etc/quantum/rootwrap.conf" to
>>>>>>>>>> /etc/quantum/dhcp_agent.ini and that took care of that problem.
>>>>>>>>>>
>>>>>>>>>>  I managed to remove the libvirt errors by disabling apparmor.
>>>>>>>>>>
>>>>>>>>>>  All the ports on my public network are still listed as "DOWN"
>>>>>>>>>> I have managed to remove all of the errors and warnings from quantum but
>>>>>>>>>> those ports will still not come up.   I really am lost.
>>>>>>>>>>
>>>>>>>>>>  Thanks again for the post, I am not sure what to try next :/
>>>>>>>>>>
>>>>>>>>>>  Sam
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  On Tue, Jun 18, 2013 at 10:13 AM, Filipe Manco <
>>>>>>>>>> filipe.manco@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> From what I can see in the logs you must create the file
>>>>>>>>>>> /etc/sudoers.d/quantum_sudoers with the following contents:
>>>>>>>>>>>
>>>>>>>>>>>  Defaults:quantum !requiretty
>>>>>>>>>>> quantum ALL = (root) NOPASSWD: /usr/bin/quantum-rootwrap
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  About the libvirt error edit the file /etc/libvirt/qemu.confand add the following:
>>>>>>>>>>>
>>>>>>>>>>>  cgroup_device_acl = [
>>>>>>>>>>>     "/dev/null", "/dev/full", "/dev/zero",
>>>>>>>>>>>     "/dev/random", "/dev/urandom",
>>>>>>>>>>>     "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
>>>>>>>>>>>     "/dev/rtc","/dev/hpet" , "/dev/net/tun"
>>>>>>>>>>> ]
>>>>>>>>>>>
>>>>>>>>>>>  Probably this won't fix all of your issues. The logs ofI don't
>>>>>>>>>>> the l3 agent will be helpful.
>>>>>>>>>>>
>>>>>>>>>>>  Filipe Manco
>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/6/18 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>>>>>>>>
>>>>>>>>>>>>   I may have found the cause of my problem, but I am unsure of
>>>>>>>>>>>> the solution.  In my libvirt log file I found many error
>>>>>>>>>>>> messages similar to this:
>>>>>>>>>>>>
>>>>>>>>>>>>  2013-06-18 13:12:19.812+0000: 8353: warning :
>>>>>>>>>>>> virAuditSend:135 : Failed to send audit message virt=kvm resrc=net
>>>>>>>>>>>> reason=open vm="instance-00000033"
>>>>>>>>>>>> uuid=bca8a09e-46aa-408b-81cd-2432068361c1 net=FA:16:3E:71:7F:68
>>>>>>>>>>>> path="/dev/net/tun" rdev=0A:C8: Operation not permitted
>>>>>>>>>>>>
>>>>>>>>>>>>  Sam
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  On Mon, Jun 17, 2013 at 8:52 PM, Samuel Winchenbach <
>>>>>>>>>>>> swinchen@xxxxxxxxx> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>   Here is a bunch more information from quantum:
>>>>>>>>>>>>> http://pastie.org/pastes/8053820/text
>>>>>>>>>>>>>
>>>>>>>>>>>>>  If anyone has any ideas I would really appreciate it.
>>>>>>>>>>>>>  Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jun 17, 2013 at 5:07 PM, Samuel Winchenbach <
>>>>>>>>>>>>> swinchen@xxxxxxxxx> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have been stuck on a problem for a couple of days now.  I
>>>>>>>>>>>>>> am using Grizzly on Ubuntu 12.04 LTS.  I can launch vms, create networks,
>>>>>>>>>>>>>> subnets, routers, etc.  The problem is quantum reports that all fors on the
>>>>>>>>>>>>>> public network are "DOWN"  for example:
>>>>>>>>>>>>>> http://pastie.org/pastes/8053283/text
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does anyone have any hints or tips on what might be causing
>>>>>>>>>>>>>> this, or the errors listed below in the quantum logs?  Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> quantum configuration: http://pastie.org/pastes/8053100/text
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> nova configuration: http://pastie.org/pastes/8043800/text
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> quantum logs: http://pastie.org/pastes/8053269/text (this is
>>>>>>>>>>>>>> everything logged during the creation the networks, launching of vm
>>>>>>>>>>>>>> instances, allocating the floating ip and assigning it to a VM)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  _______________________________________________
>>>>>>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>