openstack team mailing list archive

[Dan Wendlandt <dan@xxxxxxxxxx>]

Salvatore is right here.  The key point is that the limits of standard L2 +
VLANs aren't just about the number of bits in the VLAN tag: even if you buy
gear that supports more VLANs in hardware (including Q-in-Q), more important
are the issues with broadcast scalability, MAC table size, inability to L2
to leverage multiple active paths, inability to fail-over quickly to
alternate paths, and inability to span an L3 backbone that joins disparate
datacenters.  We built open vswitch to have L2-in-L3 tunneling capabilities
exactly because of these limitations and will be working with the openstack
team to make sure open vswitch can be used in openstack whether you prefer
to us VLAN or more advanced tunneling options.

Dan


On Fri, Feb 4, 2011 at 2:02 AM, Salvatore Orlando <
Salvatore.Orlando@xxxxxxxxxxxxx> wrote:

> Hi,
>
> I agree with Aimon and Patrick that a VLAN per customer would be a good fit
> only for small deployments.
> Moreover, if I'm not wrong, datacentres are typically organized in small
> layer-2 PoD in the Access layer interconnected by L3 routers (I'm referring
> to the Cisco Data Center Infrastructure design guide); to the best of my
> knowledge L3 routers do not forward VLAN tags. In the access layer, PoDs are
> typically small in order to limit the impact of L2 broadcast traffic, and, I
> believe, to make spanning tree working more efficiently.
> One thing we might consider in order to overcome this limit  is to employ
> L3 tunnelling protocols for VM traffic; in my opinion this might have
> several benefits:
> - No need to perform any configuration, such as setting ports on trunking
> mode, on physical network switches: they would not see VLAN-tagged frames,
> but only IP traffic;
> - MAC forwarding tables on physical network switches will hardly be
> saturated, even when using 'cheap' switches, as the only MACs they would see
> are the endpoints of the tunnels;
> - L2 broadcast domain will still be contained in size, even if distributed
> over several PoD in the same data centre and potentially over several data
> centres.
>
> One (potentially big) issue with tunnelling protocols is the topology of
> the network created by the tunnel themselves; it might define traffic
> bottlenecks, or even contain loops.
>
> I have been using Open vSwitch for a few months now, which has been
> mentioned on several blueprints in the nova project. It supports GRE tunnels
> and VLAN tagging (though AFAIK only 802.1Q); it could potentially be a good
> fit for implementing multi-tenant support for large deployments.
> Open vSwitch currently runs on both Xen and KVM hypervisors; however it is
> not supported in ESX, and (If I'm not wrong) in Hyper-V.
>
> Regards,
> Salvatore
>
> -----Original Message-----
> From: openstack-bounces+salvatore.orlando=eu.citrix.com@
> lists.launchpad.net [mailto:openstack-bounces+salvatore.orlando<openstack-bounces%2Bsalvatore.orlando>
> =eu.citrix.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Aimon Bustardo
> Sent: 04 February 2011 07:14
> To: Patrick Ancillotti
> Cc: <openstack@xxxxxxxxxxxxxxxxxxx>
> Subject: Re: [Openstack] Pondering multi-tenant needs in nova.
>
> Hi Patrick,
>
> that would be great if you would go into details. I am most interested in
> this as it directly effects our cloud platforms adoption of OpenStack and
> the subsequent networking blueprint we have designed (trying to get a hold
> of the networking blueprint Ewan is leading also).
>
>
> Thank you,
>
>
> Aimon
>
> On 2/3/11 10:18 PM, Patrick Ancillotti wrote:
> > Aimon,
> >
> > You're correct of course for simply defining a customer per VLAN as
> realistically we wouldn't hit 16M+ customers in any regional area as it
> stands today ;) but there are other issues with QinQ at large scale,
> especially with Layer 2 domains of the size that we're envisaging in the
> long run... many of which we can go into detail if you want?
> >
> > For those interested in QinQ : http://en.wikipedia.org/wiki/802.1QinQ
> >
> > Thanks,
> > Patrick
> >
> > On 3 Feb 2011, at 23:45, Aimon Bustardo wrote:
> >
> >> Hi Patrick, by using 802.1QiQ aren't we increasing the total VLAN
> >> count to 4096*4096 (16277216) possible VLANS, each with the
> >> possibility of using the same /8 network? In which case he in table
> >> storage of MACs is the limiting factor.
> >>
> >>
> >> Aimon
> >>
> >> On 2/3/11 8:03 PM, Patrick Ancillotti wrote:
> >>> Hey Guys,
> >>>
> >>> I think Paul may have gotten a bit mixed up between VLAN and CAM tables
> on switches. The VLAN part of an ethernet frame is 12 bits (0 - 4095) which
> limits it accordingly. CAM tables however are a limit within switching gear
> that lists the MAC addresses and their respective source and destination
> ports. The lower end Cisco switches like the 2960 class switches have a
> limit of around 8000 MAC addresses, whereas others higher in the stack such
> as the 4948E class have limits upwards of 55000 MAC addresses, although some
> vendors have CAM tables in the hundreds of thousands for even their mid
> range switches.
> >>>
> >>> That said, VLAN's are extremely limited, even with QinQ (VLANs within
> VLANs) when you're talking about 50 000 customers, or even 500 000 customers
> in the same layer 2 domain, and in many cases using smaller layer 2 domains
> creates unfortunately small service areas for capacity.
> >>>
> >>> For more info:
> >>> http://en.wikipedia.org/wiki/Virtual_LAN
> >>> http://en.wikipedia.org/wiki/CAM_Table
> >>>
> >>> Thanks,
> >>> Patrick
> >>>
> >>> On 3 Feb 2011, at 07:47, Paul Voccio wrote:
> >>>
> >>>> Diego,
> >>>>
> >>>> Due to our networking topology, having a vlan per customer isn't
> >>>> really feasible. Most switches are limited at 4k or 8k or even 32k.
> >>>> With more customers than these switches can reasonably accommodate,
> >>>> having a single vlan per customer either limits the portability
> >>>> within a cloud or limits the scale at which you can build your
> >>>> cloud. Open vSwitch will alleviate some of this pain, but until we
> >>>> get it in XenServer, we're somewhat stuck on flat networking.
> >>>>
> >>>> Paul
> >>>>
> >>>> On 2/3/11 4:20 AM, "Diego Parrilla Santamaría"
> >>>> <diego.parrilla.santamaria@xxxxxxxxx> wrote:
> >>>>
> >>>>> Hi Monsyne,
> >>>>>
> >>>>> it's a very interesting topic and I'm curious about the reason why
> >>>>> you are using the Flat Networking set up. From the conversations
> >>>>> in other threads it seems the Service Providers prefer different
> >>>>> networking
> >>>>> approaches: VLAN oriented basically.
> >>>>>
> >>>>> Regards
> >>>>> Diego
> >>>>>
> >>>>> -
> >>>>> Diego Parrilla
> >>>>> nubeblog.com | nubeblog@xxxxxxxxxxxx | twitter.com/nubeblog
> >>>>> +34 649 94 43 29
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 3, 2011 at 2:37 AM, Monsyne Dragon
> >>>>> <mdragon@xxxxxxxxxxxxx>
> >>>>> wrote:
> >>>>>> I am sorting out some possible implementations for the
> >>>>>> multi-tenant-accounting blueprint, and the related
> >>>>>> system-usage-records bp, and I just wanted to run this by anyone
> >>>>>> interested in such matters.
> >>>>>>
> >>>>>> Basically, for multitenant purposes we need to introduce the
> >>>>>> concept of an 'account' in nova, representing a customer,  that
> >>>>>> basically acts as a label for a group of resources (instances,
> >>>>>> etc), and for access control (i.e customer a cannot mess w/
> >>>>>> customer b's stuff)
> >>>>>>
> >>>>>> There was some confusion on how best to implement this, in
> >>>>>> relation to nova's project concept.  Projects are kind of like
> >>>>>> what we want an account to be, but there are some associations
> >>>>>> (like one project per network) which are not valid for our flat
> >>>>>> networking setup.  I am kind of straw-polling on which is better
> >>>>>> here:
> >>>>>>
> >>>>>> The options are:
> >>>>>> 1) Create a new 'account' concept in nova,  with an account
> >>>>>> basically being a subgroup of a project (providers would use a
> >>>>>> single, default project, with additional projects added if needed
> >>>>>> for separate brands, or resellers, etc), add in access control
> >>>>>> per account as well as project, and make sure apis/auth specify
> >>>>>> account appropriately,  have some way for a default account to
> >>>>>> used (per project) so account doesn't get in the way for
> >>>>>> non-multitenant users.
> >>>>>>
> >>>>>> 2) having account == nova's "project", and changing the network
> >>>>>> associations, etc so projects can support our model (as well as
> >>>>>> current models).  Support for associating accounts (projects)
> >>>>>> together for resellers, etc would either be delegated outside of
> >>>>>> nova or added later (it's not a current requirement).
> >>>>>>
> >>>>>> In either case, accounts would be identified by name, which would
> >>>>>> be an opaque string an outside system/person would assign, and
> >>>>>> could structure to their needs (ie. for associating accounts with
> >>>>>> common prefixes, etc)
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> --
> >>>>>>  -Monsyne Dragon
> >>>>>>  work:         210-312-4190
> >>>>>>  mobile        210-441-0965
> >>>>>>  google voice: 210-338-0336
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Confidentiality Notice: This e-mail message (including any
> >>>>>> attached or embedded documents) is intended for the exclusive and
> >>>>>> confidential use of the individual or entity to which this
> >>>>>> message is addressed, and unless otherwise expressly indicated,
> >>>>>> is confidential and privileged information of Rackspace.
> >>>>>> Any dissemination, distribution or copying of the enclosed
> >>>>>> material is prohibited.
> >>>>>> If you receive this transmission in error, please notify us
> >>>>>> immediately by e-mail at abuse@xxxxxxxxxxxxx, and delete the
> >>>>>> original message.
> >>>>>> Your cooperation is appreciated.
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>>> Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>>>>> More help   : https://help.launchpad.net/ListHelp
> >>>>>>
> >>>>> _______________________________________________
> >>>>> Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>> Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>>>> More help   : https://help.launchpad.net/ListHelp
> >>>>
> >>>> Confidentiality Notice: This e-mail message (including any attached
> >>>> or embedded documents) is intended for the exclusive and
> >>>> confidential use of the individual or entity to which this message
> >>>> is addressed, and unless otherwise expressly indicated, is
> confidential and privileged information of Rackspace.
> >>>> Any dissemination, distribution or copying of the enclosed material is
> prohibited.
> >>>> If you receive this transmission in error, please notify us
> >>>> immediately by e-mail at abuse@xxxxxxxxxxxxx, and delete the original
> message.
> >>>> Your cooperation is appreciated.
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >>>> Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>>> More help   : https://help.launchpad.net/ListHelp
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >>> More help   : https://help.launchpad.net/ListHelp
> >> --
> >> Aimon Bustardo | Morph Labs | +1.310.625.0608 (mobile) |
> >> +1.310.437.4898 (office) | www.morphlabs.com | aimon@xxxxxx
> >>
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> >> More help   : https://help.launchpad.net/ListHelp
> >
> >
> > Confidentiality Notice: This e-mail message (including any attached or
> > embedded documents) is intended for the exclusive and confidential use
> > of the individual or entity to which this message is addressed, and
> > unless otherwise expressly indicated, is confidential and privileged
> information of Rackspace.
> > Any dissemination, distribution or copying of the enclosed material is
> prohibited.
> > If you receive this transmission in error, please notify us
> > immediately by e-mail at abuse@xxxxxxxxxxxxx, and delete the original
> message.
> > Your cooperation is appreciated.
> >
>
> --
> Aimon Bustardo | Morph Labs | +1.310.625.0608 (mobile) | +1.310.437.4898
> (office) | www.morphlabs.com | aimon@xxxxxx
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> More help   : https://help.launchpad.net/ListHelp
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
> More help   : https://help.launchpad.net/ListHelp
>



-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira Networks, Inc.
www.nicira.com | www.openvswitch.org
Sr. Product Manager
cell: 650-906-2650
~~~~~~~~~~~~~~~~~~~~~~~~~~~