openstack team mailing list archive

["Sumit Naiksatam (snaiksat)" <snaiksat@xxxxxxxxx>]

Hi Dan,

 

Thanks for your prompt response on this.

 

As I understand from your explanation, the knowledge of the "edge
binding" resides with Nova, and in the case of my example that "edge
binding" is "br100". Is that correct? If so, how does Nova acquire this
knowledge? I ask because, my expectation was that "br100" (i.e. the
Linux bridge) is created and managed by Quantum (and/or the plugin
underneath). 

 

Also, if we move away from using the interface type "bridge" (in the
context of libvirt) to "ethernet" as you suggest, would that be the only
"type" that would be supported? If so, that will not work with
VN-Tag/VEPA since that requires a type "direct". If we are indeed not
limiting to just the "ethernet" type, then how does Nova know which type
to use (assuming that the knowledge of the underlying network
connectivity resides with Quantum)?

 

It's good to hear that the work on Quantum and Nova refactoring is going
well. While that is in progress, it might be helpful to get some
skeletal flowcharts (or even textual descriptions) describing the flows
(with the interaction between Nova and Quantum fleshed out).

 

Thanks,

~Sumit. 

 

From: Dan Wendlandt [mailto:dan@xxxxxxxxxx] 
Sent: Monday, May 30, 2011 11:15 PM
To: Sumit Naiksatam (snaiksat)
Cc: openstack@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Openstack] [NetStack] "Attach resource to port" semantics

 

Hi Sumit,

 

This is a great question to be asking, as it gets to the heart of how
Quantum will work with Nova (or any any other OpenStack service that
plugs into the network).  As you suggest, our goal here is to try and
come up with a clean separation between the responsibilities of Nova and
the responsibilities of Quantum. 

 

The way I think about it, the act of "attaching a VM NIC to the network"
actually happens at two distinct layers: logical (managed by Quantum)
and, for lack of a better name, non-logical (managed by Nova).  

 

Non-Logical: This occurs when the Nova spins up a VM and plugs the VM
NIC into the vswitch.  For example, with Nova using libvirt, this
amounts to creating a linux device representing the VM NIC (e.g., tap0)
and associating it with a vswitch (e.g., br0).  With Quantum, this act
alone should not actually cause the VM to be able to send and receive
packets, unless the logical connectivity of the VM NIC has already been
setup (see below).  Instead, it is just establishing a channel by which
vswitch controlled by Quantum can pass packets to and from the VM NIC
once the proper logical connectivity has been specified.  With Quantum,
Nova must have an "interface-id" for each VM NIC.  When Nova spins up a
VM and creates a new port on a vswitch, it communicates to Quantum the
interface-id associated with that port on the vswitch.  We refer to as
the "edge binding" (see slides #7 and #14 in
http://www.slideshare.net/danwent/quantum-diablo-summary).  

 

Logical: This type of "plugging" is done by the tenant via the Quantum
API.  The Quantum API exposes the concept of networks and ports on those
networks to model this logical connectivity.  Interface-ids can be
"attached" to a port, indicating the network connectivity that should be
seen by the VM NIC with the associated interface-id (e.g., two VMs
attached to ports on the same logical network can connect directly at
L2, others cannot).  A simple implementation of a logical network could
be to put implement each logical network as a different VLAN within a
single physical L2 domain, with a Quantum plugin managing the vswitch to
put each port on the appropriate VLAN (see slide #15).  

 

The motivation for this design is that Nova does not need to be involved
when the logical connectivity of a VM changes. Nova just needs to worry
about making sure the vswitch (or physical switch if using VNTag/VEPA)
managed by the Quantum plugin has a channel to pass bits to and from
each VM NIC, and knows the identify of each of those channels (the edge
binding).  It is up to the Quantum plugin managing the vswitch to make
sure the right logical connectivity (or lack thereof) is implemented.  

 

Along with folks from the Nova team, we've been discussing the details
of how this model applies to different Nova "virt" layer
implementations, including libvirt, XenServer, VMware, etc.  As you
suggest, the libvirt model where Nova specifies only a bridge to connect
to is less flexible, so we're looking at using an <interface> elements
of type "ethernet" instead of type "bridge" as an alternative (Ryu did
some work on this in a branch).  

 

Implementation work on an experimental version of Quantum has been
progressing well, so in another week or two I would expect that we can
have a full stack of code that people can play with using a slightly
tweaked version of Nova.  Having this available will hopefully make the
above description of a logical + non-logical plugging step more concrete
and will serve as a good vehicle for feedback from the community.  Happy
to chat more about this tomorrow at the Networking meeting on IRC.

 

dan

 

 

 

On Mon, May 30, 2011 at 9:01 PM, Sumit Naiksatam (snaiksat)
<snaiksat@xxxxxxxxx> wrote:

Hi,

 

I would like to request clarity on the semantics of the "attach resource
to port" operation proposed in Quantum in the context of spawning a VM.
Let's discuss this with respect to a specific example. As of today,
nova-compute generates the domain configuration (libvirt.xml), including
the network interface configuration as below -

 

        <interface type='bridge'>

            <source bridge='br100'/>

            <mac address='02:16:3e:78:4d:84'/>

            <!--   <model type='virtio'/>  CANT RUN virtio network right
now -->

            <filterref filter="nova-instance-instance-00000001">

                <parameter name="IP" value="10.1.0.2" />

                <parameter name="DHCPSERVER" value="10.1.0.1" />

                <parameter name="PROJNET" value="10.1.0.0" />

                <parameter name="PROJMASK" value="255.255.255.128" />

            </filterref>

        </interface>

 

By virtue of specifying "br100" as the bridge to connect to,
nova-compute is configuring the attachment from the virtual NIC (on the
VM) to the network, and the "attachment" operation is (implicitly)
realized when the VM is spawned (and running).

 

With the introduction of the "attach resource to port" operation, we are
introducing an explicit step in the process of the VM spawning. It
implies that prior to invoking the "attach resource to port" operation,
nova-compute would not know which bridge to connect to (i.e. it would
not know the identity of "br100"). So does nova-compute first spawn the
VM without the network interface configuration? If that is the case, it
brings us to the next question, which is, what does the "attach resource
to port" operation actually do? There are at least two possibilities
here -

 

(1) The operation merely results in the identification (and, if
required, the configuration) of "br100" as the bridge to connect the
VM's network interface to. If that is the case, we need a way to
propagate this information (i.e. "br100") back to the entity which
invoked the "attach resource to port" operation. Subsequently, that same
entity will have to perform another action to craft the VM's network
interface configuration (with the acquired knowledge of "br100") and
also add this network interface to the configuration of the VM (by
making appropriate libvirt calls). Adding this network interface
configuration could be done, both, prior to starting the VM, or even
while the VM is running.

 

OR 

 

(2) The operation not only identifies/configuration of "br100", but also
crafts the VM's network interface configuration, and adds that interface
to the VM (by making appropriate libvirt calls). Note that this second
option amounts to Quantum controlling a resource on the VM (which
ideally should be outside the scope of Quantum).

 

OR, is it something else?

 

Kindly let me know what you thoughts are on this. As projected above,
these semantics have implications on how nova-compute will deal with
spawning VMs (and it might be different from what it is today), and
hence it might be important to address this issue sooner.

 

Thanks,

~Sumit.

PS: As I write this, it occurs to me that it's not very clear as to what
the "create port" operation would do in the above case as well, since no
actual port is created on the bridge. Any thoughts?

 


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp




-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt 
Nicira Networks, Inc. 
www.nicira.com | www.openvswitch.org
Sr. Product Manager 
cell: 650-906-2650
~~~~~~~~~~~~~~~~~~~~~~~~~~~