openstack team mailing list archive

[Eric Day <eday@xxxxxxxxxxxx>]

Well, this all depends on how granular you setup your zones. If you
have a zone hierarchy that goes all the way down to the rack level,
your requests would be sufficient without needing to identify a
particular object (volume).

Within a particular service such as nova, you could allow
'near:volume-001' which would know how to parse the object and lookup
the zone, but you could also get the zone tag for 'volume-001' and
use that for the 'near' value as well.

Again, if you want to hide the topology, you can have a simple zone
ID<->real location service shared between projects and only expose
the IDs via public APIs.

-Eric

On Fri, Feb 11, 2011 at 12:38:18PM -0800, Vishvananda Ishaya wrote:
> I just don't think you can get enough information from zone.  Volumes/Instances might
> need to be literally in the same rack.  The same could be true for lbs/firewalls.  Coarse
> zones might be good for some cases, butI think locality to actual objects is really useful
> in the long run.  We don't necessarily need to provide it for all objects, but we should
> for the ones where it makes sense.  We can fallback to course zones for objects
> that don't need it.
> 
> Vish
> 
> On Feb 11, 2011, at 12:22 PM, Eric Day wrote:
> 
> > If we go the admin-api lookup route, I worry about this as the number
> > of services grow. If nova needs to support launching instances near
> > any other object in an openstack service, we may be looking at: swift
> > objects, glance images, firewalls, load balancers, queue workers,
> > database masters, database replicas, etc. Should nova need to speak all
> > of those APIs to lookup 'near' request tokens? If we can abstract away
> > the actual object and instead have a zone (now suggesting completely
> > opaque so we can keep it accurate without giving away topology), we
> > can simply pass these zones IDs and let the deployment only need to
> > understand how to resolve zones (one service) rather than all services.
> > 
> > For the swift case you mention, the client requesting an instance could
> > ask for current zones for all replicas of a swift object and pass all
> > three zones into the nova request: near-on-of:a,b,c  where a,b,c are
> > the three opaque zones names. Nova could then pick the best zone of the
> > three to launch an instance on without need to talk to swift directly.
> > 
> > -Eric
> > 
> > On Fri, Feb 11, 2011 at 12:01:47PM -0800, Vishvananda Ishaya wrote:
> >> Yes, i think that cross-project should be requests across an admin api.  I don't think high level zone info is enough information for the scheduler
> >> to decide where the instance should go.  Especially with things like swift, where there are three copies that may be in different zones altogether.
> >> 
> >> Vish
> >> On Feb 11, 2011, at 11:55 AM, Eric Day wrote:
> >> 
> >>> Hey Vish!
> >>> 
> >>> On Fri, Feb 11, 2011 at 12:10:14PM -0600, Vishvananda Ishaya wrote:
> >>>>  I think that providers may not wish to expose the internal structure of
> >>>>  their network to the degree that you are suggesting. I prefer the idea of
> >>>>  near other object with an internal lookup into zone.
> >>> 
> >>> Providers can always obscure the naming to hide topology, or
> >>> services can have options to only expose a certain depth of
> >>> the hierarchy (ie, truncate at the 'datacenter level', hide all
> >>> room/rack/switch zone level details). So even though volume-001
> >>> may be in rack12.room2.dc1.dfw.rackspace.com, the API only returns
> >>> dc1.dfw.rackspace.com. We can of course allow for both too when the
> >>> granularity is needed within a project:
> >>> 
> >>> Within nova requests, allow:
> >>> 
> >>> near-volume:volume-001
> >>> 
> >>> But across projects:
> >>> in-zone: dc1.dfw.rackspace.com
> >>> 
> >>> If we don't expose object zones at all, how would you support
> >>> cross-project lookups? If nova gets a request 'near:swift-12345',
> >>> would it be configured to query swift to get the zone via an admin API?
> >>> 
> >>> We could also expose detailed object zones but keep them completely
> >>> opaque, and instead require each service to allow pluggable
> >>> scheduler/zone mapping. For example, a volume may return zone
> >>> '87af1b7c', which internally all services could lookup and translate
> >>> to where that maps in the deployment hierarchy. I was hoping DNS names
> >>> were abstract enough to reduce complexity, but perhaps they are not. :)
> >>> 
> >>> -Eric
> >>> 
> >>>>  On Feb 11, 2011 9:38 AM, "Eric Day" <eday@xxxxxxxxxxxx> wrote:
> >>>>> Hi Sandy,
> >>>>> 
> >>>>> I agree with using tags for full scheduler selection, it's something
> >>>>> I've been pushing for from the start. The request contains any number
> >>>>> of k/v pairs, the services provide any number of k/v pairs, and the
> >>>>> scheduler performs a match (some required, some optional, ...). I see
> >>>>> the URI/zone as one of those tags, not something we need to overload
> >>>>> to contain all of the capabilities. It should only be a hierarchical
> >>>>> "location", which may be geographic location, organizational location
> >>>>> (dept, ...), or some other type (however you decide to construct
> >>>>> your zones).
> >>>>> 
> >>>>> For example, imagine a dynamic del.icio.us tag that allowed for domain
> >>>>> name filtering on bookmarks (give me all bookmarks with tags [book
> >>>>> review] [domain:slashdot.org]). For Nova, this means issuing requests
> >>>>> like "create instance with [GPU] [Fast disk] [zone:dc1.example.com]".
> >>>>> 
> >>>>> The important thing is that this is not a tag specific to a particular
> >>>>> service. For example, Swift would never care or need to understand a
> >>>>> 'GPU' tag, but it can share and understand zone tags.
> >>>>> 
> >>>>> -Eric
> >>>>> 
> >>>>> On Fri, Feb 11, 2011 at 12:40:44PM +0000, Sandy Walsh wrote:
> >>>>>> Heh, hate to be the one to bust up the URI love-fest :)
> >>>>>> 
> >>>>>> The issue I have with a single URI being used as the heuristic for node
> >>>>  selection is that it is very rigid.
> >>>>>> 
> >>>>>> Different business units have different views on the network:
> >>>>>> * Operations may view it as geography/data centers.
> >>>>>> * Consumers may view it as technical ability (gpu's, fast disk, good
> >>>>  inter-server speed, etc)
> >>>>>> * Sales/marketing may view it as the number of martinis they can buy ;)
> >>>>>> 
> >>>>>> Trees become unmanageable/hard to visualize for users beyond a couple
> >>>>  hundred nodes. We are lucky that our geographical/DC-based hierarchy is
> >>>>  relatively flat. This is why I was initially pushing for a tag-based
> >>>>  system for selection (aka Zone/Host Capabilities).
> >>>>>> 
> >>>>>> Consider the way delicio.us works. They manage many millions of URL's
> >>>>  and tags are an effective way to slice & dice your way through the data:
> >>>>>> "Show me all the URL's on [OpenStack] [Python] [Zones] [Scheduler]" ...
> >>>>  blam.
> >>>>>> 
> >>>>>> This is also the way the old Trader services worked:
> >>>>>> "I want a [wax transfer] [color] printer that can has [30ppm] and
> >>>>  [300dpi] on [Floor 2]"
> >>>>>> 
> >>>>>> "Near" simply has to mean the distance in zones from the most-optimal
> >>>>  zones, based on the tags.
> >>>>>> 
> >>>>>> "I want a new instance with [GPU] and [Fast Disk] [Good inter-instance
> >>>>  network speed] [near] [DRW] [DC1]"
> >>>>>> * where "[near]" implies "as close as possible to" in zone distance.
> >>>>>> 
> >>>>>> Personally I don't like overloading the zone name to have a
> >>>>  "meaningful" URI when we can get the same functionality with
> >>>>  Capabilities/Tags already. And we already know we need Capability support
> >>>>  anyway. Especially if it means enforcing a rigid hierarchy.
> >>>>>> 
> >>>>>> $0.02
> >>>>>> 
> >>>>>> -S
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> ________________________________________
> >>>>>> From: openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx
> >>>>  [openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx] on
> >>>>  behalf of Eric Day [eday@xxxxxxxxxxxx]
> >>>>>> Sent: Friday, February 11, 2011 4:30 AM
> >>>>>> To: Justin Santa Barbara
> >>>>>> Cc: openstack@xxxxxxxxxxxxxxxxxxx; Devin Carlen
> >>>>>> Subject: Re: [Openstack] Allowing clients to pass capability requests
> >>>>  through tags?
> >>>>>> 
> >>>>>> The main reason I was proposing full location/zone of objects is to
> >>>>>> allow this type of 'near' scheduling to happen without understanding
> >>>>>> what the actual object is. For example, imagine we want to start an
> >>>>>> instance near a particular swift object. We could query the swift
> >>>>>> object and in the metadata there could be a 'zone' tag (well, three,
> >>>>>> one for each copy). For example:
> >>>>>> 
> >>>>>> get swift-12345: zone=rack12.room2.dc1.dfw.rackspace.com
> >>>>>> 
> >>>>>> I can now use that zone name to:
> >>>>>> 
> >>>>>> create_instance: openstack:near=rack12.room2.dc1.dfw.rackspace.com
> >>>>>> 
> >>>>>> The deployment can decide what 'near' is (perhaps a measure of link
> >>>>>> speed or latency). This way a particular deployment that uses the
> >>>>>> same URI/zone names across projects can account for locality without
> >>>>>> knowing what objects from different services are. If it were just
> >>>>>> 'near=swift-12345', it would need to understand what a swift object
> >>>>>> was and perform that lookup to find out where it is.
> >>>>>> 
> >>>>>> So you can still grab a zone tag from a volume you created:
> >>>>>> 
> >>>>>> get vol-000001: rack4.room2.dc1.dfw.rackspace.com
> >>>>>> 
> >>>>>> and use the zone to launch an instance with:
> >>>>>> 
> >>>>>> create_instance: openstack:near=rack4.room2.dc1.dfw.rackspace.com
> >>>>>> 
> >>>>>> We can also write schedulers/tools for a particular deployment
> >>>>>> that understands the zones to just say 'always prefer in
> >>>>>> dc1.dfw.rackspace.com', because power is cheaper there right now, or
> >>>>>> 'test.dc1.dfw.rackspace.com' because that is my test zone (perhaps
> >>>>>> only enabled for certain accounts in the scheduler too).
> >>>>>> 
> >>>>>> -Eric
> >>>>>> 
> >>>>>> On Thu, Feb 10, 2011 at 03:38:42PM -0800, Justin Santa Barbara wrote:
> >>>>>>> I think the blueprint was largely complementary to the multi-zone
> >>>>  stuff;
> >>>>>>> this is more about how the client _requests_ a particular
> >>>>>>> location/capability through the API. The multi-zone blueprint seems
> >>>>  to be
> >>>>>>> more about how nova would satisfy those requests (in a non-trivial
> >>>>  zone
> >>>>>>> structure.)
> >>>>>>> The root motivator is indeed getting a 'good' connection to a storage
> >>>>>>> volume. I'm thinking of iSCSI SAN storage here, so in my case this
> >>>>>>> probably means the SAN device with the least number of switches in
> >>>>>>> between. There could well be SAN devices in each rack (e.g. Solaris
> >>>>>>> volume nodes), or the devices could even be running on the host
> >>>>  nodes, and
> >>>>>>> I don't believe that zones in the EC2 sense are sufficient here.
> >>>>>>> But I guess that if the zone hierarchy went all the way down to the
> >>>>  rack
> >>>>>>> (or machine), that would work. So I could create a volume and it
> >>>>  would
> >>>>>>> come back with a location of "rack4.room2.dc1.dfw.rackspace.com" and
> >>>>  I
> >>>>>>> could then request allocation of machines in that same rack? Is that
> >>>>  the
> >>>>>>> vision of the nested zones?
> >>>>>>> I do have a concern that long-term if we _only_ use zones, that's
> >>>>  trying
> >>>>>>> to multiplex a lot of information into the zone hierarchy, and we can
> >>>>>>> really only put one attribute in there. I also like the flexibility
> >>>>  of
> >>>>>>> the 'openstack:near=vol-000001' request, because then the cloud can
> >>>>  decide
> >>>>>>> how near to place the instance based on its knowledge of the
> >>>>  topology, and
> >>>>>>> the clients can be oblivious to the storage system and arrangement.
> >>>>  But,
> >>>>>>> my immediate requirement would indeed be satisfied if the zones went
> >>>>  down
> >>>>>>> to the rack/machine level.
> >>>>>>> An alternative way to look at zones and instance-types is that
> >>>>  they're
> >>>>>>> actually just fail-if-not-satisfiable tags of the creation request
> >>>>>>> (openstack:+zone=us-east-1a and openstack:+instancetype=m1.large)
> >>>>  They're
> >>>>>>> only distinguished attributes because AWS doesn't have an
> >>>>>>> extensibility mechanism, which this blueprint would give us.
> >>>>>>> Justin
> >>>>>>> 
> >>>>>>> On Thu, Feb 10, 2011 at 3:12 PM, Devin Carlen <devcamcar@xxxxxx>
> >>>>  wrote:
> >>>>>>> 
> >>>>>>> I haven't totally digested this blueprint yet but it seems like there
> >>>>  is
> >>>>>>> some overlap with what is being discussed with the multi zone
> >>>>  metadata
> >>>>>>> stuff. One approach might be to handle this awt the scheduler level
> >>>>>>> though and try to ensure things are always in the same zone when
> >>>>>>> appropriate.
> >>>>>>> I think the bigger question you raise is how to request local volumes
> >>>>>>> when possible, yes?
> >>>>>>> 
> >>>>>>> Devin
> >>>>>>> On Feb 10, 2011, at 3:37 PM, Justin Santa Barbara
> >>>>  <justin@xxxxxxxxxxxx>
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> Does anyone have any thoughts/objections on the blueprint I posted
> >>>>  for
> >>>>>>> allowing clients to pass capability-requests through tags? I'm
> >>>>>>> planning on starting implementation soon, so if people think this is
> >>>>  a
> >>>>>>> bad idea I'd rather know before I start coding!
> >>>>>>> Blueprint:
> >>>>  https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
> >>>>>>> Wiki:
> >>>>  https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
> >>>>>>> And a quick TLDR:
> >>>>>>> API clients need a way to request e.g. placement of machines near
> >>>>  each
> >>>>>>> other / near volumes, or that a volume be created with a particular
> >>>>>>> RAID level, or that a machine be created in a HIPAA compliant
> >>>>>>> environment. (This is complementary to the work on hierarchical zones
> >>>>>>> & URL naming, I believe)
> >>>>>>> I propose using the instance tags for this, e.g. specifying
> >>>>>>> openstack:near=vol-000001 when creating an instance to request
> >>>>>>> locating the instance 'close to' that volume.
> >>>>>>> By default these requests would be best-effort and
> >>>>  ignored-if-unknown;
> >>>>>>> if the client wants to specify that something is required and should
> >>>>>>> fail if not understood or not satisfiable, they could use a "+" e.g.
> >>>>>>> openstack:+location=*.dc1.north.rackspace.com
> >>>>>>> Controversially (?), this would not be supported for clients using
> >>>>  the
> >>>>>>> AWS API, because tags can only be specified once the instance has
> >>>>>>> already been created.
> >>>>>>> Feedback appreciated!
> >>>>>>> Justin
> >>>>>>> 
> >>>>>>> _______________________________________________
> >>>>>>> Mailing list: https://launchpad.net/~openstack
> >>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>>>> Unsubscribe : https://launchpad.net/~openstack
> >>>>>>> More help : https://help.launchpad.net/ListHelp
> >>>>>> 
> >>>>>>> _______________________________________________
> >>>>>>> Mailing list: https://launchpad.net/~openstack
> >>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>>>> Unsubscribe : https://launchpad.net/~openstack
> >>>>>>> More help : https://help.launchpad.net/ListHelp
> >>>>>> 
> >>>>>> 
> >>>>>> _______________________________________________
> >>>>>> Mailing list: https://launchpad.net/~openstack
> >>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>>> Unsubscribe : https://launchpad.net/~openstack
> >>>>>> More help : https://help.launchpad.net/ListHelp
> >>>>>> 
> >>>>>> 
> >>>>>> Confidentiality Notice: This e-mail message (including any attached or
> >>>>>> embedded documents) is intended for the exclusive and confidential use
> >>>>  of the
> >>>>>> individual or entity to which this message is addressed, and unless
> >>>>  otherwise
> >>>>>> expressly indicated, is confidential and privileged information of
> >>>>  Rackspace.
> >>>>>> Any dissemination, distribution or copying of the enclosed material is
> >>>>  prohibited.
> >>>>>> If you receive this transmission in error, please notify us immediately
> >>>>  by e-mail
> >>>>>> at abuse@xxxxxxxxxxxxx, and delete the original message.
> >>>>>> Your cooperation is appreciated.
> >>>>> 
> >>>>> _______________________________________________
> >>>>> Mailing list: https://launchpad.net/~openstack
> >>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>> Unsubscribe : https://launchpad.net/~openstack
> >>>>> More help : https://help.launchpad.net/ListHelp