openstack team mailing list archive

[Vishvananda Ishaya <vishvananda@xxxxxxxxx>]

I just don't think you can get enough information from zone.  Volumes/Instances might
need to be literally in the same rack.  The same could be true for lbs/firewalls.  Coarse
zones might be good for some cases, butI think locality to actual objects is really useful
in the long run.  We don't necessarily need to provide it for all objects, but we should
for the ones where it makes sense.  We can fallback to course zones for objects
that don't need it.

Vish

On Feb 11, 2011, at 12:22 PM, Eric Day wrote:

> If we go the admin-api lookup route, I worry about this as the number
> of services grow. If nova needs to support launching instances near
> any other object in an openstack service, we may be looking at: swift
> objects, glance images, firewalls, load balancers, queue workers,
> database masters, database replicas, etc. Should nova need to speak all
> of those APIs to lookup 'near' request tokens? If we can abstract away
> the actual object and instead have a zone (now suggesting completely
> opaque so we can keep it accurate without giving away topology), we
> can simply pass these zones IDs and let the deployment only need to
> understand how to resolve zones (one service) rather than all services.
> 
> For the swift case you mention, the client requesting an instance could
> ask for current zones for all replicas of a swift object and pass all
> three zones into the nova request: near-on-of:a,b,c  where a,b,c are
> the three opaque zones names. Nova could then pick the best zone of the
> three to launch an instance on without need to talk to swift directly.
> 
> -Eric
> 
> On Fri, Feb 11, 2011 at 12:01:47PM -0800, Vishvananda Ishaya wrote:
>> Yes, i think that cross-project should be requests across an admin api.  I don't think high level zone info is enough information for the scheduler
>> to decide where the instance should go.  Especially with things like swift, where there are three copies that may be in different zones altogether.
>> 
>> Vish
>> On Feb 11, 2011, at 11:55 AM, Eric Day wrote:
>> 
>>> Hey Vish!
>>> 
>>> On Fri, Feb 11, 2011 at 12:10:14PM -0600, Vishvananda Ishaya wrote:
>>>>  I think that providers may not wish to expose the internal structure of
>>>>  their network to the degree that you are suggesting. I prefer the idea of
>>>>  near other object with an internal lookup into zone.
>>> 
>>> Providers can always obscure the naming to hide topology, or
>>> services can have options to only expose a certain depth of
>>> the hierarchy (ie, truncate at the 'datacenter level', hide all
>>> room/rack/switch zone level details). So even though volume-001
>>> may be in rack12.room2.dc1.dfw.rackspace.com, the API only returns
>>> dc1.dfw.rackspace.com. We can of course allow for both too when the
>>> granularity is needed within a project:
>>> 
>>> Within nova requests, allow:
>>> 
>>> near-volume:volume-001
>>> 
>>> But across projects:
>>> in-zone: dc1.dfw.rackspace.com
>>> 
>>> If we don't expose object zones at all, how would you support
>>> cross-project lookups? If nova gets a request 'near:swift-12345',
>>> would it be configured to query swift to get the zone via an admin API?
>>> 
>>> We could also expose detailed object zones but keep them completely
>>> opaque, and instead require each service to allow pluggable
>>> scheduler/zone mapping. For example, a volume may return zone
>>> '87af1b7c', which internally all services could lookup and translate
>>> to where that maps in the deployment hierarchy. I was hoping DNS names
>>> were abstract enough to reduce complexity, but perhaps they are not. :)
>>> 
>>> -Eric
>>> 
>>>>  On Feb 11, 2011 9:38 AM, "Eric Day" <eday@xxxxxxxxxxxx> wrote:
>>>>> Hi Sandy,
>>>>> 
>>>>> I agree with using tags for full scheduler selection, it's something
>>>>> I've been pushing for from the start. The request contains any number
>>>>> of k/v pairs, the services provide any number of k/v pairs, and the
>>>>> scheduler performs a match (some required, some optional, ...). I see
>>>>> the URI/zone as one of those tags, not something we need to overload
>>>>> to contain all of the capabilities. It should only be a hierarchical
>>>>> "location", which may be geographic location, organizational location
>>>>> (dept, ...), or some other type (however you decide to construct
>>>>> your zones).
>>>>> 
>>>>> For example, imagine a dynamic del.icio.us tag that allowed for domain
>>>>> name filtering on bookmarks (give me all bookmarks with tags [book
>>>>> review] [domain:slashdot.org]). For Nova, this means issuing requests
>>>>> like "create instance with [GPU] [Fast disk] [zone:dc1.example.com]".
>>>>> 
>>>>> The important thing is that this is not a tag specific to a particular
>>>>> service. For example, Swift would never care or need to understand a
>>>>> 'GPU' tag, but it can share and understand zone tags.
>>>>> 
>>>>> -Eric
>>>>> 
>>>>> On Fri, Feb 11, 2011 at 12:40:44PM +0000, Sandy Walsh wrote:
>>>>>> Heh, hate to be the one to bust up the URI love-fest :)
>>>>>> 
>>>>>> The issue I have with a single URI being used as the heuristic for node
>>>>  selection is that it is very rigid.
>>>>>> 
>>>>>> Different business units have different views on the network:
>>>>>> * Operations may view it as geography/data centers.
>>>>>> * Consumers may view it as technical ability (gpu's, fast disk, good
>>>>  inter-server speed, etc)
>>>>>> * Sales/marketing may view it as the number of martinis they can buy ;)
>>>>>> 
>>>>>> Trees become unmanageable/hard to visualize for users beyond a couple
>>>>  hundred nodes. We are lucky that our geographical/DC-based hierarchy is
>>>>  relatively flat. This is why I was initially pushing for a tag-based
>>>>  system for selection (aka Zone/Host Capabilities).
>>>>>> 
>>>>>> Consider the way delicio.us works. They manage many millions of URL's
>>>>  and tags are an effective way to slice & dice your way through the data:
>>>>>> "Show me all the URL's on [OpenStack] [Python] [Zones] [Scheduler]" ...
>>>>  blam.
>>>>>> 
>>>>>> This is also the way the old Trader services worked:
>>>>>> "I want a [wax transfer] [color] printer that can has [30ppm] and
>>>>  [300dpi] on [Floor 2]"
>>>>>> 
>>>>>> "Near" simply has to mean the distance in zones from the most-optimal
>>>>  zones, based on the tags.
>>>>>> 
>>>>>> "I want a new instance with [GPU] and [Fast Disk] [Good inter-instance
>>>>  network speed] [near] [DRW] [DC1]"
>>>>>> * where "[near]" implies "as close as possible to" in zone distance.
>>>>>> 
>>>>>> Personally I don't like overloading the zone name to have a
>>>>  "meaningful" URI when we can get the same functionality with
>>>>  Capabilities/Tags already. And we already know we need Capability support
>>>>  anyway. Especially if it means enforcing a rigid hierarchy.
>>>>>> 
>>>>>> $0.02
>>>>>> 
>>>>>> -S
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ________________________________________
>>>>>> From: openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx
>>>>  [openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx] on
>>>>  behalf of Eric Day [eday@xxxxxxxxxxxx]
>>>>>> Sent: Friday, February 11, 2011 4:30 AM
>>>>>> To: Justin Santa Barbara
>>>>>> Cc: openstack@xxxxxxxxxxxxxxxxxxx; Devin Carlen
>>>>>> Subject: Re: [Openstack] Allowing clients to pass capability requests
>>>>  through tags?
>>>>>> 
>>>>>> The main reason I was proposing full location/zone of objects is to
>>>>>> allow this type of 'near' scheduling to happen without understanding
>>>>>> what the actual object is. For example, imagine we want to start an
>>>>>> instance near a particular swift object. We could query the swift
>>>>>> object and in the metadata there could be a 'zone' tag (well, three,
>>>>>> one for each copy). For example:
>>>>>> 
>>>>>> get swift-12345: zone=rack12.room2.dc1.dfw.rackspace.com
>>>>>> 
>>>>>> I can now use that zone name to:
>>>>>> 
>>>>>> create_instance: openstack:near=rack12.room2.dc1.dfw.rackspace.com
>>>>>> 
>>>>>> The deployment can decide what 'near' is (perhaps a measure of link
>>>>>> speed or latency). This way a particular deployment that uses the
>>>>>> same URI/zone names across projects can account for locality without
>>>>>> knowing what objects from different services are. If it were just
>>>>>> 'near=swift-12345', it would need to understand what a swift object
>>>>>> was and perform that lookup to find out where it is.
>>>>>> 
>>>>>> So you can still grab a zone tag from a volume you created:
>>>>>> 
>>>>>> get vol-000001: rack4.room2.dc1.dfw.rackspace.com
>>>>>> 
>>>>>> and use the zone to launch an instance with:
>>>>>> 
>>>>>> create_instance: openstack:near=rack4.room2.dc1.dfw.rackspace.com
>>>>>> 
>>>>>> We can also write schedulers/tools for a particular deployment
>>>>>> that understands the zones to just say 'always prefer in
>>>>>> dc1.dfw.rackspace.com', because power is cheaper there right now, or
>>>>>> 'test.dc1.dfw.rackspace.com' because that is my test zone (perhaps
>>>>>> only enabled for certain accounts in the scheduler too).
>>>>>> 
>>>>>> -Eric
>>>>>> 
>>>>>> On Thu, Feb 10, 2011 at 03:38:42PM -0800, Justin Santa Barbara wrote:
>>>>>>> I think the blueprint was largely complementary to the multi-zone
>>>>  stuff;
>>>>>>> this is more about how the client _requests_ a particular
>>>>>>> location/capability through the API. The multi-zone blueprint seems
>>>>  to be
>>>>>>> more about how nova would satisfy those requests (in a non-trivial
>>>>  zone
>>>>>>> structure.)
>>>>>>> The root motivator is indeed getting a 'good' connection to a storage
>>>>>>> volume. I'm thinking of iSCSI SAN storage here, so in my case this
>>>>>>> probably means the SAN device with the least number of switches in
>>>>>>> between. There could well be SAN devices in each rack (e.g. Solaris
>>>>>>> volume nodes), or the devices could even be running on the host
>>>>  nodes, and
>>>>>>> I don't believe that zones in the EC2 sense are sufficient here.
>>>>>>> But I guess that if the zone hierarchy went all the way down to the
>>>>  rack
>>>>>>> (or machine), that would work. So I could create a volume and it
>>>>  would
>>>>>>> come back with a location of "rack4.room2.dc1.dfw.rackspace.com" and
>>>>  I
>>>>>>> could then request allocation of machines in that same rack? Is that
>>>>  the
>>>>>>> vision of the nested zones?
>>>>>>> I do have a concern that long-term if we _only_ use zones, that's
>>>>  trying
>>>>>>> to multiplex a lot of information into the zone hierarchy, and we can
>>>>>>> really only put one attribute in there. I also like the flexibility
>>>>  of
>>>>>>> the 'openstack:near=vol-000001' request, because then the cloud can
>>>>  decide
>>>>>>> how near to place the instance based on its knowledge of the
>>>>  topology, and
>>>>>>> the clients can be oblivious to the storage system and arrangement.
>>>>  But,
>>>>>>> my immediate requirement would indeed be satisfied if the zones went
>>>>  down
>>>>>>> to the rack/machine level.
>>>>>>> An alternative way to look at zones and instance-types is that
>>>>  they're
>>>>>>> actually just fail-if-not-satisfiable tags of the creation request
>>>>>>> (openstack:+zone=us-east-1a and openstack:+instancetype=m1.large)
>>>>  They're
>>>>>>> only distinguished attributes because AWS doesn't have an
>>>>>>> extensibility mechanism, which this blueprint would give us.
>>>>>>> Justin
>>>>>>> 
>>>>>>> On Thu, Feb 10, 2011 at 3:12 PM, Devin Carlen <devcamcar@xxxxxx>
>>>>  wrote:
>>>>>>> 
>>>>>>> I haven't totally digested this blueprint yet but it seems like there
>>>>  is
>>>>>>> some overlap with what is being discussed with the multi zone
>>>>  metadata
>>>>>>> stuff. One approach might be to handle this awt the scheduler level
>>>>>>> though and try to ensure things are always in the same zone when
>>>>>>> appropriate.
>>>>>>> I think the bigger question you raise is how to request local volumes
>>>>>>> when possible, yes?
>>>>>>> 
>>>>>>> Devin
>>>>>>> On Feb 10, 2011, at 3:37 PM, Justin Santa Barbara
>>>>  <justin@xxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> Does anyone have any thoughts/objections on the blueprint I posted
>>>>  for
>>>>>>> allowing clients to pass capability-requests through tags? I'm
>>>>>>> planning on starting implementation soon, so if people think this is
>>>>  a
>>>>>>> bad idea I'd rather know before I start coding!
>>>>>>> Blueprint:
>>>>  https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
>>>>>>> Wiki:
>>>>  https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
>>>>>>> And a quick TLDR:
>>>>>>> API clients need a way to request e.g. placement of machines near
>>>>  each
>>>>>>> other / near volumes, or that a volume be created with a particular
>>>>>>> RAID level, or that a machine be created in a HIPAA compliant
>>>>>>> environment. (This is complementary to the work on hierarchical zones
>>>>>>> & URL naming, I believe)
>>>>>>> I propose using the instance tags for this, e.g. specifying
>>>>>>> openstack:near=vol-000001 when creating an instance to request
>>>>>>> locating the instance 'close to' that volume.
>>>>>>> By default these requests would be best-effort and
>>>>  ignored-if-unknown;
>>>>>>> if the client wants to specify that something is required and should
>>>>>>> fail if not understood or not satisfiable, they could use a "+" e.g.
>>>>>>> openstack:+location=*.dc1.north.rackspace.com
>>>>>>> Controversially (?), this would not be supported for clients using
>>>>  the
>>>>>>> AWS API, because tags can only be specified once the instance has
>>>>>>> already been created.
>>>>>>> Feedback appreciated!
>>>>>>> Justin
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>> 
>>>>>> 
>>>>>> Confidentiality Notice: This e-mail message (including any attached or
>>>>>> embedded documents) is intended for the exclusive and confidential use
>>>>  of the
>>>>>> individual or entity to which this message is addressed, and unless
>>>>  otherwise
>>>>>> expressly indicated, is confidential and privileged information of
>>>>  Rackspace.
>>>>>> Any dissemination, distribution or copying of the enclosed material is
>>>>  prohibited.
>>>>>> If you receive this transmission in error, please notify us immediately
>>>>  by e-mail
>>>>>> at abuse@xxxxxxxxxxxxx, and delete the original message.
>>>>>> Your cooperation is appreciated.
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help : https://help.launchpad.net/ListHelp