openstack team mailing list archive

[Brian Schott <bfschott@xxxxxxxxx>]

In addition to zone ids being opaque, they have to be anonymized.  On AWS, they randomize the assignment of availability zones per account because "us-east-1a" datacenter was melting the floor tiles due to load while "us-east-1c" datacenter was idling.  This causes problems for those of us who manage development and deployment accounts.  
Unexpected consequences of lazy cloud application developers in a hurry.
 
As one of those lazy developers, I don't really want visibility into or have to understand the provider's physical infrastructure or even logical organization for that matter.  The only value I see in zones is that spreading across them gives me fault tolerance and selecting one gives me somewhat better network performance (or saves bandwidth charges).

Exposing your topology can be a bad thing.  

Brian Schott
bfschott@xxxxxxxxx



On Feb 11, 2011, at 3:01 PM, Vishvananda Ishaya wrote:

> Yes, i think that cross-project should be requests across an admin api.  I don't think high level zone info is enough information for the scheduler
> to decide where the instance should go.  Especially with things like swift, where there are three copies that may be in different zones altogether.
> 
> Vish
> On Feb 11, 2011, at 11:55 AM, Eric Day wrote:
> 
>> Hey Vish!
>> 
>> On Fri, Feb 11, 2011 at 12:10:14PM -0600, Vishvananda Ishaya wrote:
>>>  I think that providers may not wish to expose the internal structure of
>>>  their network to the degree that you are suggesting. I prefer the idea of
>>>  near other object with an internal lookup into zone.
>> 
>> Providers can always obscure the naming to hide topology, or
>> services can have options to only expose a certain depth of
>> the hierarchy (ie, truncate at the 'datacenter level', hide all
>> room/rack/switch zone level details). So even though volume-001
>> may be in rack12.room2.dc1.dfw.rackspace.com, the API only returns
>> dc1.dfw.rackspace.com. We can of course allow for both too when the
>> granularity is needed within a project:
>> 
>> Within nova requests, allow:
>> 
>> near-volume:volume-001
>> 
>> But across projects:
>> in-zone: dc1.dfw.rackspace.com
>> 
>> If we don't expose object zones at all, how would you support
>> cross-project lookups? If nova gets a request 'near:swift-12345',
>> would it be configured to query swift to get the zone via an admin API?
>> 
>> We could also expose detailed object zones but keep them completely
>> opaque, and instead require each service to allow pluggable
>> scheduler/zone mapping. For example, a volume may return zone
>> '87af1b7c', which internally all services could lookup and translate
>> to where that maps in the deployment hierarchy. I was hoping DNS names
>> were abstract enough to reduce complexity, but perhaps they are not. :)
>> 
>> -Eric
>> 
>>>  On Feb 11, 2011 9:38 AM, "Eric Day" <eday@xxxxxxxxxxxx> wrote:
>>>> Hi Sandy,
>>>> 
>>>> I agree with using tags for full scheduler selection, it's something
>>>> I've been pushing for from the start. The request contains any number
>>>> of k/v pairs, the services provide any number of k/v pairs, and the
>>>> scheduler performs a match (some required, some optional, ...). I see
>>>> the URI/zone as one of those tags, not something we need to overload
>>>> to contain all of the capabilities. It should only be a hierarchical
>>>> "location", which may be geographic location, organizational location
>>>> (dept, ...), or some other type (however you decide to construct
>>>> your zones).
>>>> 
>>>> For example, imagine a dynamic del.icio.us tag that allowed for domain
>>>> name filtering on bookmarks (give me all bookmarks with tags [book
>>>> review] [domain:slashdot.org]). For Nova, this means issuing requests
>>>> like "create instance with [GPU] [Fast disk] [zone:dc1.example.com]".
>>>> 
>>>> The important thing is that this is not a tag specific to a particular
>>>> service. For example, Swift would never care or need to understand a
>>>> 'GPU' tag, but it can share and understand zone tags.
>>>> 
>>>> -Eric
>>>> 
>>>> On Fri, Feb 11, 2011 at 12:40:44PM +0000, Sandy Walsh wrote:
>>>>> Heh, hate to be the one to bust up the URI love-fest :)
>>>>> 
>>>>> The issue I have with a single URI being used as the heuristic for node
>>>  selection is that it is very rigid.
>>>>> 
>>>>> Different business units have different views on the network:
>>>>> * Operations may view it as geography/data centers.
>>>>> * Consumers may view it as technical ability (gpu's, fast disk, good
>>>  inter-server speed, etc)
>>>>> * Sales/marketing may view it as the number of martinis they can buy ;)
>>>>> 
>>>>> Trees become unmanageable/hard to visualize for users beyond a couple
>>>  hundred nodes. We are lucky that our geographical/DC-based hierarchy is
>>>  relatively flat. This is why I was initially pushing for a tag-based
>>>  system for selection (aka Zone/Host Capabilities).
>>>>> 
>>>>> Consider the way delicio.us works. They manage many millions of URL's
>>>  and tags are an effective way to slice & dice your way through the data:
>>>>> "Show me all the URL's on [OpenStack] [Python] [Zones] [Scheduler]" ...
>>>  blam.
>>>>> 
>>>>> This is also the way the old Trader services worked:
>>>>> "I want a [wax transfer] [color] printer that can has [30ppm] and
>>>  [300dpi] on [Floor 2]"
>>>>> 
>>>>> "Near" simply has to mean the distance in zones from the most-optimal
>>>  zones, based on the tags.
>>>>> 
>>>>> "I want a new instance with [GPU] and [Fast Disk] [Good inter-instance
>>>  network speed] [near] [DRW] [DC1]"
>>>>> * where "[near]" implies "as close as possible to" in zone distance.
>>>>> 
>>>>> Personally I don't like overloading the zone name to have a
>>>  "meaningful" URI when we can get the same functionality with
>>>  Capabilities/Tags already. And we already know we need Capability support
>>>  anyway. Especially if it means enforcing a rigid hierarchy.
>>>>> 
>>>>> $0.02
>>>>> 
>>>>> -S
>>>>> 
>>>>> 
>>>>> 
>>>>> ________________________________________
>>>>> From: openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx
>>>  [openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx] on
>>>  behalf of Eric Day [eday@xxxxxxxxxxxx]
>>>>> Sent: Friday, February 11, 2011 4:30 AM
>>>>> To: Justin Santa Barbara
>>>>> Cc: openstack@xxxxxxxxxxxxxxxxxxx; Devin Carlen
>>>>> Subject: Re: [Openstack] Allowing clients to pass capability requests
>>>  through tags?
>>>>> 
>>>>> The main reason I was proposing full location/zone of objects is to
>>>>> allow this type of 'near' scheduling to happen without understanding
>>>>> what the actual object is. For example, imagine we want to start an
>>>>> instance near a particular swift object. We could query the swift
>>>>> object and in the metadata there could be a 'zone' tag (well, three,
>>>>> one for each copy). For example:
>>>>> 
>>>>> get swift-12345: zone=rack12.room2.dc1.dfw.rackspace.com
>>>>> 
>>>>> I can now use that zone name to:
>>>>> 
>>>>> create_instance: openstack:near=rack12.room2.dc1.dfw.rackspace.com
>>>>> 
>>>>> The deployment can decide what 'near' is (perhaps a measure of link
>>>>> speed or latency). This way a particular deployment that uses the
>>>>> same URI/zone names across projects can account for locality without
>>>>> knowing what objects from different services are. If it were just
>>>>> 'near=swift-12345', it would need to understand what a swift object
>>>>> was and perform that lookup to find out where it is.
>>>>> 
>>>>> So you can still grab a zone tag from a volume you created:
>>>>> 
>>>>> get vol-000001: rack4.room2.dc1.dfw.rackspace.com
>>>>> 
>>>>> and use the zone to launch an instance with:
>>>>> 
>>>>> create_instance: openstack:near=rack4.room2.dc1.dfw.rackspace.com
>>>>> 
>>>>> We can also write schedulers/tools for a particular deployment
>>>>> that understands the zones to just say 'always prefer in
>>>>> dc1.dfw.rackspace.com', because power is cheaper there right now, or
>>>>> 'test.dc1.dfw.rackspace.com' because that is my test zone (perhaps
>>>>> only enabled for certain accounts in the scheduler too).
>>>>> 
>>>>> -Eric
>>>>> 
>>>>> On Thu, Feb 10, 2011 at 03:38:42PM -0800, Justin Santa Barbara wrote:
>>>>>> I think the blueprint was largely complementary to the multi-zone
>>>  stuff;
>>>>>> this is more about how the client _requests_ a particular
>>>>>> location/capability through the API. The multi-zone blueprint seems
>>>  to be
>>>>>> more about how nova would satisfy those requests (in a non-trivial
>>>  zone
>>>>>> structure.)
>>>>>> The root motivator is indeed getting a 'good' connection to a storage
>>>>>> volume. I'm thinking of iSCSI SAN storage here, so in my case this
>>>>>> probably means the SAN device with the least number of switches in
>>>>>> between. There could well be SAN devices in each rack (e.g. Solaris
>>>>>> volume nodes), or the devices could even be running on the host
>>>  nodes, and
>>>>>> I don't believe that zones in the EC2 sense are sufficient here.
>>>>>> But I guess that if the zone hierarchy went all the way down to the
>>>  rack
>>>>>> (or machine), that would work. So I could create a volume and it
>>>  would
>>>>>> come back with a location of "rack4.room2.dc1.dfw.rackspace.com" and
>>>  I
>>>>>> could then request allocation of machines in that same rack? Is that
>>>  the
>>>>>> vision of the nested zones?
>>>>>> I do have a concern that long-term if we _only_ use zones, that's
>>>  trying
>>>>>> to multiplex a lot of information into the zone hierarchy, and we can
>>>>>> really only put one attribute in there. I also like the flexibility
>>>  of
>>>>>> the 'openstack:near=vol-000001' request, because then the cloud can
>>>  decide
>>>>>> how near to place the instance based on its knowledge of the
>>>  topology, and
>>>>>> the clients can be oblivious to the storage system and arrangement.
>>>  But,
>>>>>> my immediate requirement would indeed be satisfied if the zones went
>>>  down
>>>>>> to the rack/machine level.
>>>>>> An alternative way to look at zones and instance-types is that
>>>  they're
>>>>>> actually just fail-if-not-satisfiable tags of the creation request
>>>>>> (openstack:+zone=us-east-1a and openstack:+instancetype=m1.large)
>>>  They're
>>>>>> only distinguished attributes because AWS doesn't have an
>>>>>> extensibility mechanism, which this blueprint would give us.
>>>>>> Justin
>>>>>> 
>>>>>> On Thu, Feb 10, 2011 at 3:12 PM, Devin Carlen <devcamcar@xxxxxx>
>>>  wrote:
>>>>>> 
>>>>>> I haven't totally digested this blueprint yet but it seems like there
>>>  is
>>>>>> some overlap with what is being discussed with the multi zone
>>>  metadata
>>>>>> stuff. One approach might be to handle this awt the scheduler level
>>>>>> though and try to ensure things are always in the same zone when
>>>>>> appropriate.
>>>>>> I think the bigger question you raise is how to request local volumes
>>>>>> when possible, yes?
>>>>>> 
>>>>>> Devin
>>>>>> On Feb 10, 2011, at 3:37 PM, Justin Santa Barbara
>>>  <justin@xxxxxxxxxxxx>
>>>>>> wrote:
>>>>>> 
>>>>>> Does anyone have any thoughts/objections on the blueprint I posted
>>>  for
>>>>>> allowing clients to pass capability-requests through tags? I'm
>>>>>> planning on starting implementation soon, so if people think this is
>>>  a
>>>>>> bad idea I'd rather know before I start coding!
>>>>>> Blueprint:
>>>  https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
>>>>>> Wiki:
>>>  https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
>>>>>> And a quick TLDR:
>>>>>> API clients need a way to request e.g. placement of machines near
>>>  each
>>>>>> other / near volumes, or that a volume be created with a particular
>>>>>> RAID level, or that a machine be created in a HIPAA compliant
>>>>>> environment. (This is complementary to the work on hierarchical zones
>>>>>> & URL naming, I believe)
>>>>>> I propose using the instance tags for this, e.g. specifying
>>>>>> openstack:near=vol-000001 when creating an instance to request
>>>>>> locating the instance 'close to' that volume.
>>>>>> By default these requests would be best-effort and
>>>  ignored-if-unknown;
>>>>>> if the client wants to specify that something is required and should
>>>>>> fail if not understood or not satisfiable, they could use a "+" e.g.
>>>>>> openstack:+location=*.dc1.north.rackspace.com
>>>>>> Controversially (?), this would not be supported for clients using
>>>  the
>>>>>> AWS API, because tags can only be specified once the instance has
>>>>>> already been created.
>>>>>> Feedback appreciated!
>>>>>> Justin
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>> 
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>> 
>>>>> 
>>>>> Confidentiality Notice: This e-mail message (including any attached or
>>>>> embedded documents) is intended for the exclusive and confidential use
>>>  of the
>>>>> individual or entity to which this message is addressed, and unless
>>>  otherwise
>>>>> expressly indicated, is confidential and privileged information of
>>>  Rackspace.
>>>>> Any dissemination, distribution or copying of the enclosed material is
>>>  prohibited.
>>>>> If you receive this transmission in error, please notify us immediately
>>>  by e-mail
>>>>> at abuse@xxxxxxxxxxxxx, and delete the original message.
>>>>> Your cooperation is appreciated.
>>>> 
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help : https://help.launchpad.net/ListHelp
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp