← Back to team overview

openstack team mailing list archive

Re: Swift Consistency Guarantees?

 

Actually I got that wrong for GET swift would try to serve the object even
if there is only one replica left for PUT it would need to have at least 2
successful write to succeed assuming you have a 3 replicas cluster.

On Fri, Jan 20, 2012 at 2:06 PM, Chmouel Boudjnah <chmouel@xxxxxxxxxxxxx>wrote:

> As Stephen mentionned if there is only one replica left Swift would not
> serve it.
>
> Chmouel.
>
>
> On Fri, Jan 20, 2012 at 1:58 PM, Nikolaus Rath <Nikolaus@xxxxxxxx> wrote:
>
>> Hi,
>>
>> Sorry for being so persistent, but I'm still not sure what happens if
>> the 2 servers that carry the new replica are down, but the 1 server that
>> has the old replica is up. Will GET fail or return the old replica?
>>
>> Best,
>> Niko
>>
>> On 01/20/2012 02:52 PM, Stephen Broeker wrote:
>> > By default there are 3 replicas.
>> > A PUT Object will return after 2 replicas are done.
>> > So if all nodes are up then there are at least 2 replicas.
>> > If all replica nodes are down, then the GET Object will fail.
>> >
>> > On Fri, Jan 20, 2012 at 11:21 AM, Nikolaus Rath <Nikolaus@xxxxxxxx
>> > <mailto:Nikolaus@xxxxxxxx>> wrote:
>> >
>> >     Hi,
>> >
>> >     So if an object update has not yet been replicated on all nodes,
>> and all
>> >     nodes that have been updated are offline, what will happen? Will
>> swift
>> >     recognize this and give me an error, or will it silently return the
>> >     older version?
>> >
>> >     Thanks,
>> >     Nikolaus
>> >
>> >
>> >     On 01/20/2012 02:14 PM, Stephen Broeker wrote:
>> >     > If a node is down, then it is ignored.
>> >     > That is the whole point about 3 replicas.
>> >     >
>> >     > On Fri, Jan 20, 2012 at 10:43 AM, Nikolaus Rath <
>> Nikolaus@xxxxxxxx
>> >     <mailto:Nikolaus@xxxxxxxx>
>> >     > <mailto:Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>>> wrote:
>> >     >
>> >     >     Hi,
>> >     >
>> >     >     What happens if one of the nodes is down? Especially if that
>> >     node holds
>> >     >     the newest copy?
>> >     >
>> >     >     Thanks,
>> >     >     Nikolaus
>> >     >
>> >     >     On 01/20/2012 12:33 PM, Stephen Broeker wrote:
>> >     >     > The X-Newest header can be used by a GET Operation to ensure
>> >     that
>> >     >     all of the
>> >     >     > Storage Nodes (3 by default) are queried for the latest
>> copy of
>> >     >     the Object.
>> >     >     > The COPY Object operation already has this functionality.
>> >     >     >
>> >     >     > On Fri, Jan 20, 2012 at 9:12 AM, Nikolaus Rath
>> >     <Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>
>> >     >     <mailto:Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>>
>> >     >     > <mailto:Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>
>> >     <mailto:Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>>>> wrote:
>> >     >     >
>> >     >     >     Hi,
>> >     >     >
>> >     >     >     No one able to further clarify this?
>> >     >     >
>> >     >     >     Does swift offer there read-after-create consistence
>> like
>> >     >     >     non-us-standard S3? What are the precise syntax and
>> >     semantics of
>> >     >     >     X-Newest header?
>> >     >     >
>> >     >     >     Best,
>> >     >     >     Nikolaus
>> >     >     >
>> >     >     >
>> >     >     >     On 01/18/2012 10:15 AM, Nikolaus Rath wrote:
>> >     >     >     > Michael Barton <mike-launchpad@xxxxxxxxxxxxxxxx
>> >     <mailto:mike-launchpad@xxxxxxxxxxxxxxxx>
>> >     >     <mailto:mike-launchpad@xxxxxxxxxxxxxxxx
>> >     <mailto:mike-launchpad@xxxxxxxxxxxxxxxx>>
>> >     >     >     <mailto:mike-launchpad@xxxxxxxxxxxxxxxx
>> >     <mailto:mike-launchpad@xxxxxxxxxxxxxxxx>
>> >     >     <mailto:mike-launchpad@xxxxxxxxxxxxxxxx
>> >     <mailto:mike-launchpad@xxxxxxxxxxxxxxxx>>>> writes:
>> >     >     >     >> On Tue, Jan 17, 2012 at 4:55 PM, Nikolaus Rath
>> >     >     <Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>
>> >     <mailto:Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>>
>> >     >     >     <mailto:Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>
>> >     <mailto:Nikolaus@xxxxxxxx <mailto:Nikolaus@xxxxxxxx>>>> wrote:
>> >     >     >     >>> Amazon S3 and Google Storage make very explicit
>> (non-)
>> >     >     consistency
>> >     >     >     >>> guarantees for stored objects. I'm looking for a
>> similar
>> >     >     >     documentation
>> >     >     >     >>> about OpenStack's Swift, but haven't had much
>> success.
>> >     >     >     >>
>> >     >     >     >> I don't think there's any documentation on this, but
>> >     it would
>> >     >     >     probably
>> >     >     >     >> be good to write up.  Consistency in Swift is very
>> >     similar
>> >     >     to S3.
>> >     >     >     >> That is, there aren't many non-eventual consistency
>> >     guarantees.
>> >     >     >     >>
>> >     >     >     >> Listing updates can happen asynchronously (especially
>> >     under
>> >     >     >     load), and
>> >     >     >     >> older versions of files can show up in requests
>> (deletes
>> >     >     are just a
>> >     >     >     >> new "deleted" version of the file).
>> >     >     >     >
>> >     >     >     > Ah, ok. Thanks a lot for stating this so explicitly.
>> >     There seems
>> >     >     >     to be a
>> >     >     >     > lot of confusion about this, now I can at least point
>> >     people to
>> >     >     >     > something.
>> >     >     >     >
>> >     >     >     >> Swift can generally be relied on for read-after-write
>> >     >     consistency,
>> >     >     >     >> like S3's regions other than the the US Standard
>> region.
>> >     >      The reason
>> >     >     >     >> S3 in US Standard doesn't have this guarantee is
>> because
>> >     >     it's more
>> >     >     >     >> geographically widespread - something Swift isn't
>> good at
>> >     >     yet.  I can
>> >     >     >     >> imagine we'll have the same limitation when we get
>> there.
>> >     >     >     >
>> >     >     >     > Do you mean read-after-create consistency? Because
>> >     below you
>> >     >     say about
>> >     >     >     > read-after-write:
>> >     >     >     >
>> >     >     >     >>> - If I receive a (non-error) response to a PUT
>> >     request, am I
>> >     >     >     guaranteed
>> >     >     >     >>> that the object will be immediately included in all
>> >     object
>> >     >     >     listings in
>> >     >     >     >>> every possible situation?
>> >     >     >     >>
>> >     >     >     >> Nope.
>> >     >     >     >
>> >     >     >     > ..so is there such a guarantee for PUTs of *new*
>> objects
>> >     >     (like S3 non
>> >     >     >     > us-classic), or does "can generally be relied on" just
>> >     mean
>> >     >     that the
>> >     >     >     > chances for new puts are better?
>> >     >     >     >
>> >     >     >     >> Also like S3, Swift can't make any strong guarantees
>> >     about
>> >     >     >     >> read-after-update or read-after-delete consistency.
>> >      We do
>> >     >     have an
>> >     >     >     >> "X-Newest" header that can be added to GETs and
>> HEADs to
>> >     >     make the
>> >     >     >     >> proxy do a quorum of backend servers and return the
>> >     newest
>> >     >     available
>> >     >     >     >> version, which greatly improves these, at the cost of
>> >     latency.
>> >     >     >     >
>> >     >     >     > That sounds very interesting. Could you give some more
>> >     >     details on what
>> >     >     >     > exactly is guaranteed when using this header? What
>> happens
>> >     >     if the
>> >     >     >     server
>> >     >     >     > having the newest copy is down?
>> >     >     >     >
>> >     >     >     >>> - If the swift server looses an object, will the
>> >     object name
>> >     >     >     still be
>> >     >     >     >>> returned in object listings? Will attempts to
>> >     retrieve it
>> >     >     result
>> >     >     >     in 404
>> >     >     >     >>> errors (as if it never existed) or a different
>> error?
>> >     >     >     >>
>> >     >     >     >> It will show up in listings, but give a 404 when you
>> >     attempt to
>> >     >     >     >> retrieve it.  I'm not sure how we can improve that
>> >     with Swift's
>> >     >     >     >> general model, but feel free to make suggestions.
>> >     >     >     >
>> >     >     >     > From an application programmers point of view, it
>> >     would be very
>> >     >     >     helpful
>> >     >     >     > if lost objects could be distinguished from
>> non-existing
>> >     >     object by a
>> >     >     >     > different HTTP error. Trying to access a non-existing
>> >     object may
>> >     >     >     > indicate a bug in the application, so it would be
>> nice to
>> >     >     know when it
>> >     >     >     > happens.
>> >     >     >     >
>> >     >     >     > Also, it would be very helpful if there was a way to
>> list
>> >     >     all lost
>> >     >     >     > objects without having to issue HEAD requests for
>> every
>> >     >     stored object.
>> >     >     >     > Could this information be added to the XML and JSON
>> >     output of
>> >     >     >     container
>> >     >     >     > listings? Then an application would have the chance to
>> >     >     periodically
>> >     >     >     > check for lost data, rather than having to handle all
>> lost
>> >     >     objects at
>> >     >     >     > the instant they're required.
>> >     >     >     >
>> >     >     >     >
>> >     >     >     > I am working on a swift backend for S3QL
>> >     >     >     > (http://code.google.com/p/s3ql/), a program that
>> exposes
>> >     >     online cloud
>> >     >     >     > storage as a local UNIX file system. To prevent data
>> >     >     corruption, there
>> >     >     >     > are two requirements that I'm currently struggling to
>> >     >     provide with the
>> >     >     >     > swift backend:
>> >     >     >     >
>> >     >     >     > - There needs to be a way to reliably check if one
>> object
>> >     >     (holding the
>> >     >     >     >   file system metadata) is the newest version.
>> >     >     >     >
>> >     >     >     >   The S3 backend does this by requiring storage in
>> the non
>> >     >     us-classic
>> >     >     >     >   regions and using list-after-create consistency
>> with a
>> >     >     marker object
>> >     >     >     >   that has has a "generation number" of the metadata
>> >     >     embedded in its
>> >     >     >     >   name.
>> >     >     >     >
>> >     >     >     >   I'm not yet sure if this would work with swift as
>> well
>> >     >     (the google
>> >     >     >     >   storage backend just relies on the strong
>> >     read-after-write
>> >     >     >     >   consistency).
>> >     >     >     >
>> >     >     >     > - The file system checker needs a way to identify lost
>> >     objects.
>> >     >     >     >
>> >     >     >     >   Here the S3 backend just relies on the durability
>> >     >     guarantee that
>> >     >     >     >   effectively no object will ever be lost.
>> >     >     >     >
>> >     >     >     >   Again, I'm not sure how to implement this for swift.
>> >     >     >     >
>> >     >     >     >
>> >     >     >     > Any suggestions?
>> >     >     >     >
>> >     >     >     >
>> >     >     >     >
>> >     >     >     > Best,
>> >     >     >     >
>> >     >     >     >    -Nikolaus
>> >     >     >     >
>> >     >     >
>> >     >     >
>> >     >     >       -Nikolaus
>> >     >     >
>> >     >     >     --
>> >     >     >      »Time flies like an arrow, fruit flies like a Banana.«
>> >     >     >
>> >     >     >      PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD
>> B7F8
>> >     >     AE4E 425C
>> >     >     >
>> >     >     >     _______________________________________________
>> >     >     >     Mailing list: https://launchpad.net/~openstack
>> >     >     >     Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>> >     >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
>> >     >     >     Unsubscribe : https://launchpad.net/~openstack
>> >     >     >     More help   : https://help.launchpad.net/ListHelp
>> >     >     >
>> >     >     >
>> >     >
>> >     >
>> >     >       -Nikolaus
>> >     >
>> >     >     --
>> >     >      »Time flies like an arrow, fruit flies like a Banana.«
>> >     >
>> >     >      PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8
>> >     AE4E 425C
>> >     >
>> >     >
>> >
>> >
>> >       -Nikolaus
>> >
>> >     --
>> >      »Time flies like an arrow, fruit flies like a Banana.«
>> >
>> >      PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
>> >
>> >
>>
>>
>>   -Nikolaus
>>
>> --
>>  »Time flies like an arrow, fruit flies like a Banana.«
>>
>>  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>

Follow ups

References