openstack team mailing list archive

[Oleg Gelbukh <ogelbukh@xxxxxxxxxxxx>]

Alex,

Thank you for important point and interesting information on large-scale
Swift performance!

Can you please explain a little what these times stand for? Is this a
single process runtime, or the time needed to converge cluster in case of
device failure, or something else?

--
Best regards,
Oleg

On Fri, Apr 13, 2012 at 7:23 AM, Alex Yang <alex890714@xxxxxxxxx> wrote:

> In my view, the biggest problem of swift is not the new features but the
> improvement of performance.
>
> At first, we knew that container-updater, *-auditor, *-replicator will
> loop all the files in the disk during the running interval. When the number
> of files is very large, the process of replicator, updater and auditor will
> spend much time. So, the time of eventual consistency is very long.
>
> Our practice of  Sina Web Service Team https://launchpad.net/~sws:
>
> total accounts:          121,961;
>
> total containers:        160,703;
>
> total objects:             14,291,519;
>
> total storage usage:  1.3T
>
> account replication time:    10 hours;
>
> container replication time:  10 hours;
>
> object replication time:       48 hours;
>
> account audit time:               2 hours;
>
> container audit time:             9 hours;
>
> container update time:        19 hours;
>
> This is terrible If we develop quota upon account db. There is long time
> for eventual consistency.
>
> Secondly,  there is a vicious circle during replication. The replicator
> will query the account-server, container-server and object-server to
> compare the metadata, and determine whether to sync. When the number of
> files is very large, the frequent query make the account-server,
> container-server and object-server become a bottle neck. This will
> influence the process of proxy-server to work with back-end servers. There
> a lot of Timeout(10s) ERROR in proxy-server's log and the load-average is
> very high.
> So, some PUT, POST operations failed, and the replicator to sync, and fail
> more, and sync more...
> .
> In my opinion, we need to improve the process of replication and container
> update by using event drive framework or something else..
> My leader may discuss this topic at Design Summit,
> http://openstackconferencespring2012.sched.org/speaker/huicheng
>
>
> 2012/4/13 John Dickinson <me@xxxxxx>
>
>> Swift keeps total bytes, container, and object count (eventually)
>> up-to-date in the account metadata. There are also log processing tools
>> (like slogging - http://github.com/notmyname/slogging) that can provide
>> usage information (including bandwidth) based on swift logs.
>>
>> While I think that it's appropriate for swift to generate the usage
>> information (via internal processes or log processing), the appropriate
>> place for quotas is in whatever system handles the concept of a user
>> (normally the auth system). This way quotas are enforced by revoking or
>> limiting access of the auth token.
>>
>> --John
>>
>>
>> On Apr 12, 2012, at 11:53 AM, Frederik Van Hecke wrote:
>>
>> > Hi Kuo,
>> >
>> > One option would be to keep the usage information (num files, num
>> bytes, etc) per container / account in an sqlite DB, just like it is done
>> for account and container info.
>> >
>> > To avoid having to loop through all data at regular intervals (to
>> update the info), additional logic could be added to the api methods to
>> update the sqlite DB's when new files are added, files are deleted, etc.
>> Such approach will require more lines of code, but will be far less
>> stressful on performance.
>> >
>> > (the brute-force approach to loop through it at regular intervals will
>> be hell on performance on large deployments..)
>> >
>> >
>> > For data transfer billing based on download / upload amounts, a similar
>> approach could be used.
>> >
>> > If no one else is looking into this, I would certainly be willing to
>> help to help get this started.
>> >
>> >
>> > Kind regards,
>> > Frederik Van Hecke
>> >
>> > T:  +32487733713
>> > E:  frederik@xxxxxxxxx
>> > W: www.cluttr.be
>> >
>> >
>> >
>> >
>> >
>> > This e-mail and any attachments thereto may contain information which
>> is confidential and/or protected by intellectual property rights and are
>> intended for the sole use of the recipient(s)named above. Any use of the
>> information contained herein (including, but not limited to, total or
>> partial reproduction, communication or distribution in any form) by persons
>> other than the designated recipient(s) is prohibited. If you have received
>> this e-mail in error, please notify the sender either by telephone or by
>> e-mail and delete the material from any computer. Thank you for your
>> cooperation.
>> >
>> >
>> >
>> > On Thu, Apr 12, 2012 at 17:45, Kuo Hugo <tonytkdk@xxxxxxxxx> wrote:
>> > Hi folks ,
>> >
>> > I'm thinking about the better approach to manage "an user" or "an
>> account" space usage quota in swift.
>> > Is  there any related blueprint or sub-project even an idea around ?
>> > Any suggestion of benefits to be an external service or to be a
>> middle-ware in swift-proxy ?
>> >
>> > I'm concerning about such feature will reduce the performance of entire
>> Swift environment.
>> >
>> > Appreciate :>
>> >
>> >
>> >
>> > --
>> > +Hugo Kuo+
>> > tonytkdk@xxxxxxxxx
>> > +886 935004793
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help   : https://help.launchpad.net/ListHelp
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help   : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
>   Email:       alex890714@xxxxxxxxx
> GitHub:       https://github.com/AlexYangYu
>    Blog:        http://alexyang.sinaapp.com
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>