← Back to team overview

openstack team mailing list archive

Re: [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift

 

Hey Leander,

Can you post what performance you are getting?  If they are all
sharing the same GigE network, you might also check that the links
aren't being saturated, as it is pretty easy to saturate pushing 200k
files around.

--
Chuck

On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert
<leanderbb@xxxxxxxxx> wrote:
> Well, I've fixed the node size and disabled the all the replicator and
> auditor processes. However, it is even slower now than it was before :/. Any
> suggestions?
>
>
> On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert
> <leanderbb@xxxxxxxxx> wrote:
>>
>> Ok, thanks for all the tips/help.
>>
>> Regards,
>>
>> Leander
>>
>>
>> On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen
>> <Robert.vanLeeuwen@xxxxxxxxxxxxx> wrote:
>>>
>>> > Allow me to rephrase.
>>> > I've read somewhere (can't remember where) that it would be faster to
>>> > upload files if they would be uploaded to separate containeres.
>>> > This was suggested for a standard swift installation with a certain
>>> > replication factor.
>>> > Since I'll be uploading the files with the replicators turned off, does
>>> > it really matter if I insert a group of them in separate containeres?
>>>
>>> My guess is this concerns the SQLite database load distribution.
>>> So yes, it still matters.
>>>
>>> Just to be clear: turning replicators off does not matter at ALL when
>>> putting files in a healthy cluster.
>>> Files will be "replicated" / put on all required nodes at the moment the
>>> put request is done.
>>> The put request will only give an OK when there is quorum writing the
>>> file (the file is stored on more than half of the required object nodes)
>>> The replicator daemons do not have anything to do with this.
>>>
>>> Cheers,
>>> Robert
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>


Follow ups

References