openstack team mailing list archive

[Chuck Thier <cthier@xxxxxxxxx>]

That should be fine, but it doesn't have any way of reporting stats
currently.  You could use tools like ifstat to look at how much
bandwidth you are using.  You can also look at how much cpu the swift
tool is using.  Depending on how your data is setup, you could run
several swift-client processes in parallel until you max either your
network or cpu.  I would start with one client first, until you max it
out, then move on to the next.

--
Chuck

On Mon, Jan 14, 2013 at 10:45 AM, Leander Bessa Beernaert
<leanderbb@xxxxxxxxx> wrote:
> I'm currently using the swift client to upload files, would you recommend
> another approach?
>
>
> On Mon, Jan 14, 2013 at 4:43 PM, Chuck Thier <cthier@xxxxxxxxx> wrote:
>>
>> Using swift stat probably isn't the best way to determine cluster
>> performance, as those stats are updated async, and could be delayed
>> quite a bit as you are heavily loading the cluster.  It also might be
>> worthwhile to use a tool like swift-bench to test your cluster to make
>> sure it is properly setup before loading data into the system.
>>
>> --
>> Chuck
>>
>> On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert
>> <leanderbb@xxxxxxxxx> wrote:
>> > I'm getting around 5-6.5 GB a day of bytes written on Swift. I
>> > calculated
>> > this by calling "swift stat && sleep 60s && swift stat". I did some
>> > calculation based on those values to get to the end result.
>> >
>> > Currently I'm resetting swift with a node size of 64, since 90% of the
>> > files
>> > are less than 70KB in size. I think that might help.
>> >
>> >
>> > On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier <cthier@xxxxxxxxx> wrote:
>> >>
>> >> Hey Leander,
>> >>
>> >> Can you post what performance you are getting?  If they are all
>> >> sharing the same GigE network, you might also check that the links
>> >> aren't being saturated, as it is pretty easy to saturate pushing 200k
>> >> files around.
>> >>
>> >> --
>> >> Chuck
>> >>
>> >> On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert
>> >> <leanderbb@xxxxxxxxx> wrote:
>> >> > Well, I've fixed the node size and disabled the all the replicator
>> >> > and
>> >> > auditor processes. However, it is even slower now than it was before
>> >> > :/.
>> >> > Any
>> >> > suggestions?
>> >> >
>> >> >
>> >> > On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert
>> >> > <leanderbb@xxxxxxxxx> wrote:
>> >> >>
>> >> >> Ok, thanks for all the tips/help.
>> >> >>
>> >> >> Regards,
>> >> >>
>> >> >> Leander
>> >> >>
>> >> >>
>> >> >> On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen
>> >> >> <Robert.vanLeeuwen@xxxxxxxxxxxxx> wrote:
>> >> >>>
>> >> >>> > Allow me to rephrase.
>> >> >>> > I've read somewhere (can't remember where) that it would be
>> >> >>> > faster
>> >> >>> > to
>> >> >>> > upload files if they would be uploaded to separate containeres.
>> >> >>> > This was suggested for a standard swift installation with a
>> >> >>> > certain
>> >> >>> > replication factor.
>> >> >>> > Since I'll be uploading the files with the replicators turned
>> >> >>> > off,
>> >> >>> > does
>> >> >>> > it really matter if I insert a group of them in separate
>> >> >>> > containeres?
>> >> >>>
>> >> >>> My guess is this concerns the SQLite database load distribution.
>> >> >>> So yes, it still matters.
>> >> >>>
>> >> >>> Just to be clear: turning replicators off does not matter at ALL
>> >> >>> when
>> >> >>> putting files in a healthy cluster.
>> >> >>> Files will be "replicated" / put on all required nodes at the
>> >> >>> moment
>> >> >>> the
>> >> >>> put request is done.
>> >> >>> The put request will only give an OK when there is quorum writing
>> >> >>> the
>> >> >>> file (the file is stored on more than half of the required object
>> >> >>> nodes)
>> >> >>> The replicator daemons do not have anything to do with this.
>> >> >>>
>> >> >>> Cheers,
>> >> >>> Robert
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> Mailing list: https://launchpad.net/~openstack
>> >> >>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> >> >>> Unsubscribe : https://launchpad.net/~openstack
>> >> >>> More help   : https://help.launchpad.net/ListHelp
>> >> >>>
>> >> >>
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Mailing list: https://launchpad.net/~openstack
>> >> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> >> > Unsubscribe : https://launchpad.net/~openstack
>> >> > More help   : https://help.launchpad.net/ListHelp
>> >> >
>> >
>> >
>
>