openstack team mailing list archive

[Leander Bessa Beernaert <leanderbb@xxxxxxxxx>]

I'm currently using the swift client to upload files, would you recommend
another approach?


On Mon, Jan 14, 2013 at 4:43 PM, Chuck Thier <cthier@xxxxxxxxx> wrote:

> Using swift stat probably isn't the best way to determine cluster
> performance, as those stats are updated async, and could be delayed
> quite a bit as you are heavily loading the cluster.  It also might be
> worthwhile to use a tool like swift-bench to test your cluster to make
> sure it is properly setup before loading data into the system.
>
> --
> Chuck
>
> On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert
> <leanderbb@xxxxxxxxx> wrote:
> > I'm getting around 5-6.5 GB a day of bytes written on Swift. I calculated
> > this by calling "swift stat && sleep 60s && swift stat". I did some
> > calculation based on those values to get to the end result.
> >
> > Currently I'm resetting swift with a node size of 64, since 90% of the
> files
> > are less than 70KB in size. I think that might help.
> >
> >
> > On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier <cthier@xxxxxxxxx> wrote:
> >>
> >> Hey Leander,
> >>
> >> Can you post what performance you are getting?  If they are all
> >> sharing the same GigE network, you might also check that the links
> >> aren't being saturated, as it is pretty easy to saturate pushing 200k
> >> files around.
> >>
> >> --
> >> Chuck
> >>
> >> On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert
> >> <leanderbb@xxxxxxxxx> wrote:
> >> > Well, I've fixed the node size and disabled the all the replicator and
> >> > auditor processes. However, it is even slower now than it was before
> :/.
> >> > Any
> >> > suggestions?
> >> >
> >> >
> >> > On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert
> >> > <leanderbb@xxxxxxxxx> wrote:
> >> >>
> >> >> Ok, thanks for all the tips/help.
> >> >>
> >> >> Regards,
> >> >>
> >> >> Leander
> >> >>
> >> >>
> >> >> On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen
> >> >> <Robert.vanLeeuwen@xxxxxxxxxxxxx> wrote:
> >> >>>
> >> >>> > Allow me to rephrase.
> >> >>> > I've read somewhere (can't remember where) that it would be faster
> >> >>> > to
> >> >>> > upload files if they would be uploaded to separate containeres.
> >> >>> > This was suggested for a standard swift installation with a
> certain
> >> >>> > replication factor.
> >> >>> > Since I'll be uploading the files with the replicators turned off,
> >> >>> > does
> >> >>> > it really matter if I insert a group of them in separate
> >> >>> > containeres?
> >> >>>
> >> >>> My guess is this concerns the SQLite database load distribution.
> >> >>> So yes, it still matters.
> >> >>>
> >> >>> Just to be clear: turning replicators off does not matter at ALL
> when
> >> >>> putting files in a healthy cluster.
> >> >>> Files will be "replicated" / put on all required nodes at the moment
> >> >>> the
> >> >>> put request is done.
> >> >>> The put request will only give an OK when there is quorum writing
> the
> >> >>> file (the file is stored on more than half of the required object
> >> >>> nodes)
> >> >>> The replicator daemons do not have anything to do with this.
> >> >>>
> >> >>> Cheers,
> >> >>> Robert
> >> >>>
> >> >>> _______________________________________________
> >> >>> Mailing list: https://launchpad.net/~openstack
> >> >>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >> >>> Unsubscribe : https://launchpad.net/~openstack
> >> >>> More help   : https://help.launchpad.net/ListHelp
> >> >>>
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Mailing list: https://launchpad.net/~openstack
> >> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >> > Unsubscribe : https://launchpad.net/~openstack
> >> > More help   : https://help.launchpad.net/ListHelp
> >> >
> >
> >
>