openstack team mailing list archive

[Leander Bessa Beernaert <leanderbb@xxxxxxxxx>]

Also, I'm unable to run the swift-bench with keystone.

I always get this error:

Traceback (most recent call last):
  File "/usr/bin/swift-bench", line 149, in <module>
    controller.run()
  File "/usr/lib/python2.7/dist-packages/swift/common/bench.py", line 159,
in run
    puts = BenchPUT(self.logger, self.conf, self.names)
  File "/usr/lib/python2.7/dist-packages/swift/common/bench.py", line 241,
in __init__
    Bench.__init__(self, logger, conf, names)
  File "/usr/lib/python2.7/dist-packages/swift/common/bench.py", line 55,
in __init__
    auth_version=self.auth_version)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 281,
in get_auth
    if (kwargs['os_options'].get('object_storage_url') and
KeyError: 'os_options'


Here's my config file:

[bench]
auth = http://192.168.111.202:5000/v2.0
user = [tenant]:[user]
key = [key]
auth_version = 2
log-level = INFO
timeout = 10

put_concurrency = 10
get_concurrency = 10

lower_object_size = 200000
upper_object_size = 20000


num_objects = 1000
num_gets = 10000
num_containers = 20

delete = yes


On Mon, Jan 14, 2013 at 5:01 PM, Leander Bessa Beernaert <
leanderbb@xxxxxxxxx> wrote:

> I currently have 4 machines running 10 clients each uploading 1/40th of
> the data. More than 40 simultaneous clientes starts to severely affect
> Keystone's ability to handle these operations.
>
>
> On Mon, Jan 14, 2013 at 4:58 PM, Chuck Thier <cthier@xxxxxxxxx> wrote:
>
>> That should be fine, but it doesn't have any way of reporting stats
>> currently.  You could use tools like ifstat to look at how much
>> bandwidth you are using.  You can also look at how much cpu the swift
>> tool is using.  Depending on how your data is setup, you could run
>> several swift-client processes in parallel until you max either your
>> network or cpu.  I would start with one client first, until you max it
>> out, then move on to the next.
>>
>> --
>> Chuck
>>
>> On Mon, Jan 14, 2013 at 10:45 AM, Leander Bessa Beernaert
>> <leanderbb@xxxxxxxxx> wrote:
>> > I'm currently using the swift client to upload files, would you
>> recommend
>> > another approach?
>> >
>> >
>> > On Mon, Jan 14, 2013 at 4:43 PM, Chuck Thier <cthier@xxxxxxxxx> wrote:
>> >>
>> >> Using swift stat probably isn't the best way to determine cluster
>> >> performance, as those stats are updated async, and could be delayed
>> >> quite a bit as you are heavily loading the cluster.  It also might be
>> >> worthwhile to use a tool like swift-bench to test your cluster to make
>> >> sure it is properly setup before loading data into the system.
>> >>
>> >> --
>> >> Chuck
>> >>
>> >> On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert
>> >> <leanderbb@xxxxxxxxx> wrote:
>> >> > I'm getting around 5-6.5 GB a day of bytes written on Swift. I
>> >> > calculated
>> >> > this by calling "swift stat && sleep 60s && swift stat". I did some
>> >> > calculation based on those values to get to the end result.
>> >> >
>> >> > Currently I'm resetting swift with a node size of 64, since 90% of
>> the
>> >> > files
>> >> > are less than 70KB in size. I think that might help.
>> >> >
>> >> >
>> >> > On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier <cthier@xxxxxxxxx>
>> wrote:
>> >> >>
>> >> >> Hey Leander,
>> >> >>
>> >> >> Can you post what performance you are getting?  If they are all
>> >> >> sharing the same GigE network, you might also check that the links
>> >> >> aren't being saturated, as it is pretty easy to saturate pushing
>> 200k
>> >> >> files around.
>> >> >>
>> >> >> --
>> >> >> Chuck
>> >> >>
>> >> >> On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert
>> >> >> <leanderbb@xxxxxxxxx> wrote:
>> >> >> > Well, I've fixed the node size and disabled the all the replicator
>> >> >> > and
>> >> >> > auditor processes. However, it is even slower now than it was
>> before
>> >> >> > :/.
>> >> >> > Any
>> >> >> > suggestions?
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert
>> >> >> > <leanderbb@xxxxxxxxx> wrote:
>> >> >> >>
>> >> >> >> Ok, thanks for all the tips/help.
>> >> >> >>
>> >> >> >> Regards,
>> >> >> >>
>> >> >> >> Leander
>> >> >> >>
>> >> >> >>
>> >> >> >> On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen
>> >> >> >> <Robert.vanLeeuwen@xxxxxxxxxxxxx> wrote:
>> >> >> >>>
>> >> >> >>> > Allow me to rephrase.
>> >> >> >>> > I've read somewhere (can't remember where) that it would be
>> >> >> >>> > faster
>> >> >> >>> > to
>> >> >> >>> > upload files if they would be uploaded to separate
>> containeres.
>> >> >> >>> > This was suggested for a standard swift installation with a
>> >> >> >>> > certain
>> >> >> >>> > replication factor.
>> >> >> >>> > Since I'll be uploading the files with the replicators turned
>> >> >> >>> > off,
>> >> >> >>> > does
>> >> >> >>> > it really matter if I insert a group of them in separate
>> >> >> >>> > containeres?
>> >> >> >>>
>> >> >> >>> My guess is this concerns the SQLite database load distribution.
>> >> >> >>> So yes, it still matters.
>> >> >> >>>
>> >> >> >>> Just to be clear: turning replicators off does not matter at ALL
>> >> >> >>> when
>> >> >> >>> putting files in a healthy cluster.
>> >> >> >>> Files will be "replicated" / put on all required nodes at the
>> >> >> >>> moment
>> >> >> >>> the
>> >> >> >>> put request is done.
>> >> >> >>> The put request will only give an OK when there is quorum
>> writing
>> >> >> >>> the
>> >> >> >>> file (the file is stored on more than half of the required
>> object
>> >> >> >>> nodes)
>> >> >> >>> The replicator daemons do not have anything to do with this.
>> >> >> >>>
>> >> >> >>> Cheers,
>> >> >> >>> Robert
>> >> >> >>>
>> >> >> >>> _______________________________________________
>> >> >> >>> Mailing list: https://launchpad.net/~openstack
>> >> >> >>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> >> >> >>> Unsubscribe : https://launchpad.net/~openstack
>> >> >> >>> More help   : https://help.launchpad.net/ListHelp
>> >> >> >>>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > Mailing list: https://launchpad.net/~openstack
>> >> >> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> >> >> > Unsubscribe : https://launchpad.net/~openstack
>> >> >> > More help   : https://help.launchpad.net/ListHelp
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>>
>
>