openstack team mailing list archive

[Kuo Hugo <tonytkdk@xxxxxxxxx>]

Hi Jonathan ,

How did you perform "delete all the objects in the storage" ?  Those
deleted objects still consume inodes in tombstone status until the reclaim
time.
Would you mind to compare the result of $> sudo cat /proc/slabinfo | grep
xfs   ,  before/after set the vfs_cache_pressure

spongebob@patrick1:~$ sudo cat /proc/slabinfo | grep xfs
xfs_ili                70153  70182    216   18    1 : tunables    0    0
 0 : slabdata   3899   3899      0
xfs_inode         169738 170208   1024   16    4 : tunables    0    0    0
: slabdata  10638  10638      0
xfs_efd_item          60     60    400   20    2 : tunables    0    0    0
: slabdata      3      3      0
xfs_buf_item         234    234    224   18    1 : tunables    0    0    0
: slabdata     13     13      0
xfs_trans             28     28    280   14    1 : tunables    0    0    0
: slabdata      2      2      0
xfs_da_state          32     32    488   16    2 : tunables    0    0    0
: slabdata      2      2      0
xfs_btree_cur         38     38    208   19    1 : tunables    0    0    0
: slabdata      2      2      0
xfs_log_ticket        40     40    200   20    1 : tunables    0    0    0
: slabdata      2      2      0


Hi Robert,
The performance degradation still there even only main swift workers are
running in storage node. ( stop replicator/updater/auditor ). In my
knowing.
I'll check xs_dir_lookup and xs_ig_missed here. Thanks







+Hugo Kuo+
hugo@xxxxxxxxxxxxxx
tonytkdk@xxxxxxxxx
+886 935004793


2013/6/18 Jonathan Lu <jojokururu@xxxxxxxxx>

> On 2013/6/17 18:59, Robert van Leeuwen wrote:
>
>> I'm facing the issue about the performance degradation, and once I
>>> glanced that changing the value in /proc/sys
>>> /vm/vfs_cache_pressure will do a favour.
>>> Can anyone explain to me whether and why it is useful?
>>>
>> Hi,
>>
>> When this is set to a lower value the kernel will try to keep the
>> inode/dentry cache longer in memory.
>> Since the swift replicator is scanning the filesystem continuously it
>> will eat up a lot of iops if those are not in memory.
>>
>> To see if a lot of cache misses are happening, for xfs, you can look at
>> xs_dir_lookup and xs_ig_missed.
>> ( look at http://xfs.org/index.php/**Runtime_Stats<http://xfs.org/index.php/Runtime_Stats>)
>>
>> We greatly benefited from setting this to a low value but we have quite a
>> lot of files on a node ( 30 million)
>> Note that setting this to zero will result in the OOM killer killing the
>> machine sooner or later.
>> (especially if files are moved around due to a cluster change ;)
>>
>> Cheers,
>> Robert van Leeuwen
>>
>
> Hi,
>     We set this to a low value(20) and the performance is better than
> before. It seems quite useful.
>
>     According to your description, this issue is related with the object
> quantity in the storage. We delete all the objects in the storage but it
> doesn't help anything. The only method to recover is to format and re-mount
> the storage node. We try to install swift on different environment but this
> degradation problem seems to be an inevitable one.
>
> Cheers,
> Jonathan Lu
>
>
> ______________________________**_________________
> Mailing list: https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
> More help   : https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>