openstack team mailing list archive

[Kuo Hugo <tonytkdk@xxxxxxxxx>]

Hi Huang,

Storage nodes will run out of memory for caching inode eventually right?
 Have you ever measure the upper limit of caching capability of your
storage nodes?


Hi Jonathan,

The default reclaiming time is 7 days. Did you wait for 7 days or just
change the setting to 1 min in conf file ?

Hugo

+Hugo Kuo+
hugo@xxxxxxxxxxxxxx
tonytkdk@xxxxxxxxx
+886 935004793


2013/6/18 Jonathan Lu <jojokururu@xxxxxxxxx>

>  Hi Hugo,
>     I know the tombstone mechanism. In my opinion after the reclaim time,
> the object of xxx.tombstone will be deleted at all. Is that right? Maybe I
> misunderstand the doc :( ...
>     We try to "colddown" the swift system ( just wait for the reclaiming)
> and test, but the result is not satisfying.
>
> Thanks,
> Jonathan Lu
>
>
> On 2013/6/18 11:04, Kuo Hugo wrote:
>
> Hi Jonathan ,
>
>  How did you perform "delete all the objects in the storage" ?  Those
> deleted objects still consume inodes in tombstone status until the reclaim
> time.
> Would you mind to compare the result of $> sudo cat /proc/slabinfo | grep
> xfs   ,  before/after set the vfs_cache_pressure
>
>  spongebob@patrick1:~$ sudo cat /proc/slabinfo | grep xfs
> xfs_ili                70153  70182    216   18    1 : tunables    0    0
>    0 : slabdata   3899   3899      0
>  xfs_inode         169738 170208   1024   16    4 : tunables    0    0
>  0 : slabdata  10638  10638      0
> xfs_efd_item          60     60    400   20    2 : tunables    0    0    0
> : slabdata      3      3      0
> xfs_buf_item         234    234    224   18    1 : tunables    0    0    0
> : slabdata     13     13      0
> xfs_trans             28     28    280   14    1 : tunables    0    0    0
> : slabdata      2      2      0
> xfs_da_state          32     32    488   16    2 : tunables    0    0    0
> : slabdata      2      2      0
> xfs_btree_cur         38     38    208   19    1 : tunables    0    0    0
> : slabdata      2      2      0
> xfs_log_ticket        40     40    200   20    1 : tunables    0    0    0
> : slabdata      2      2      0
>
>
>  Hi Robert,
> The performance degradation still there even only main swift workers are
> running in storage node. ( stop replicator/updater/auditor ). In my
> knowing.
> I'll check xs_dir_lookup and xs_ig_missed here. Thanks
>
>
>
>
>
>
>
>  +Hugo Kuo+
> hugo@xxxxxxxxxxxxxx
>  tonytkdk@xxxxxxxxx
>  +886 935004793
>
>
> 2013/6/18 Jonathan Lu <jojokururu@xxxxxxxxx>
>
>> On 2013/6/17 18:59, Robert van Leeuwen wrote:
>>
>>>  I'm facing the issue about the performance degradation, and once I
>>>> glanced that changing the value in /proc/sys
>>>> /vm/vfs_cache_pressure will do a favour.
>>>> Can anyone explain to me whether and why it is useful?
>>>>
>>> Hi,
>>>
>>> When this is set to a lower value the kernel will try to keep the
>>> inode/dentry cache longer in memory.
>>> Since the swift replicator is scanning the filesystem continuously it
>>> will eat up a lot of iops if those are not in memory.
>>>
>>> To see if a lot of cache misses are happening, for xfs, you can look at
>>> xs_dir_lookup and xs_ig_missed.
>>> ( look at http://xfs.org/index.php/Runtime_Stats )
>>>
>>> We greatly benefited from setting this to a low value but we have quite
>>> a lot of files on a node ( 30 million)
>>> Note that setting this to zero will result in the OOM killer killing the
>>> machine sooner or later.
>>> (especially if files are moved around due to a cluster change ;)
>>>
>>> Cheers,
>>> Robert van Leeuwen
>>>
>>
>>  Hi,
>>     We set this to a low value(20) and the performance is better than
>> before. It seems quite useful.
>>
>>     According to your description, this issue is related with the object
>> quantity in the storage. We delete all the objects in the storage but it
>> doesn't help anything. The only method to recover is to format and re-mount
>> the storage node. We try to install swift on different environment but this
>> degradation problem seems to be an inevitable one.
>>
>> Cheers,
>> Jonathan Lu
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>