openstack team mailing list archive

[Phil Holden <Phil.Holden@xxxxxxxxxxxxxxxxx>]

Hello,

In 
    https://lists.launchpad.net/openstack/msg16969.html
John Dickinson wrote 

> The 404s on object PUTs are probably related to the timeout errors you 
> are seeing on the container servers. This may be because of IO 
> contention on your hardware (eg overtaxed drives). How does the disk IO 
> look on your physical hardware?

I should have clarified that one or two of the threads, where each thread 
has its own container to work on, gets stuck like this, while the other 
threads continue OK.  For a transient problem on one container it will 
retry and eventually continue.  If it was an overload issue I would 
expect it to move from container to container, but all threads would make
some kind of progress as they retry.  

> The disk full errors may be because you are running out of inodes on 
> the filesystem. You can check this with `df -i`. This is possible if 
> you are using many small files.

This occurred with the filesystems configured with 
    mkfs.xfs -i size=1024,maxpct=0
maxpct=0 is unlimited inodes.  The filesystem is mounted with the inode64 
option, although as each filesystem in this test system is only 10 Gbytes 
32-bit inode numbers should be adequate.  df on the filesystems reported 
59% space usage and 31% inode usage (+/- 1% or 2%).  

As ext4, the filesystems are configured with 5 million inodes, but only 
~25% of them are used as the filesystems get near to full, so this is a 
bit over-provisioned.  


                 Regards
                   - Phil -



NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU.  UK. Company number 02723032.  This e-mail message and any attachment is confidential. It may not be disclosed to or used by anyone other than the intended recipient. If you have received this e-mail in error please notify the sender immediately then delete it from your system. Whilst every effort has been made to check this mail is virus free we accept no responsibility for software viruses and you should check for viruses before opening any attachments. Opinions, conclusions and other information in this email and any attachments which do not relate to the official business of the company are neither given by the company nor endorsed by it.

This email message has been scanned for viruses by Mimecast