yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #38283
[Bug 1493783] [NEW] nova-compute ceph "too many open files"
Public bug reported:
We’ve deployed OpenStack with Ceph as a storage (rbd). The instances are
being boot using cinder volumes through Ceph too.
Sometimes the following ERROR appears at the nova-compute.log: [Errno
24] Too many open files
If we get the list of open files from any nova-compute process, we’ll
see the following:
root@qk-3:~# ps -auxwww | grep nova-compute
nova 137430 2.4 0.0 2395468 106888 ? Ssl Sep07 65:54 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
root@qk-3:~# lsof -p 137430 | wc -l
5406
Almost all open files is:
nova-comp 137430 nova 5232r FIFO 0,8 0t0 948057084 pipe
nova-comp 137430 nova 5233u unix 0xffff881ad3192300 0t0 948012913 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5234r FIFO 0,8 0t0 948057088 pipe
nova-comp 137430 nova 5235u unix 0xffff881ad3192680 0t0 948057085 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5236r FIFO 0,8 0t0 948073475 pipe
nova-comp 137430 nova 5237u unix 0xffff881ad3192d80 0t0 948073473 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5238r FIFO 0,8 0t0 948073522 pipe
nova-comp 137430 nova 5239u unix 0xffff881ad3195080 0t0 948073476 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5240r FIFO 0,8 0t0 948073526 pipe
nova-comp 137430 nova 5241u unix 0xffff881ad3197700 0t0 948073523 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5242r FIFO 0,8 0t0 948073532 pipe
nova-comp 137430 nova 5243u unix 0xffff881ad3196c80 0t0 948073527 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5244r FIFO 0,8 0t0 948073535 pipe
nova-comp 137430 nova 5245u unix 0xffff881ad3196580 0t0 948073533 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5246r FIFO 0,8 0t0 948163602 pipe
nova-comp 137430 nova 5247u unix 0xffff881ad3195b00 0t0 948073536 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5248r FIFO 0,8 0t0 948160472 pipe
nova-comp 137430 nova 5249u unix 0xffff88301f2ec600 0t0 948163603 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5250r FIFO 0,8 0t0 948160475 pipe
nova-comp 137430 nova 5251u unix 0xffff881ad3190000 0t0 948160473 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5252r FIFO 0,8 0t0 948160490 pipe
nova-comp 137430 nova 5253u unix 0xffff881ad3190380 0t0 948160476 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5254r FIFO 0,8 0t0 948160493 pipe
nova-comp 137430 nova 5255u unix 0xffff881ad3195780 0t0 948160491 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5256r FIFO 0,8 0t0 948160498 pipe
nova-comp 137430 nova 5257u unix 0xffff881fafe18000 0t0 948160494 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5258r FIFO 0,8 0t0 948261835 pipe
nova-comp 137430 nova 5259u unix 0xffff881fafe18e00 0t0 948160499 /var/run/ceph/guests/ceph-client.cinder.137430.35666576.asok
nova-comp 137430 nova 5260r FIFO 0,8 0t0 948265362 pipe
nova-comp 137430 nova 5261u unix 0xffff881fafe19180 0t0 948261836 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5262r FIFO 0,8 0t0 948265500 pipe
nova-comp 137430 nova 5263u unix 0xffff881fafe19500 0t0 948265363 /var/run/ceph/guests/ceph-client.cinder.137430.35318896.asok
And:
root@qk-3:~# ls -la /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok
ls: cannot access /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok: No such file or directory
As you can realize, all of these files were already closed.
If we restart nova-compute process, the number of open file will be
140-150, but within a few days it grows up to 4000-5000-10000 files and
we get the ”too many open files" Error again.
Below is the ceph.conf from nova-compute node:
[global]
fsid = 91ef7ba7-463b-4410-8128-82217052d2dd
mon_initial_members = ceph-1, ceph-2, ns
mon_host = 192.168.2.4,192.168.2.5,192.168.2.6
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 3
public_network = 192.168.2.0/24
cluster_network = 192.168.1.0/24
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = false
rbd_cache_size = 2 GiB
rbd_cache_max_dirty = 2 GiB
rbd_cache_target_dirty = 256 MiB
rbd_cache_max_dirty_age = 1.0
admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/qemu/qemu-guest-$pid.log
rbd concurrent management ops = 20
Part of nova-compute:
disk_cachemodes="network=writeback"
inject_password = false
inject_key = false
inject_partition = -2
hw_disk_discard = unmap
root@qk-3:~# dpkg -l | grep nova
ii nova-common 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 1:2.22.0-0ubuntu1~cloud0 all client library for OpenStack Compute API
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
We’ve deployed OpenStack with Ceph as a storage (rbd). The instances are
being boot using cinder volumes through Ceph too.
Sometimes the following ERROR appears at the nova-compute.log: [Errno
24] Too many open files
If we get the list of open files from any nova-compute process, we’ll
see the following:
root@qk-3:~# ps -auxwww | grep nova-compute
nova 137430 2.4 0.0 2395468 106888 ? Ssl Sep07 65:54 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
root@qk-3:~# lsof -p 137430 | wc -l
5406
Almost all open files is:
nova-comp 137430 nova 5232r FIFO 0,8 0t0 948057084 pipe
nova-comp 137430 nova 5233u unix 0xffff881ad3192300 0t0 948012913 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5234r FIFO 0,8 0t0 948057088 pipe
nova-comp 137430 nova 5235u unix 0xffff881ad3192680 0t0 948057085 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5236r FIFO 0,8 0t0 948073475 pipe
nova-comp 137430 nova 5237u unix 0xffff881ad3192d80 0t0 948073473 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5238r FIFO 0,8 0t0 948073522 pipe
nova-comp 137430 nova 5239u unix 0xffff881ad3195080 0t0 948073476 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5240r FIFO 0,8 0t0 948073526 pipe
nova-comp 137430 nova 5241u unix 0xffff881ad3197700 0t0 948073523 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5242r FIFO 0,8 0t0 948073532 pipe
nova-comp 137430 nova 5243u unix 0xffff881ad3196c80 0t0 948073527 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5244r FIFO 0,8 0t0 948073535 pipe
nova-comp 137430 nova 5245u unix 0xffff881ad3196580 0t0 948073533 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5246r FIFO 0,8 0t0 948163602 pipe
nova-comp 137430 nova 5247u unix 0xffff881ad3195b00 0t0 948073536 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5248r FIFO 0,8 0t0 948160472 pipe
nova-comp 137430 nova 5249u unix 0xffff88301f2ec600 0t0 948163603 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5250r FIFO 0,8 0t0 948160475 pipe
nova-comp 137430 nova 5251u unix 0xffff881ad3190000 0t0 948160473 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5252r FIFO 0,8 0t0 948160490 pipe
nova-comp 137430 nova 5253u unix 0xffff881ad3190380 0t0 948160476 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5254r FIFO 0,8 0t0 948160493 pipe
nova-comp 137430 nova 5255u unix 0xffff881ad3195780 0t0 948160491 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5256r FIFO 0,8 0t0 948160498 pipe
nova-comp 137430 nova 5257u unix 0xffff881fafe18000 0t0 948160494 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5258r FIFO 0,8 0t0 948261835 pipe
nova-comp 137430 nova 5259u unix 0xffff881fafe18e00 0t0 948160499 /var/run/ceph/guests/ceph-client.cinder.137430.35666576.asok
nova-comp 137430 nova 5260r FIFO 0,8 0t0 948265362 pipe
nova-comp 137430 nova 5261u unix 0xffff881fafe19180 0t0 948261836 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5262r FIFO 0,8 0t0 948265500 pipe
nova-comp 137430 nova 5263u unix 0xffff881fafe19500 0t0 948265363 /var/run/ceph/guests/ceph-client.cinder.137430.35318896.asok
And:
root@qk-3:~# ls -la /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok
ls: cannot access /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok: No such file or directory
As you can realize, all of these files were already closed.
If we restart nova-compute process, the number of open file will be
- 140-150, but within a few days it grows up to 4000-500-10000 files and
+ 140-150, but within a few days it grows up to 4000-5000-10000 files and
we get the ”too many open files" Error again.
-
Below is the ceph.conf from nova-compute node:
[global]
fsid = 91ef7ba7-463b-4410-8128-82217052d2dd
mon_initial_members = ceph-1, ceph-2, ns
mon_host = 192.168.2.4,192.168.2.5,192.168.2.6
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 3
public_network = 192.168.2.0/24
cluster_network = 192.168.1.0/24
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = false
rbd_cache_size = 2 GiB
rbd_cache_max_dirty = 2 GiB
rbd_cache_target_dirty = 256 MiB
rbd_cache_max_dirty_age = 1.0
admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/qemu/qemu-guest-$pid.log
rbd concurrent management ops = 20
Part of nova-compute:
disk_cachemodes="network=writeback"
inject_password = false
inject_key = false
inject_partition = -2
hw_disk_discard = unmap
root@qk-3:~# dpkg -l | grep nova
ii nova-common 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 1:2.22.0-0ubuntu1~cloud0 all client library for OpenStack Compute API
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1493783
Title:
nova-compute ceph "too many open files"
Status in OpenStack Compute (nova):
New
Bug description:
We’ve deployed OpenStack with Ceph as a storage (rbd). The instances
are being boot using cinder volumes through Ceph too.
Sometimes the following ERROR appears at the nova-compute.log: [Errno
24] Too many open files
If we get the list of open files from any nova-compute process, we’ll
see the following:
root@qk-3:~# ps -auxwww | grep nova-compute
nova 137430 2.4 0.0 2395468 106888 ? Ssl Sep07 65:54 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
root@qk-3:~# lsof -p 137430 | wc -l
5406
Almost all open files is:
nova-comp 137430 nova 5232r FIFO 0,8 0t0 948057084 pipe
nova-comp 137430 nova 5233u unix 0xffff881ad3192300 0t0 948012913 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5234r FIFO 0,8 0t0 948057088 pipe
nova-comp 137430 nova 5235u unix 0xffff881ad3192680 0t0 948057085 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5236r FIFO 0,8 0t0 948073475 pipe
nova-comp 137430 nova 5237u unix 0xffff881ad3192d80 0t0 948073473 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5238r FIFO 0,8 0t0 948073522 pipe
nova-comp 137430 nova 5239u unix 0xffff881ad3195080 0t0 948073476 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5240r FIFO 0,8 0t0 948073526 pipe
nova-comp 137430 nova 5241u unix 0xffff881ad3197700 0t0 948073523 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5242r FIFO 0,8 0t0 948073532 pipe
nova-comp 137430 nova 5243u unix 0xffff881ad3196c80 0t0 948073527 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5244r FIFO 0,8 0t0 948073535 pipe
nova-comp 137430 nova 5245u unix 0xffff881ad3196580 0t0 948073533 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5246r FIFO 0,8 0t0 948163602 pipe
nova-comp 137430 nova 5247u unix 0xffff881ad3195b00 0t0 948073536 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5248r FIFO 0,8 0t0 948160472 pipe
nova-comp 137430 nova 5249u unix 0xffff88301f2ec600 0t0 948163603 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5250r FIFO 0,8 0t0 948160475 pipe
nova-comp 137430 nova 5251u unix 0xffff881ad3190000 0t0 948160473 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5252r FIFO 0,8 0t0 948160490 pipe
nova-comp 137430 nova 5253u unix 0xffff881ad3190380 0t0 948160476 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5254r FIFO 0,8 0t0 948160493 pipe
nova-comp 137430 nova 5255u unix 0xffff881ad3195780 0t0 948160491 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok
nova-comp 137430 nova 5256r FIFO 0,8 0t0 948160498 pipe
nova-comp 137430 nova 5257u unix 0xffff881fafe18000 0t0 948160494 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok
nova-comp 137430 nova 5258r FIFO 0,8 0t0 948261835 pipe
nova-comp 137430 nova 5259u unix 0xffff881fafe18e00 0t0 948160499 /var/run/ceph/guests/ceph-client.cinder.137430.35666576.asok
nova-comp 137430 nova 5260r FIFO 0,8 0t0 948265362 pipe
nova-comp 137430 nova 5261u unix 0xffff881fafe19180 0t0 948261836 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok
nova-comp 137430 nova 5262r FIFO 0,8 0t0 948265500 pipe
nova-comp 137430 nova 5263u unix 0xffff881fafe19500 0t0 948265363 /var/run/ceph/guests/ceph-client.cinder.137430.35318896.asok
And:
root@qk-3:~# ls -la /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok
ls: cannot access /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok: No such file or directory
As you can realize, all of these files were already closed.
If we restart nova-compute process, the number of open file will be
140-150, but within a few days it grows up to 4000-5000-10000 files
and we get the ”too many open files" Error again.
Below is the ceph.conf from nova-compute node:
[global]
fsid = 91ef7ba7-463b-4410-8128-82217052d2dd
mon_initial_members = ceph-1, ceph-2, ns
mon_host = 192.168.2.4,192.168.2.5,192.168.2.6
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 3
public_network = 192.168.2.0/24
cluster_network = 192.168.1.0/24
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = false
rbd_cache_size = 2 GiB
rbd_cache_max_dirty = 2 GiB
rbd_cache_target_dirty = 256 MiB
rbd_cache_max_dirty_age = 1.0
admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/qemu/qemu-guest-$pid.log
rbd concurrent management ops = 20
Part of nova-compute:
disk_cachemodes="network=writeback"
inject_password = false
inject_key = false
inject_partition = -2
hw_disk_discard = unmap
root@qk-3:~# dpkg -l | grep nova
ii nova-common 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 1:2.22.0-0ubuntu1~cloud0 all client library for OpenStack Compute API
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1493783/+subscriptions
Follow ups
