yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1956602] [NEW] cinder_volume docker down due to Resource temporarily unavailable error message

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rahul <1956602@xxxxxxxxxxxxxxxxxx>
Date: Thu, 06 Jan 2022 15:37:06 -0000
Reply-to: Bug 1956602 <1956602@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Brief Description
----------------------------------------------------------------------------------------------------------------------------------------------
Issue:
After deployment of Openstack Victoria release created multiple VM's on both computes node (compute0 and compute1) because of this resource allocation is high on the openstack cluster, there is less available memory left on both the compute nodes

#Memory details on compute0
[root@Comp0 ~]# free -g
              total        used        free      shared  buff/cache   available
Mem:            314         297           9           0           7           6
Swap:            31          28           3
[root@Comp0 ~]#

#Memory details on compute1
[root@Comp1 ~]# free -g
              total        used        free      shared  buff/cache   available
Mem:            314         298           8           0           7           5
Swap:            31          23           8
[root@Comp1 ~]#

Recently issue occured on the comp0 node in which cinder_volume docker
service is in Exit state and when trying to start docker service for
cinder_volume OCI runtime create failed" Error message occurred.

[root@Comp0 ~]# docker ps -a | grep cinder
eb54440db7f7   kolla/centos-binary-cinder-backup:victoria               "dumb-init --single-…"   3 months ago   Up 49 minutes                        cinder_backup
b99bc1f7e761   kolla/centos-binary-cinder-volume:victoria               "dumb-init --single-…"   3 months ago   Exited (255) 2 hours ago             cinder_volume

[root@Comp0 ~]# docker start cinder_volume
Error response from daemon: OCI runtime create failed: container with id exists: b99bc1f7e7619a2d428bc4aeb15976ec84d3970be5261429c07e5e736c97554d: unknown
Error: failed to start containers: cinder_volume


We have identified the cause and applied the required solution.
While debugging the issue found analysis docker logs and found the following 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Running command: 'cinder-volume --config-file /etc/cinder/cinder.conf'
OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 4 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 5 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 6 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 8 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 9 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 11 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 12 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 13 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 14 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 15 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 16 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 17 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 18 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 19 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 20 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 24 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 25 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 26 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 27 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 28 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 29 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 30 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 31 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 32 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 33 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 34 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 35 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 36 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 37 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 40: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max


Thread::try_create(): pthread_create failed with error 11/builddir/build/BUILD/ceph-14.2.22/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f822e92bb80 time 2021-12-30 11:59:22.464232
/builddir/build/BUILD/ceph-14.2.22/src/common/Thread.cc: 157: FAILED ceph_assert(ret == 0)
 ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Through these logs not able to exactly understand which resources these logs referring to. 
We were not be able to find the exact root cause of this problem, Kindly please share your valuable inputs regarding this issue.

** Affects: cinder
     Importance: Undecided
         Status: New

** Project changed: nova => cinder

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1956602

Title:
  cinder_volume docker down due to Resource temporarily unavailable
  error message

Status in Cinder:
  New

Bug description:
  Brief Description
  ----------------------------------------------------------------------------------------------------------------------------------------------
  Issue:
  After deployment of Openstack Victoria release created multiple VM's on both computes node (compute0 and compute1) because of this resource allocation is high on the openstack cluster, there is less available memory left on both the compute nodes

  #Memory details on compute0
  [root@Comp0 ~]# free -g
                total        used        free      shared  buff/cache   available
  Mem:            314         297           9           0           7           6
  Swap:            31          28           3
  [root@Comp0 ~]#

  #Memory details on compute1
  [root@Comp1 ~]# free -g
                total        used        free      shared  buff/cache   available
  Mem:            314         298           8           0           7           5
  Swap:            31          23           8
  [root@Comp1 ~]#

  Recently issue occured on the comp0 node in which cinder_volume docker
  service is in Exit state and when trying to start docker service for
  cinder_volume OCI runtime create failed" Error message occurred.

  [root@Comp0 ~]# docker ps -a | grep cinder
  eb54440db7f7   kolla/centos-binary-cinder-backup:victoria               "dumb-init --single-…"   3 months ago   Up 49 minutes                        cinder_backup
  b99bc1f7e761   kolla/centos-binary-cinder-volume:victoria               "dumb-init --single-…"   3 months ago   Exited (255) 2 hours ago             cinder_volume

  [root@Comp0 ~]# docker start cinder_volume
  Error response from daemon: OCI runtime create failed: container with id exists: b99bc1f7e7619a2d428bc4aeb15976ec84d3970be5261429c07e5e736c97554d: unknown
  Error: failed to start containers: cinder_volume

  
  We have identified the cause and applied the required solution.
  While debugging the issue found analysis docker logs and found the following 
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  Running command: 'cinder-volume --config-file /etc/cinder/cinder.conf'
  OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 4 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 5 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 6 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 8 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 9 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 11 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 12 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 13 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 14 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 15 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 16 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 17 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 18 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 19 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 20 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 24 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 25 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 26 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 27 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 28 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 29 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 30 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 31 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 32 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 33 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 34 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 35 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 36 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 37 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
  OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 40: Resource temporarily unavailable
  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max

  
  Thread::try_create(): pthread_create failed with error 11/builddir/build/BUILD/ceph-14.2.22/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f822e92bb80 time 2021-12-30 11:59:22.464232
  /builddir/build/BUILD/ceph-14.2.22/src/common/Thread.cc: 157: FAILED ceph_assert(ret == 0)
   ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

  Through these logs not able to exactly understand which resources these logs referring to. 
  We were not be able to find the exact root cause of this problem, Kindly please share your valuable inputs regarding this issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1956602/+subscriptions