yahoo-eng-team team mailing list archive

[Taihsiang Ho <1880828@xxxxxxxxxxxxxxxxxx>]

** Also affects: charm-nova-compute
   Importance: Undecided
       Status: New

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1880828

Title:
  New instance is always in "spawning" status

Status in OpenStack nova-compute charm:
  New
Status in OpenStack Compute (nova):
  New
Status in OpenStack Bundles:
  New

Bug description:
  bundle: openstack-base-bionic-train https://github.com/openstack-charmers/openstack-bundles/blob/master/development/openstack-base-bionic-train/bundle.yaml
  hardware: 2 d05 and 2 d06 (the log of the compute node is from one of the d06)

  When trying to create new instances on the deployed openstack, the
  instance is always in the status of "spawning"

  [Steps to Reproduce]
  1. Deploy with the above bundle and hardware by following the instruction of https://jaas.ai/openstack-base/bundle/67
  2. Wait about 1.5 until the deployment is ready. By ready it means every unit shows its message as "ready" e.g. https://paste.ubuntu.com/p/k48YVnPyVZ/
  3. Follow the instruction of https://jaas.ai/openstack-base/bundle/67 until the step of "openstack server create" to create new instance. This step is also summarized in details in this gist code snippet https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-02-basic-test-just-deployed-sh

  [Expected Behavior]
  An instance is created a few second later

  [Actual Behavior]
  The status of the instance is always (> 20 minutes) "spawning"

  [Additional Information]

  1. By waiting for 0.5 ~ 1 hr first, and then restart the nova-compute
  by "sudo service nova-compute restart", may make the creation behavior
  back to normal. You may refer to this snippet to see this workaround
  https://gist.github.com/tai271828/b0c00a611e703046dd52da12a66226b0#file-03
  -nova-compute-debugging-sh

  2. By applying the above workaround and then check log from the
  corresponding nova-compute node, you could find lines like:

   818 2020-05-26 02:33:17.845 417168 INFO nova.compute.claims [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7f4     d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Claim successful on node kreiken.maas
   819 2020-05-26 02:33:18.657 417168 INFO nova.virt.libvirt.driver [req-9b570186-d136-4df5-8b90-6dd3e8b67f3e 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9     fe7f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 54e49495-c1c2-4567-8493-2c412d676e5c] Creating image
   820 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [req-318355e9-213c-466b-bb9a-f8170ee7ff95 91f35a24c0b1465fbd147d70e439646e e16f574f6448430aac827c968e5145be - e61235e9fe7     f4d0284f035c7588dd2b9 e61235e9fe7f4d0284f035c7588dd2b9] [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Instance failed to spawn: nova.exception.ImageUnacceptable: Image e9149aae     -246a-4eb9-963e-d7f25286ba7a is unacceptable: Unable to convert image to raw: Image /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part is unacceptable: Un     able to convert image to raw: Unexpected error while running command.
   821 Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a6372     26c8ffded93a57489.converted
   822 Exit code: -15
   823 Stdout: ''
   824 Stderr: ''
   825 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Traceback (most recent call last):
   826 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf]   File "/usr/lib/python3/dist-packages/nova/virt/images.py", line 128, i     n _convert_image
   827 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf]     compress)
   828 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf]   File "/usr/lib/python3/dist-packages/nova/privsep/qemu.py", line 73, i     n unprivileged_convert_image
   829 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf]     processutils.execute(*cmd)
   830 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf]   File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py"     , line 424, in execute
   831 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf]     cmd=sanitized_cmd)
   832 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error wh     ile running command.
   833 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Command: qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance     s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted
   834 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Exit code: -15
   835 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stdout: ''
   836 2020-05-26 02:33:30.871 417168 ERROR nova.compute.manager [instance: 1f9d4cb9-f540-4e33-b36e-23e71268abcf] Stderr: ''

  
  3. Before restarting any nova-compute service manually, if you try to issue this command on the compute node, the command will also never end:

  qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instance
  s/_base/9b8156fbecaa194804a637226c8ffded93a57489.part
  /var/lib/nova/instances/_base/9b8156fbecaa194804a637226c8ffded93a57489.converted

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-compute/+bug/1880828/+subscriptions