yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1680363] [NEW] Raise Unexpected API Error when get console log from stopped instance

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rui Chen <chenrui.momo@xxxxxxxxx>
Date: Thu, 06 Apr 2017 08:25:36 -0000
Reply-to: Bug 1680363 <1680363@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

(openstack) console log show chenrui_vm
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.UnboundLocalError'> (HTTP 500) (Request-ID: req-26d4eed4-316f-4d3c-9d2f-040f855ee940)
(openstack) server list
+--------------------------------------+------------+---------+--------------------------------------------------------+--------------------------+
| ID                                   | Name       | Status  | Networks                                               | Image Name               |
+--------------------------------------+------------+---------+--------------------------------------------------------+--------------------------+
| 2cf68078-52f7-4f15-92dc-7333927220f1 | chenrui_vm | SHUTOFF | private=fdb4:4f0f:960b:0:f816:3eff:fe4b:25c3, 10.0.0.8 | cirros-0.3.5-x86_64-disk |
+--------------------------------------+------------+---------+--------------------------------------------------------+--------------------------+

n-cpu.log

nd /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:473
2017-04-06 16:19:09.814 17016 DEBUG oslo_messaging._drivers.amqpdriver [req-26d4eed4-316f-4d3c-9d2f-040f855ee940 admin admin] CAST unique_id: ada9ebc67fe04118b1c191362bf53d7b NOTIFY exchange 'nova' topic 'versioned_notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:473
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server [req-26d4eed4-316f-4d3c-9d2f-040f855ee940 admin admin] Exception during message handling
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 77, in wrapped
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 68, in wrapped
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4543, in get_console_output
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     output = self.driver.get_console_output(context, instance)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2801, in get_console_output
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     data = self._flush_libvirt_console(pty)
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server UnboundLocalError: local variable 'pty' referenced before assignment
2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server
2017-04-06 16:19:09.817 17016 DEBUG oslo_messaging._drivers.amqpdriver [req-26d4eed4-316f-4d3c-9d2f-040f855ee940 admin admin] sending reply msg_id: 9f78f1c6f0af419ca73ae288b2082902 reply queue: reply_d1293bd1e962444caa9999b71ff92cf2 time elapsed: 0.602079593111s _send_reply /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:90


stack@szxbzci0004 ~/logs $ virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     instance-00000006              shut off

stack@szxbzci0004 ~/logs $ virsh dumpxml instance-00000006
...
    <interface type='bridge'>
      <mac address='fa:16:3e:4b:25:c3'/>
      <source bridge='qbr86d9c781-31'/>
      <target dev='tap86d9c781-31'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <log file='/opt/stack/data/nova/instances/2cf68078-52f7-4f15-92dc-7333927220f1/console.log' append='off'/>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <log file='/opt/stack/data/nova/instances/2cf68078-52f7-4f15-92dc-7333927220f1/console.log' append='off'/>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>

...

stack@szxbzci0004 ~/logs $ virsh version
Compiled against library: libvirt 2.5.0
Using library: libvirt 2.5.0
Using API: QEMU 2.5.0
Running hypervisor: QEMU 2.8.0

stack@szxbzci0004 ~/nova (master) $ git log -1
commit 53296a07027ea434cdae90709c7298ad2fb346c6
Merge: 7660d0c f828f6d
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Wed Apr 5 22:08:00 2017 +0000

    Merge "docs: update description for
AggregateInstanceExtraSpecsFilter"

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1680363

Title:
  Raise Unexpected API Error when get console log from stopped instance

Status in OpenStack Compute (nova):
  New

Bug description:
  (openstack) console log show chenrui_vm
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.UnboundLocalError'> (HTTP 500) (Request-ID: req-26d4eed4-316f-4d3c-9d2f-040f855ee940)
  (openstack) server list
  +--------------------------------------+------------+---------+--------------------------------------------------------+--------------------------+
  | ID                                   | Name       | Status  | Networks                                               | Image Name               |
  +--------------------------------------+------------+---------+--------------------------------------------------------+--------------------------+
  | 2cf68078-52f7-4f15-92dc-7333927220f1 | chenrui_vm | SHUTOFF | private=fdb4:4f0f:960b:0:f816:3eff:fe4b:25c3, 10.0.0.8 | cirros-0.3.5-x86_64-disk |
  +--------------------------------------+------------+---------+--------------------------------------------------------+--------------------------+

  n-cpu.log

  nd /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:473
  2017-04-06 16:19:09.814 17016 DEBUG oslo_messaging._drivers.amqpdriver [req-26d4eed4-316f-4d3c-9d2f-040f855ee940 admin admin] CAST unique_id: ada9ebc67fe04118b1c191362bf53d7b NOTIFY exchange 'nova' topic 'versioned_notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:473
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server [req-26d4eed4-316f-4d3c-9d2f-040f855ee940 admin admin] Exception during message handling
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 77, in wrapped
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 68, in wrapped
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4543, in get_console_output
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     output = self.driver.get_console_output(context, instance)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2801, in get_console_output
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server     data = self._flush_libvirt_console(pty)
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server UnboundLocalError: local variable 'pty' referenced before assignment
  2017-04-06 16:19:09.816 17016 ERROR oslo_messaging.rpc.server
  2017-04-06 16:19:09.817 17016 DEBUG oslo_messaging._drivers.amqpdriver [req-26d4eed4-316f-4d3c-9d2f-040f855ee940 admin admin] sending reply msg_id: 9f78f1c6f0af419ca73ae288b2082902 reply queue: reply_d1293bd1e962444caa9999b71ff92cf2 time elapsed: 0.602079593111s _send_reply /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:90

  
  stack@szxbzci0004 ~/logs $ virsh list --all
   Id    Name                           State
  ----------------------------------------------------
   -     instance-00000006              shut off

  stack@szxbzci0004 ~/logs $ virsh dumpxml instance-00000006
  ...
      <interface type='bridge'>
        <mac address='fa:16:3e:4b:25:c3'/>
        <source bridge='qbr86d9c781-31'/>
        <target dev='tap86d9c781-31'/>
        <model type='virtio'/>
        <driver name='qemu'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <serial type='pty'>
        <log file='/opt/stack/data/nova/instances/2cf68078-52f7-4f15-92dc-7333927220f1/console.log' append='off'/>
        <target port='0'/>
      </serial>
      <console type='pty'>
        <log file='/opt/stack/data/nova/instances/2cf68078-52f7-4f15-92dc-7333927220f1/console.log' append='off'/>
        <target type='serial' port='0'/>
      </console>
      <input type='mouse' bus='ps2'/>
      <input type='keyboard' bus='ps2'/>

  ...

  stack@szxbzci0004 ~/logs $ virsh version
  Compiled against library: libvirt 2.5.0
  Using library: libvirt 2.5.0
  Using API: QEMU 2.5.0
  Running hypervisor: QEMU 2.8.0

  stack@szxbzci0004 ~/nova (master) $ git log -1
  commit 53296a07027ea434cdae90709c7298ad2fb346c6
  Merge: 7660d0c f828f6d
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Wed Apr 5 22:08:00 2017 +0000

      Merge "docs: update description for
  AggregateInstanceExtraSpecsFilter"

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1680363/+subscriptions
Follow ups

[Bug 1680363] Re: Raise Unexpected API Error when get console log from stopped instance
From: OpenStack Infra, 2017-04-13