yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #19866
[Bug 1365259] [NEW] Lack of failure information (Failure due to "host key verification failure")displayed, during instance migration from one host to another
Public bug reported:
I am using (1 controller + compute1 + compute 2) openstack environment.
During live migrate server from one compute host to another using CLI -
If migration is failed due to host key verification failure from one
compute host to another, then failure information should be as a output
to console. otherwise user will not be able to know what is happening.
For user, migration is successful but actually it is failed.
Set of operation is as -
1.
root@nechldcst-PowerEdge-2950:# nova list
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| 1aea212b-0bee-498b-a10d-5b58a69e3293 | test-server | ACTIVE | - | Running | demo-net=203.0.113.26 |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
2.
root@nechldcst-PowerEdge-2950:# nova migrate 1aea212b-0bee-498b-a10d-5b58a69e3293
root@nechldcst-PowerEdge-2950:#
>>>>>>> At this point user thinks that migration is successful but see below -
3.
root@nechldcst-PowerEdge-2950:# nova list
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| 1aea212b-0bee-498b-a10d-5b58a69e3293 | test-server | ERROR | - | Running | demo-net=203.0.113.26 |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
4.
root@nechldcst-PowerEdge-2950:# nova show 1aea212b-0bee-498b-a10d-5b58a69e3293
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute2 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2014-09-04T03:41:08.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-09-04T03:41:06Z |
| demo-net network | 203.0.113.26 |
| fault | {"message": "Unexpected error while running command. |
| | Command: ssh 10.0.3.23 mkdir -p /var/lib/nova/instances/1aea212b-0bee-498b-a10d-5b58a69e3293 |
| | Exit code: 255 |
| | Stdout: '' |
| | Stderr: 'Host key verification failed.\\r\ |
| | '", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 290, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 3459, in resize_instance |
| | block_device_info) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 4945, in migrate_disk_and_power_off |
| | utils.execute('ssh', dest, 'mkdir', '-p', inst_base) |
| | File \"/usr/lib/python2.7/dist-packages/nova/utils.py\", line 165, in execute |
| | return processutils.execute(*cmd, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py\", line 193, in execute |
| | cmd=' '.join(cmd)) |
| | ", "created": "2014-09-04T03:42:01Z"} |
| flavor | m1.small (2) |
| hostId | 57ce883a190fe7261854233ace00ca32da46f1d5da7e7d0d3174c79c |
| id | 1aea212b-0bee-498b-a10d-5b58a69e3293 |
| image | cirros-0.3.2-x86_64 (831f7dab-42c4-4bf2-a60e-e8fe9b692381) |
| key_name | - |
| metadata | {} |
| name | test-server |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default |
| status | ERROR |
| tenant_id | a1c37cc595024369aa2124b50adaa0b8 |
| updated | 2014-09-04T03:42:01Z |
| user_id | 0c0e03c722dc4cdfbe4d7fbb4d2b2a21 |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
root@nechldcst-PowerEdge-2950:#
What compute logs say - /var/log/nova/nova-comute.log
Command: ssh 10.0.3.23 mkdir -p /var/lib/nova/instances/1aea212b-0bee-498b-a10d-5b58a69e3293
Exit code: 255
Stdout: ''
Stderr: 'Host key verification failed.\r\n' to caller
2014-09-04 09:12:01.678 19604 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function\n pass\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function\n function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 248, in decorated_function\n migration.instance_uuid, exc_info=True)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 235, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3459, in resize_instance\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4945, in migrate_disk_and_power_off\n utils.execute(\'ssh\', dest, \'mkdir\', \'-p\', inst_base)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 193, in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError: Unexpected error while running command.\nCommand: ssh 10.0.3.23 mkdir -p /var/lib/nova/instances/1aea212b-0bee-498b-a10d-5b58a69e3293\nExit code: 255\nStdout: ''\nStderr: 'Host key verification failed.\\r\\n'\n"]
2014-09-04 09:12:14.166 19604 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
So the idea is that, there should be some mechanism which handles and gives proper failure details when migrate command is issued.
** Affects: nova
Importance: Undecided
Status: New
** Summary changed:
- Lack of failure information (Failure due to host key verification fail)displayed, during instance migration from one host to another
+ Lack of failure information (Failure due to "host key verification failure")displayed, during instance migration from one host to another
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1365259
Title:
Lack of failure information (Failure due to "host key verification
failure")displayed, during instance migration from one host to another
Status in OpenStack Compute (Nova):
New
Bug description:
I am using (1 controller + compute1 + compute 2) openstack
environment.
During live migrate server from one compute host to another using CLI
-
If migration is failed due to host key verification failure from one
compute host to another, then failure information should be as a
output to console. otherwise user will not be able to know what is
happening.
For user, migration is successful but actually it is failed.
Set of operation is as -
1.
root@nechldcst-PowerEdge-2950:# nova list
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| 1aea212b-0bee-498b-a10d-5b58a69e3293 | test-server | ACTIVE | - | Running | demo-net=203.0.113.26 |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
2.
root@nechldcst-PowerEdge-2950:# nova migrate 1aea212b-0bee-498b-a10d-5b58a69e3293
root@nechldcst-PowerEdge-2950:#
>>>>>>> At this point user thinks that migration is successful but see below -
3.
root@nechldcst-PowerEdge-2950:# nova list
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| 1aea212b-0bee-498b-a10d-5b58a69e3293 | test-server | ERROR | - | Running | demo-net=203.0.113.26 |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
4.
root@nechldcst-PowerEdge-2950:# nova show 1aea212b-0bee-498b-a10d-5b58a69e3293
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute2 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2014-09-04T03:41:08.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-09-04T03:41:06Z |
| demo-net network | 203.0.113.26 |
| fault | {"message": "Unexpected error while running command. |
| | Command: ssh 10.0.3.23 mkdir -p /var/lib/nova/instances/1aea212b-0bee-498b-a10d-5b58a69e3293 |
| | Exit code: 255 |
| | Stdout: '' |
| | Stderr: 'Host key verification failed.\\r\ |
| | '", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 290, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 3459, in resize_instance |
| | block_device_info) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 4945, in migrate_disk_and_power_off |
| | utils.execute('ssh', dest, 'mkdir', '-p', inst_base) |
| | File \"/usr/lib/python2.7/dist-packages/nova/utils.py\", line 165, in execute |
| | return processutils.execute(*cmd, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py\", line 193, in execute |
| | cmd=' '.join(cmd)) |
| | ", "created": "2014-09-04T03:42:01Z"} |
| flavor | m1.small (2) |
| hostId | 57ce883a190fe7261854233ace00ca32da46f1d5da7e7d0d3174c79c |
| id | 1aea212b-0bee-498b-a10d-5b58a69e3293 |
| image | cirros-0.3.2-x86_64 (831f7dab-42c4-4bf2-a60e-e8fe9b692381) |
| key_name | - |
| metadata | {} |
| name | test-server |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default |
| status | ERROR |
| tenant_id | a1c37cc595024369aa2124b50adaa0b8 |
| updated | 2014-09-04T03:42:01Z |
| user_id | 0c0e03c722dc4cdfbe4d7fbb4d2b2a21 |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
root@nechldcst-PowerEdge-2950:#
What compute logs say - /var/log/nova/nova-comute.log
Command: ssh 10.0.3.23 mkdir -p /var/lib/nova/instances/1aea212b-0bee-498b-a10d-5b58a69e3293
Exit code: 255
Stdout: ''
Stderr: 'Host key verification failed.\r\n' to caller
2014-09-04 09:12:01.678 19604 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function\n pass\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function\n function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 248, in decorated_function\n migration.instance_uuid, exc_info=True)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 235, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3459, in resize_instance\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4945, in migrate_disk_and_power_off\n utils.execute(\'ssh\', dest, \'mkdir\', \'-p\', inst_base)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 165, in execute\n return processutils.execute(*cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 193, in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError: Unexpected error while running command.\nCommand: ssh 10.0.3.23 mkdir -p /var/lib/nova/instances/1aea212b-0bee-498b-a10d-5b58a69e3293\nExit code: 255\nStdout: ''\nStderr: 'Host key verification failed.\\r\\n'\n"]
2014-09-04 09:12:14.166 19604 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
So the idea is that, there should be some mechanism which handles and gives proper failure details when migrate command is issued.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1365259/+subscriptions
Follow ups
References
