yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #06862
[Bug 1258315] [NEW] Nova resize returns error code 255
Public bug reported:
When running "nova resize" command on an active VM, it completed without
any error. However the VM went into error state. The scheduler somehow
dispatched the job to an incorrect hypervisor (i.e. 10.52.224.45 instead
of actual hosting hypervisor wpc0043 which is 10.52.225.43) and returned
error code 255.
# nova list
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| 5eb01835-e41e-4bf3-9256-e8caa88686ff | cont011 | ACTIVE | None | Running | contpoc=10.52.225.114 |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
#nova show cont011
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| status | ACTIVE |
| updated | 2013-12-05T19:45:12Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | wpc0043.svc.eng.pdx.wd |
| key_name | None |
| image | centos64 (7cfa1455-0cd5-4e82-a7ec-418e25813415) |
| contpoc network | 10.52.225.114 |
| hostId | 9051224512ffaf16c9be843034687da080ab1e40f5e2fd57a5613d67 |
| OS-EXT-STS:vm_state | active |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001c |
| OS-SRV-USG:launched_at | 2013-12-05T19:47:07.000000 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | wpc0043.svc.eng.pdx.wd |
| flavor | m1.medium (3) |
| id | 5eb01835-e41e-4bf3-9256-e8caa88686ff |
| security_groups | [{u'name': u'default'}] |
| OS-SRV-USG:terminated_at | None |
| user_id | 35073614013e42d8908f23597c4e2c8c |
| name | cont011 |
| created | 2013-12-05T19:45:04Z |
| tenant_id | 544912394e764a099633ee4218ff822e |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| os-extended-volumes:volumes_attached | [] |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+--------------------------------------+----------------------------------------------------------+
[root@wpc0016 ~]# nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
# nova resize cont011 m1.large
[root@wpc0016 ~]# nova show cont011
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
| status | ERROR |
| updated | 2013-12-05T19:49:01Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | wpc0043.svc.eng.pdx.wd |
| key_name | None |
| image | centos64 (7cfa1455-0cd5-4e82-a7ec-418e25813415) |
| contpoc network | 10.52.225.114 |
| hostId | 9051224512ffaf16c9be843034687da080ab1e40f5e2fd57a5613d67 |
| OS-EXT-STS:vm_state | error |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001c |
| OS-SRV-USG:launched_at | 2013-12-05T19:47:07.000000 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | wpc0043.svc.eng.pdx.wd |
| flavor | m1.medium (3) |
| id | 5eb01835-e41e-4bf3-9256-e8caa88686ff |
| security_groups | [{u'name': u'default'}] |
| OS-SRV-USG:terminated_at | None |
| user_id | 35073614013e42d8908f23597c4e2c8c |
| name | cont011 |
| created | 2013-12-05T19:45:04Z |
| tenant_id | 544912394e764a099633ee4218ff822e |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| os-extended-volumes:volumes_attached | [] |
| accessIPv4 | |
| accessIPv6 | |
| fault | {u'message': u"Unexpected error while running command. |
| | Command: ssh 10.52.224.45 mkdir -p /var/lib/nova/instances/5eb01835-e41e-4bf3-9256-e8caa88686ff |
| | Exit code: 255 |
| | Stdout: '' |
| | Stderr: 'Host key verification failed.\\r\ |
| | '", u'code': 500, u'details': u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3021, in resize_instance |
| | block_device_info) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4439, in migrate_disk_and_power_off |
| | utils.execute(\'ssh\', dest, \'mkdir\', \'-p\', inst_base) |
| | File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute |
| | return processutils.execute(*cmd, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute |
| | cmd=\' \'.join(cmd)) |
| | ', u'created': u'2013-12-05T19:49:01Z'} |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
++++++++++++++++++++++++++++
The compute.log on the inetneded but incorrect hypervisor wpc0045 showed following logs.
2013-12-05 21:26:38.488 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000315: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000315/disk'
2013-12-05 21:26:38.495 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000053b: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000053b/disk'
2013-12-05 21:26:38.501 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000003d1: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000003d1/disk'
2013-12-05 21:26:38.508 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000455: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000455/disk'
2013-12-05 21:26:38.515 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000533: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000533/disk'
2013-12-05 21:26:38.521 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002db: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002db/disk'
2013-12-05 21:26:38.528 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002df: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002df/disk'
2013-12-05 21:26:38.535 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000033b: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000033b/disk'
2013-12-05 21:26:38.541 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000531: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000531/disk'
2013-12-05 21:26:38.548 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000053d: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000053d/disk'
2013-12-05 21:26:38.554 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000042f: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000042f/disk'
2013-12-05 21:26:38.561 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000054d: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000054d/disk'
2013-12-05 21:26:38.567 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000003f1: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000003f1/disk'
2013-12-05 21:26:38.574 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002d9: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002d9/disk'
2013-12-05 21:26:38.580 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000477: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000477/disk'
2013-12-05 21:26:38.586 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000547: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000547/disk'
2013-12-05 21:26:38.593 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000357: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000357/disk'
2013-12-05 21:26:38.599 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000535: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000535/disk'
2013-12-05 21:26:38.606 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002e1: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002e1/disk'
2013-12-05 21:26:38.612 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000537: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000537/disk'
2013-12-05 21:26:39.001 55931 WARNING nova.compute.resource_tracker [-] [instance: 5eb01835-e41e-4bf3-9256-e8caa88686ff] Instance not resizing, skipping migration.
2013-12-05 21:26:39.002 55931 WARNING nova.compute.resource_tracker [-] [instance: 5eb01835-e41e-4bf3-9256-e8caa88686ff] Instance not resizing, skipping migration.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1258315
Title:
Nova resize returns error code 255
Status in OpenStack Compute (Nova):
New
Bug description:
When running "nova resize" command on an active VM, it completed
without any error. However the VM went into error state. The scheduler
somehow dispatched the job to an incorrect hypervisor (i.e.
10.52.224.45 instead of actual hosting hypervisor wpc0043 which is
10.52.225.43) and returned error code 255.
# nova list
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| 5eb01835-e41e-4bf3-9256-e8caa88686ff | cont011 | ACTIVE | None | Running | contpoc=10.52.225.114 |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
#nova show cont011
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| status | ACTIVE |
| updated | 2013-12-05T19:45:12Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | wpc0043.svc.eng.pdx.wd |
| key_name | None |
| image | centos64 (7cfa1455-0cd5-4e82-a7ec-418e25813415) |
| contpoc network | 10.52.225.114 |
| hostId | 9051224512ffaf16c9be843034687da080ab1e40f5e2fd57a5613d67 |
| OS-EXT-STS:vm_state | active |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001c |
| OS-SRV-USG:launched_at | 2013-12-05T19:47:07.000000 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | wpc0043.svc.eng.pdx.wd |
| flavor | m1.medium (3) |
| id | 5eb01835-e41e-4bf3-9256-e8caa88686ff |
| security_groups | [{u'name': u'default'}] |
| OS-SRV-USG:terminated_at | None |
| user_id | 35073614013e42d8908f23597c4e2c8c |
| name | cont011 |
| created | 2013-12-05T19:45:04Z |
| tenant_id | 544912394e764a099633ee4218ff822e |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| os-extended-volumes:volumes_attached | [] |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+--------------------------------------+----------------------------------------------------------+
[root@wpc0016 ~]# nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
# nova resize cont011 m1.large
[root@wpc0016 ~]# nova show cont011
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
| status | ERROR |
| updated | 2013-12-05T19:49:01Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | wpc0043.svc.eng.pdx.wd |
| key_name | None |
| image | centos64 (7cfa1455-0cd5-4e82-a7ec-418e25813415) |
| contpoc network | 10.52.225.114 |
| hostId | 9051224512ffaf16c9be843034687da080ab1e40f5e2fd57a5613d67 |
| OS-EXT-STS:vm_state | error |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001c |
| OS-SRV-USG:launched_at | 2013-12-05T19:47:07.000000 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | wpc0043.svc.eng.pdx.wd |
| flavor | m1.medium (3) |
| id | 5eb01835-e41e-4bf3-9256-e8caa88686ff |
| security_groups | [{u'name': u'default'}] |
| OS-SRV-USG:terminated_at | None |
| user_id | 35073614013e42d8908f23597c4e2c8c |
| name | cont011 |
| created | 2013-12-05T19:45:04Z |
| tenant_id | 544912394e764a099633ee4218ff822e |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| os-extended-volumes:volumes_attached | [] |
| accessIPv4 | |
| accessIPv6 | |
| fault | {u'message': u"Unexpected error while running command. |
| | Command: ssh 10.52.224.45 mkdir -p /var/lib/nova/instances/5eb01835-e41e-4bf3-9256-e8caa88686ff |
| | Exit code: 255 |
| | Stdout: '' |
| | Stderr: 'Host key verification failed.\\r\ |
| | '", u'code': 500, u'details': u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3021, in resize_instance |
| | block_device_info) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4439, in migrate_disk_and_power_off |
| | utils.execute(\'ssh\', dest, \'mkdir\', \'-p\', inst_base) |
| | File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute |
| | return processutils.execute(*cmd, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute |
| | cmd=\' \'.join(cmd)) |
| | ', u'created': u'2013-12-05T19:49:01Z'} |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
++++++++++++++++++++++++++++
The compute.log on the inetneded but incorrect hypervisor wpc0045 showed following logs.
2013-12-05 21:26:38.488 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000315: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000315/disk'
2013-12-05 21:26:38.495 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000053b: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000053b/disk'
2013-12-05 21:26:38.501 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000003d1: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000003d1/disk'
2013-12-05 21:26:38.508 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000455: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000455/disk'
2013-12-05 21:26:38.515 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000533: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000533/disk'
2013-12-05 21:26:38.521 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002db: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002db/disk'
2013-12-05 21:26:38.528 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002df: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002df/disk'
2013-12-05 21:26:38.535 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000033b: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000033b/disk'
2013-12-05 21:26:38.541 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000531: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000531/disk'
2013-12-05 21:26:38.548 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000053d: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000053d/disk'
2013-12-05 21:26:38.554 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000042f: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000042f/disk'
2013-12-05 21:26:38.561 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000054d: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-0000054d/disk'
2013-12-05 21:26:38.567 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000003f1: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000003f1/disk'
2013-12-05 21:26:38.574 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002d9: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002d9/disk'
2013-12-05 21:26:38.580 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000477: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000477/disk'
2013-12-05 21:26:38.586 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000547: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000547/disk'
2013-12-05 21:26:38.593 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000357: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000357/disk'
2013-12-05 21:26:38.599 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000535: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000535/disk'
2013-12-05 21:26:38.606 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-000002e1: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-000002e1/disk'
2013-12-05 21:26:38.612 55931 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-00000537: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000537/disk'
2013-12-05 21:26:39.001 55931 WARNING nova.compute.resource_tracker [-] [instance: 5eb01835-e41e-4bf3-9256-e8caa88686ff] Instance not resizing, skipping migration.
2013-12-05 21:26:39.002 55931 WARNING nova.compute.resource_tracker [-] [instance: 5eb01835-e41e-4bf3-9256-e8caa88686ff] Instance not resizing, skipping migration.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1258315/+subscriptions
Follow ups
References
