yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #01828
[Bug 1160489] Re: periodically seeing KeyError: 'old_instance_type_memory_mb' in compute.log
Reviewed: https://review.openstack.org/25521
Committed: http://github.com/openstack/nova/commit/1fa7879071310fda7d8393716441dd02be38fc30
Submitter: Jenkins
Branch: milestone-proposed
commit 1fa7879071310fda7d8393716441dd02be38fc30
Author: Dan Smith <danms@xxxxxxxxxx>
Date: Tue Mar 26 13:14:56 2013 -0700
Always store old instance_type during a migration
Previously, we were trying to optimize for the migration (as opposed
to resize) case by not storing the old_ instance_type. However,
this is confusing to the vast amount of shared code that works on
migrations and resizes because the "is old == new" logic must be
replicated everywhere. To cut our losses and hopefully avoid other
hidden dragons, this patch removes the optimization and makes both
paths the same.
Fixes bug #1160489
Change-Id: I2ad7b02239baada5fd2174f5b61cdc90ec91e6f5
(cherry picked from commit 985d703d2731fe9ec968ef6698bbf39743ede32a)
** Changed in: nova
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1160489
Title:
periodically seeing KeyError: 'old_instance_type_memory_mb' in
compute.log
Status in OpenStack Compute (Nova):
Fix Released
Bug description:
When running multiple migrations concurrently, I see an error the
repeats in the compute.log file. The trace can be seen here:
http://pastebin.com/XRr70jpU
Here is the steps I followed to recreate this issue:
(1). Boot the instances
nova boot --image be8b6475-26d8-410f-aaa5-8b278d98c8f9 --flavor 1 MIGRATE1
nova boot --image be8b6475-26d8-410f-aaa5-8b278d98c8f9 --flavor 1 MIGRATE2
nova boot --image be8b6475-26d8-410f-aaa5-8b278d98c8f9 --flavor 1 MIGRATE3
[root@vs347 ~]# nova list
+--------------------------------------+----------+--------+-------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+--------+-------------------+
| ccf6212b-bb8d-4826-97db-e18b4d4cb2b0 | MIGRATE1 | ACTIVE | demonet=172.0.0.2 |
| 739a6d71-cc63-4a34-b14f-7aef6f635a31 | MIGRATE2 | ACTIVE | demonet=172.0.0.4 |
| 11dabeb8-2799-40e1-a355-4d58f935a9c6 | MIGRATE3 | ACTIVE | demonet=172.0.0.3 |
+--------------------------------------+----------+--------+-------------------+
bvt-instance-00000029 - olyblade02
bvt-instance-0000002a - olyblade02
bvt-instance-0000002b - olyblade02
(2). migrate the instances simultaneously
[root@vs347 ~]# nova migrate MIGRATE1
[root@vs347 ~]# nova migrate MIGRATE2
[root@vs347 ~]# nova migrate MIGRATE3
[root@vs347 ~]#
* State as of 4:24 p.m.
+--------------------------------------+----------+--------+-------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+--------+-------------------+
| ccf6212b-bb8d-4826-97db-e18b4d4cb2b0 | MIGRATE1 | RESIZE | demonet=172.0.0.2 |
| 739a6d71-cc63-4a34-b14f-7aef6f635a31 | MIGRATE2 | RESIZE | demonet=172.0.0.4 |
| 11dabeb8-2799-40e1-a355-4d58f935a9c6 | MIGRATE3 | RESIZE | demonet=172.0.0.3 |
+--------------------------------------+----------+--------+-------------------+
* Now they are in resize-verify state so do confirm
[root@vs347 ~]# nova resize-confirm MIGRATE1
[root@vs347 ~]# nova resize-confirm MIGRATE2
[root@vs347 ~]# nova resize-confirm MIGRATE3
[root@vs347 ~]#
[root@vs347 ~]# nova list
+--------------------------------------+----------+--------+-------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+--------+-------------------+
| ccf6212b-bb8d-4826-97db-e18b4d4cb2b0 | MIGRATE1 | ERROR | demonet=172.0.0.2 |
| 739a6d71-cc63-4a34-b14f-7aef6f635a31 | MIGRATE2 | ACTIVE | demonet=172.0.0.4 |
| 11dabeb8-2799-40e1-a355-4d58f935a9c6 | MIGRATE3 | ACTIVE | demonet=172.0.0.3 |
+--------------------------------------+----------+--------+-------------------+
[root@vs347 ~]# nova show MIGRATE1
+-------------------------------------+------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-------------------------------------+------------------------------------------------------------------------------------------------------------------------+
| status | ERROR |
| updated | 2013-03-22T21:52:21Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | vs347.rch.kstart.ibm.com |
| key_name | None |
| image | Rhel6MasterFile (be8b6475-26d8-410f-aaa5-8b278d98c8f9) |
| hostId | a3883219ffb68903fc2ecc1e0a490f3f250fdf3773f446bcb398441c |
| OS-EXT-STS:vm_state | error |
| OS-EXT-SRV-ATTR:instance_name | bvt-instance-00000029 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | olyblade01.rch.stglabs.ibm.com |
| flavor | m1.tiny (1) |
| demonet network | 172.0.0.2 |
| id | ccf6212b-bb8d-4826-97db-e18b4d4cb2b0 |
| security_groups | [{u'name': u'default'}] |
| user_id | 3ccf55fd609b45319f24fe681338886d |
| name | MIGRATE1 |
| created | 2013-03-22T21:19:00Z |
| tenant_id | 67b1c37f4ca64283908c7077e9e59997 |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| fault | {u'message': u'KeyError', u'code': 500, u'details': u'\'old_instance_type_memory_mb\' |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1956, in confirm_resize |
| | rt.confirm_resize(context, migration) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 373, in confirm_resize |
| | self.update_available_resource(elevated) |
| | File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner |
| | retval = f(*args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 269, in update_available_resource |
| | self._update_usage_from_migrations(resources, migrations) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 468, in _update_usage_from_migrations |
| | migration) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 428, in _update_usage_from_migration |
| | itype = instance_types.extract_instance_type(instance, \'old_\') |
| | File "/usr/lib/python2.6/site-packages/nova/compute/instance_types.py", line 250, in extract_instance_type |
| | instance_type[key] = type_fn(sys_meta[type_key]) |
| | ', u'created': u'2013-03-22T21:52:21Z'} |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+-------------------------------------+------------------------------------------------------------------------------------------------------------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1160489/+subscriptions
