yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #53212
[Bug 1227143] Re: InstanceInfoCache is not always updated with concurrent instance creation
This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.
If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
Valid example: CONFIRMED FOR: LIBERTY
** Changed in: nova
Status: Confirmed => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1227143
Title:
InstanceInfoCache is not always updated with concurrent instance
creation
Status in OpenStack Compute (nova):
Expired
Bug description:
Hi,
I try to launch 45 instances in parallel :
nova boot --image ${image_id} --flavor ${flavor_id} --nic net-id=${net_id} --num-instances 45 jordan-internet
Sometimes, nova fails to get the IP address of one or 2 instances :
>nova list
+--------------------------------------+------------------------------------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------------------------------------+--------+------------+-------------+------------------------+
| 31d9c78b-1d03-46a8-98eb-47143679e1c8 | jordan-internet-31d9c78b-1d03-46a8-98eb-47143679e1c8 | ACTIVE | None | Running | internet=194.2.202.177 |
| 3301a06c-60c0-48a7-97b1-4ee292ccf294 | jordan-internet-3301a06c-60c0-48a7-97b1-4ee292ccf294 | ACTIVE | None | Running | internet=194.2.202.193 |
| 357b3366-4918-4d09-8c52-4ce5a1c02156 | jordan-internet-357b3366-4918-4d09-8c52-4ce5a1c02156 | ACTIVE | None | Running | |
| 35af1894-cb5d-4797-a0aa-88df32436351 | jordan-internet-35af1894-cb5d-4797-a0aa-88df32436351 | ACTIVE | None | Running | internet=194.2.202.171 |
| 2a5afa62-68c1-4343-ba40-7fb0021d2a23 | jordan-internet-2a5afa62-68c1-4343-ba40-7fb0021d2a23 | ACTIVE | None | Running | internet=194.2.202.153 |
| 30d14347-f260-4f6f-a55a-d2e5549f84e1 | jordan-internet-30d14347-f260-4f6f-a55a-d2e5549f84e1 | ACTIVE | None | Running | internet=194.2.202.188 |
| 318a9c90-3573-4ba2-a73a-d65a3d79a2b2 | jordan-internet-318a9c90-3573-4ba2-a73a-d65a3d79a2b2 | ACTIVE | None | Running | internet=194.2.202.176 |
>nova show 357b3366-4918-4d09-8c52-4ce5a1c02156
+-------------------------------------+-------------------------------------------------------------------+
| Property | Value |
+-------------------------------------+-------------------------------------------------------------------+
| status | ACTIVE |
| updated | 2013-09-18T13:21:49Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | d-ocnclc-0001 |
| key_name | jordan |
| image | Ubuntu precise 12.04.2 LTS (cecc8e4c-e689-40f9-80de-9a43219c9c0b) |
| hostId | bf54cba4f3fdc7b0b8c1b1872de977c770fb332c2d395e7acd370a4b |
| OS-EXT-STS:vm_state | active |
| OS-EXT-SRV-ATTR:instance_name | instance-00004ab6 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | d-ocnclc-0001.adm.dev1.val.cw-labs.net |
| flavor | m1.small (6) |
| id | 357b3366-4918-4d09-8c52-4ce5a1c02156 |
| security_groups | [{u'name': u'default'}] |
| user_id | bbdcffe9d3944c35bcf6a875b6d3d235 |
| name | jordan-internet-357b3366-4918-4d09-8c52-4ce5a1c02156 |
| created | 2013-09-18T13:18:22Z |
| tenant_id | 462d7cbd479f499fa75ad9f14571d553 |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | alpha1.val |
| config_drive | |
+-------------------------------------+-------------------------------------------------------------------+
What's weird is that nova interfast-list is "correct" :
jordan@jpi-octo:~$ nova interface-list 357b3366-4918-4d09-8c52-4ce5a1c02156
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+
| ACTIVE | d5b85cae-623e-4e77-a09a-3c057af6b4c7 | 7b07d545-53da-4897-abbd-dfe142db0fc2 | 194.2.202.181 | fa:16:3e:2f:9b:51 |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+
And quantum port-show is also 'correct' :
jordan@jpi-octo:~$ quantum port-show d5b85cae-623e-4e77-a09a-3c057af6b4c7
+----------------------+--------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+--------------------------------------------------------------------------------------+
| admin_state_up | True |
| binding:capabilities | {"port_filter": true} |
| binding:vif_type | ovs |
| device_id | 357b3366-4918-4d09-8c52-4ce5a1c02156 |
| device_owner | compute:None |
| fixed_ips | {"subnet_id": "32ce548d-dd26-48f9-b7a0-78ea69f4750c", "ip_address": "194.2.202.181"} |
| id | d5b85cae-623e-4e77-a09a-3c057af6b4c7 |
| mac_address | fa:16:3e:2f:9b:51 |
| name | |
| network_id | 7b07d545-53da-4897-abbd-dfe142db0fc2 |
| security_groups | 865ca78f-d712-4828-8ec8-b53f7c3f5523 |
| status | ACTIVE |
| tenant_id | 462d7cbd479f499fa75ad9f14571d553 |
+----------------------+--------------------------------------------------------------------------------------+
So the problem lies in the SQL column 'network_info' of the table
nova.instance_info. For a problematic instance, this column remains
set as '[]' whereas for a 'correct' instance, the column contains some
useful JSON data related to network
I cannot reproduce this issue when I launch let's say 5 instances in
parallel or if I launch 45 instances one after another.
Any help would be very much appreciated. I don't fully understand what
does compute/utils.py::get_nw_info_for_instance() nor
network/quantumv2/api.py::_build_network_info_model() and
network/quantumv2/api.py::allocate_for_instance() :(
Thanks,
Jordan
Setup :
Grizzly 2013.1.3
Nova-api with osapi_compute_workers=10
Quantum with openvswitch-plugin
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1227143/+subscriptions
