yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #15482
[Bug 1328700] [NEW] On mass deletion - some VM's stuck on ERROR due to connection failed to neutron
Public bug reported:
Description of problem:
When doing mass deletion (more then 64 VM's parallely), some VM's got stuck on ERROR state with the error:
"Connection to neutron failed: Maximum attempts reached", "code": 500, "created": "2014-06-10T21:34:38Z"
Version-Release number of selected component (if applicable):
openstack-neutron-openvswitch-2014.1-26.el7ost.noarch
python-neutronclient-2.3.4-2.el7ost.noarch
python-neutron-2014.1-26.el7ost.noarch
openstack-neutron-ml2-2014.1-26.el7ost.noarch
openstack-neutron-2014.1-26.el7ost.noarch
How reproducible:
100%
Steps to Reproduce:
1. Setup an environment with a lot of ACTIVE VM's (more then 64)
2. Run mass deletion
for each in `nova list | grep ACTIVE | cut -d"|" -f3` ; do nova delete $each & done
Actual results:
[root@cougar16 ~(keystone_stress1)]$ nova list
+--------------------------------------+------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+-----------------------+
| 0731ab43-99da-409a-99b9-627287b0a80a | stress1-42 | ERROR | deleting | Running | private1=192.168.1.61 |
+--------------------------------------+------------+--------+------------+-------------+-----------------------+
[root@cougar16 ~(keystone_stress1)]$ nova show stress1-42
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | deleting |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2014-06-10T21:28:53.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-06-10T21:27:33Z |
| fault | {"message": "Connection to neutron failed: Maximum attempts reached", "code": 500, "created": "2014-06-10T21:34:38Z"} |
| flavor | mini (0) |
| hostId | 0c295b885647eb08a3c04a15eb86f9746430dd635c5f8c6291315508 |
| id | 0731ab43-99da-409a-99b9-627287b0a80a |
| image | cirros (ae31ea8c-c5ca-4ca1-9662-3545304d8e79) |
| key_name | - |
| metadata | {} |
| name | stress1-42 |
| os-extended-volumes:volumes_attached | [] |
| private1 network | 192.168.1.61 |
| security_groups | default |
| status | ERROR |
| tenant_id | 1f160feb483943249381963bc9e87eee |
| updated | 2014-06-10T21:34:21Z |
| user_id | 9630e22ff0bc410aaee6bb217b660fee |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
Expected results:
no previously ACTIVE VM's available
Additional info:
1. [root@cougar16 ~(keystone_stress1)]$ nova flavor-show mini
+----------------------------+-------+
| Property | Value |
+----------------------------+-------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 0 |
| extra_specs | {} |
| id | 0 |
| name | mini |
| os-flavor-access:is_public | True |
| ram | 64 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+-------+
2. [root@cougar16 ~(keystone_stress1)]$ nova image-show cirros
+----------------------+--------------------------------------+
| Property | Value |
+----------------------+--------------------------------------+
| OS-EXT-IMG-SIZE:size | 13147648 |
| created | 2014-06-10T18:03:36Z |
| id | ae31ea8c-c5ca-4ca1-9662-3545304d8e79 |
| minDisk | 0 |
| minRam | 0 |
| name | cirros |
| progress | 100 |
| status | ACTIVE |
| updated | 2014-06-10T18:03:44Z |
+----------------------+--------------------------------------+
3. Environment has 2 api workers and 2 rpc workers
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1328700
Title:
On mass deletion - some VM's stuck on ERROR due to connection failed
to neutron
Status in OpenStack Neutron (virtual network service):
New
Bug description:
Description of problem:
When doing mass deletion (more then 64 VM's parallely), some VM's got stuck on ERROR state with the error:
"Connection to neutron failed: Maximum attempts reached", "code": 500, "created": "2014-06-10T21:34:38Z"
Version-Release number of selected component (if applicable):
openstack-neutron-openvswitch-2014.1-26.el7ost.noarch
python-neutronclient-2.3.4-2.el7ost.noarch
python-neutron-2014.1-26.el7ost.noarch
openstack-neutron-ml2-2014.1-26.el7ost.noarch
openstack-neutron-2014.1-26.el7ost.noarch
How reproducible:
100%
Steps to Reproduce:
1. Setup an environment with a lot of ACTIVE VM's (more then 64)
2. Run mass deletion
for each in `nova list | grep ACTIVE | cut -d"|" -f3` ; do nova delete $each & done
Actual results:
[root@cougar16 ~(keystone_stress1)]$ nova list
+--------------------------------------+------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+-----------------------+
| 0731ab43-99da-409a-99b9-627287b0a80a | stress1-42 | ERROR | deleting | Running | private1=192.168.1.61 |
+--------------------------------------+------------+--------+------------+-------------+-----------------------+
[root@cougar16 ~(keystone_stress1)]$ nova show stress1-42
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | deleting |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2014-06-10T21:28:53.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-06-10T21:27:33Z |
| fault | {"message": "Connection to neutron failed: Maximum attempts reached", "code": 500, "created": "2014-06-10T21:34:38Z"} |
| flavor | mini (0) |
| hostId | 0c295b885647eb08a3c04a15eb86f9746430dd635c5f8c6291315508 |
| id | 0731ab43-99da-409a-99b9-627287b0a80a |
| image | cirros (ae31ea8c-c5ca-4ca1-9662-3545304d8e79) |
| key_name | - |
| metadata | {} |
| name | stress1-42 |
| os-extended-volumes:volumes_attached | [] |
| private1 network | 192.168.1.61 |
| security_groups | default |
| status | ERROR |
| tenant_id | 1f160feb483943249381963bc9e87eee |
| updated | 2014-06-10T21:34:21Z |
| user_id | 9630e22ff0bc410aaee6bb217b660fee |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
Expected results:
no previously ACTIVE VM's available
Additional info:
1. [root@cougar16 ~(keystone_stress1)]$ nova flavor-show mini
+----------------------------+-------+
| Property | Value |
+----------------------------+-------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 0 |
| extra_specs | {} |
| id | 0 |
| name | mini |
| os-flavor-access:is_public | True |
| ram | 64 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+-------+
2. [root@cougar16 ~(keystone_stress1)]$ nova image-show cirros
+----------------------+--------------------------------------+
| Property | Value |
+----------------------+--------------------------------------+
| OS-EXT-IMG-SIZE:size | 13147648 |
| created | 2014-06-10T18:03:36Z |
| id | ae31ea8c-c5ca-4ca1-9662-3545304d8e79 |
| minDisk | 0 |
| minRam | 0 |
| name | cirros |
| progress | 100 |
| status | ACTIVE |
| updated | 2014-06-10T18:03:44Z |
+----------------------+--------------------------------------+
3. Environment has 2 api workers and 2 rpc workers
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1328700/+subscriptions
Follow ups
References
