yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #85679
[Bug 1921804] [NEW] leftover bdm when rabbitmq unstable
Public bug reported:
Description
===========
When rabbitMQ unstable, there might be a chance that method
https://github.com/openstack/nova/blob/7a1222a8654684262a8e589d91e67f2b9a9da336/nova/compute/api.py#L4741
will timeout but bdm is successfully created.
Under such cases, volume will be shown in server show, but cannot be
detached. and volume status is available.
Steps to reproduce
==================
there might be no way to safely reproduce this failure, because when rabbitmq is
unstable, many other services will also show unusual behavior.
Expected result
===============
We should be able to remove such attachment from api without manually fixing db...
```console
root@mgt02:~# openstack server show 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e
+-------------------------------------+-------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | cn-north-3a |
| OS-EXT-SRV-ATTR:host | compute01 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute01 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000ce4c |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2021-03-29T09:06:38.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | newsql-net=192.168.1.217; service_mgt=100.114.3.41 |
| config_drive | True |
| created | 2021-03-29T09:05:19Z |
| flavor | newsql_2C8G40G_general (51db3192-cece-4b9a-9969-7916b4543beb) |
| hostId | cf1f3937a3286677b3020d817541ac33d7c8f1ca74be49b26f128093 |
| id | 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e |
| image | newsql-bini2.0.0alpha-ubuntu18.04-x64-20210112-pub (4531e3bf-0433-40c6-816b-6763f9d02c7a) |
| key_name | None |
| name | NewSQL-1abc5b28-b9e6-45cd-893d-5bb3a7732a43-3 |
| progress | 0 |
| project_id | acfcc87fc1db430880f0cb1cce410906 |
| properties | productTag='NewSQL' |
| security_groups | name='default' |
| | name='csf-NewSQL-cluster-security-group' |
| status | ACTIVE |
| updated | 2021-03-29T09:06:39Z |
| user_id | a38ef24677cc4a45a143a31c5fb59ee9 |
| volumes_attached | id='1351f861-55a7-4dc5-bf5c-31963657876d' |
+-------------------------------------+-------------------------------------------------------------------------------------------+
root@mgt02:~# openstack server remove volume 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e 1351f861-55a7-4dc5-bf5c-31963657876d
Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-2ca68e00-760d-47c2-913e-b0fe54e86296) (HTTP 400) (Request-ID: req-264aea62-a970-4241-ab14-ea7aad74cd6e)
```
Actual result
=============
openstack server remove volume failed
Environment
===========
1. stable/victoria in k8s
2. qemu+kvm
2. ceph rbd
3. neutron ml2 openvswitch
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
Description
===========
When rabbitMQ unstable, there might be a chance that method
https://github.com/openstack/nova/blob/7a1222a8654684262a8e589d91e67f2b9a9da336/nova/compute/api.py#L4741
will timeout but bdm is successfully created.
- Under such cases, such volume will be shown in server show, but cannot
- be detached.
+ Under such cases, volume will be shown in server show, but cannot be
+ detached. and volume status is available.
Steps to reproduce
==================
there might be no way to safely reproduce this failure, because when rabbitmq is
unstable, many other services will also show unusual behavior.
Expected result
===============
We should be able to remove such attachment from api without manually fixing db...
```console
root@mgt02:~# openstack server show 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e
+-------------------------------------+-------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | cn-north-3a |
| OS-EXT-SRV-ATTR:host | compute01 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute01 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000ce4c |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2021-03-29T09:06:38.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | newsql-net=192.168.1.217; service_mgt=100.114.3.41 |
| config_drive | True |
| created | 2021-03-29T09:05:19Z |
| flavor | newsql_2C8G40G_general (51db3192-cece-4b9a-9969-7916b4543beb) |
| hostId | cf1f3937a3286677b3020d817541ac33d7c8f1ca74be49b26f128093 |
| id | 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e |
| image | newsql-bini2.0.0alpha-ubuntu18.04-x64-20210112-pub (4531e3bf-0433-40c6-816b-6763f9d02c7a) |
| key_name | None |
| name | NewSQL-1abc5b28-b9e6-45cd-893d-5bb3a7732a43-3 |
| progress | 0 |
| project_id | acfcc87fc1db430880f0cb1cce410906 |
| properties | productTag='NewSQL' |
| security_groups | name='default' |
| | name='csf-NewSQL-cluster-security-group' |
| status | ACTIVE |
| updated | 2021-03-29T09:06:39Z |
| user_id | a38ef24677cc4a45a143a31c5fb59ee9 |
| volumes_attached | id='1351f861-55a7-4dc5-bf5c-31963657876d' |
+-------------------------------------+-------------------------------------------------------------------------------------------+
root@mgt02:~# openstack server remove volume 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e 1351f861-55a7-4dc5-bf5c-31963657876d
Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-2ca68e00-760d-47c2-913e-b0fe54e86296) (HTTP 400) (Request-ID: req-264aea62-a970-4241-ab14-ea7aad74cd6e)
```
Actual result
=============
openstack server remove volume failed
Environment
===========
1. stable/victoria in k8s
2. qemu+kvm
2. ceph rbd
3. neutron ml2 openvswitch
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1921804
Title:
leftover bdm when rabbitmq unstable
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
When rabbitMQ unstable, there might be a chance that method
https://github.com/openstack/nova/blob/7a1222a8654684262a8e589d91e67f2b9a9da336/nova/compute/api.py#L4741
will timeout but bdm is successfully created.
Under such cases, volume will be shown in server show, but cannot be
detached. and volume status is available.
Steps to reproduce
==================
there might be no way to safely reproduce this failure, because when rabbitmq is
unstable, many other services will also show unusual behavior.
Expected result
===============
We should be able to remove such attachment from api without manually fixing db...
```console
root@mgt02:~# openstack server show 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e
+-------------------------------------+-------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | cn-north-3a |
| OS-EXT-SRV-ATTR:host | compute01 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute01 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000ce4c |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2021-03-29T09:06:38.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | newsql-net=192.168.1.217; service_mgt=100.114.3.41 |
| config_drive | True |
| created | 2021-03-29T09:05:19Z |
| flavor | newsql_2C8G40G_general (51db3192-cece-4b9a-9969-7916b4543beb) |
| hostId | cf1f3937a3286677b3020d817541ac33d7c8f1ca74be49b26f128093 |
| id | 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e |
| image | newsql-bini2.0.0alpha-ubuntu18.04-x64-20210112-pub (4531e3bf-0433-40c6-816b-6763f9d02c7a) |
| key_name | None |
| name | NewSQL-1abc5b28-b9e6-45cd-893d-5bb3a7732a43-3 |
| progress | 0 |
| project_id | acfcc87fc1db430880f0cb1cce410906 |
| properties | productTag='NewSQL' |
| security_groups | name='default' |
| | name='csf-NewSQL-cluster-security-group' |
| status | ACTIVE |
| updated | 2021-03-29T09:06:39Z |
| user_id | a38ef24677cc4a45a143a31c5fb59ee9 |
| volumes_attached | id='1351f861-55a7-4dc5-bf5c-31963657876d' |
+-------------------------------------+-------------------------------------------------------------------------------------------+
root@mgt02:~# openstack server remove volume 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e 1351f861-55a7-4dc5-bf5c-31963657876d
Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-2ca68e00-760d-47c2-913e-b0fe54e86296) (HTTP 400) (Request-ID: req-264aea62-a970-4241-ab14-ea7aad74cd6e)
```
Actual result
=============
openstack server remove volume failed
Environment
===========
1. stable/victoria in k8s
2. qemu+kvm
2. ceph rbd
3. neutron ml2 openvswitch
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1921804/+subscriptions
Follow ups
