yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #79856
[Bug 1843008] [NEW] Nova will not detect when extending volume online fails
Public bug reported:
Description
===========
Extending volume in-use may fail, in that case, CLI "nova instance-action-list" should show ERROR in the coulume of message.However , when we catch exceptions in /nova/compute/manager.py
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
nova extends volume will fail ,however CLI "nova instance-action vmid req-id" returns success,that's wrong
===========
1.create volume-backend VM
compute]# nova boot --flavor 1 --nic net-id=27105f4f-5847-4538-862d-6565ad002c9e --boot-volume 53bc2318-ae6c-42b5-ad3b-f197f405cf1e VM
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | vm |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-1jrun74v |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | TbPSQRBic9NQ |
| config_drive | |
| created | 2019-09-06T06:41:55Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.tiny |
| flavor:ram | 512 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| host_status | |
| id | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| locked_reason | - |
| metadata | {} |
| name | VM |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 92cd1d753cf54f3496fda649212cc37b |
| trusted_image_certificates | - |
| updated | 2019-09-06T06:41:55Z |
| user_id | 85c12fb6eada44bc879da10ea394832b |
+--------------------------------------+-------------------------------------------------+
2.Setting specific scenarios to trigger exceptions in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
for example :insert 'extended_volume_id=1' in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8656-L8658 ,
pass objects.BlockDeviceMapping.get_by_volume_and_instance invalid extended_volume_id
3. extend instance's volume
root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 2 | lvmdriver-1 | true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
[root@controller compute]# cinder extend 53bc2318-ae6c-42b5-ad3b-f197f405cf1e 3
[root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 3 | lvmdriver-1 | true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
4.volume extend successfully in cinder , check extending result in nova, it shows success,In fact,extending volume in nova failed
[root@controller compute]# nova instance-action-list fe3c831c-95af-4f0b-b32a-31fb1e425462
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| Action | Request_ID | Message | Start_Time | Updated_At |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| create | req-c3220c13-1de3-435d-a765-c1e931025974 | - | 2019-09-06T06:41:54.000000 | 2019-09-06T06:42:02.000000 |
| extend_volume | req-9c233e67-0105-4a76-86aa-36fdd59593dd | - | 2019-09-06T06:42:42.000000 | 2019-09-06T06:42:42.000000 |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
[root@controller compute]# nova instance-action fe3c831c-95af-4f0b-b32a-31fb1e425462 req-9c233e67-0105-4a76-86aa-36fdd59593dd
+---------------+---------------------------------------------------------------------------+
| Property | Value |
+---------------+---------------------------------------------------------------------------+
| action | extend_volume |
| events | [{u'event': u'compute_extend_volume', |
| | u'finish_time': u'2019-09-06T06:42:42.000000', |
| | u'host': u'controller', |
| | u'hostId': u'b7bbc1111bdc01fbf6ea645b2942c2a201d5211cf997f93543ab57a9', |
| | u'result': u'Success', |
| | u'start_time': u'2019-09-06T06:42:42.000000', |
| | u'traceback': None}] |
| instance_uuid | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
| message | - |
| project_id | ade79e9fbd42422baeefd9e5d4f8a66e |
| request_id | req-9c233e67-0105-4a76-86aa-36fdd59593dd |
| start_time | 2019-09-06T06:42:42.000000 |
| updated_at | 2019-09-06T06:42:42.000000 |
| user_id | 647971cd085b4489af7ccc0126bf8c93 |
+---------------+---------------------------------------------------------------------------+
[root@controller compute]# journalctl -a --unit devstack@n-cpu.service |grep req-9c233e67-0105-4a76-86aa-36fdd59593dd
Logs:
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Received event volume-extended-53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) external_instance_event /opt/stack/nova/nova/compute/manager.py:8799}}
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Handling volume-extended event for volume 53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) extend_volume /opt/stack/nova/nova/compute/manager.py:8644}}
Sep 06 14:42:42 controller nova-compute[13260]: WARNING nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Extend volume failed, volume 1 is not attached to instance.: VolumeBDMNotFound_Remote: No volume Block Device Mapping with id 1.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1843008
Title:
Nova will not detect when extending volume online fails
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
Extending volume in-use may fail, in that case, CLI "nova instance-action-list" should show ERROR in the coulume of message.However , when we catch exceptions in /nova/compute/manager.py
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
nova extends volume will fail ,however CLI "nova instance-action vmid req-id" returns success,that's wrong
===========
1.create volume-backend VM
compute]# nova boot --flavor 1 --nic net-id=27105f4f-5847-4538-862d-6565ad002c9e --boot-volume 53bc2318-ae6c-42b5-ad3b-f197f405cf1e VM
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | vm |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-1jrun74v |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | TbPSQRBic9NQ |
| config_drive | |
| created | 2019-09-06T06:41:55Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.tiny |
| flavor:ram | 512 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| host_status | |
| id | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| locked_reason | - |
| metadata | {} |
| name | VM |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 92cd1d753cf54f3496fda649212cc37b |
| trusted_image_certificates | - |
| updated | 2019-09-06T06:41:55Z |
| user_id | 85c12fb6eada44bc879da10ea394832b |
+--------------------------------------+-------------------------------------------------+
2.Setting specific scenarios to trigger exceptions in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
for example :insert 'extended_volume_id=1' in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8656-L8658 ,
pass objects.BlockDeviceMapping.get_by_volume_and_instance invalid extended_volume_id
3. extend instance's volume
root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 2 | lvmdriver-1 | true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
[root@controller compute]# cinder extend 53bc2318-ae6c-42b5-ad3b-f197f405cf1e 3
[root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 3 | lvmdriver-1 | true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
4.volume extend successfully in cinder , check extending result in nova, it shows success,In fact,extending volume in nova failed
[root@controller compute]# nova instance-action-list fe3c831c-95af-4f0b-b32a-31fb1e425462
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| Action | Request_ID | Message | Start_Time | Updated_At |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| create | req-c3220c13-1de3-435d-a765-c1e931025974 | - | 2019-09-06T06:41:54.000000 | 2019-09-06T06:42:02.000000 |
| extend_volume | req-9c233e67-0105-4a76-86aa-36fdd59593dd | - | 2019-09-06T06:42:42.000000 | 2019-09-06T06:42:42.000000 |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
[root@controller compute]# nova instance-action fe3c831c-95af-4f0b-b32a-31fb1e425462 req-9c233e67-0105-4a76-86aa-36fdd59593dd
+---------------+---------------------------------------------------------------------------+
| Property | Value |
+---------------+---------------------------------------------------------------------------+
| action | extend_volume |
| events | [{u'event': u'compute_extend_volume', |
| | u'finish_time': u'2019-09-06T06:42:42.000000', |
| | u'host': u'controller', |
| | u'hostId': u'b7bbc1111bdc01fbf6ea645b2942c2a201d5211cf997f93543ab57a9', |
| | u'result': u'Success', |
| | u'start_time': u'2019-09-06T06:42:42.000000', |
| | u'traceback': None}] |
| instance_uuid | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
| message | - |
| project_id | ade79e9fbd42422baeefd9e5d4f8a66e |
| request_id | req-9c233e67-0105-4a76-86aa-36fdd59593dd |
| start_time | 2019-09-06T06:42:42.000000 |
| updated_at | 2019-09-06T06:42:42.000000 |
| user_id | 647971cd085b4489af7ccc0126bf8c93 |
+---------------+---------------------------------------------------------------------------+
[root@controller compute]# journalctl -a --unit devstack@n-cpu.service |grep req-9c233e67-0105-4a76-86aa-36fdd59593dd
Logs:
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Received event volume-extended-53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) external_instance_event /opt/stack/nova/nova/compute/manager.py:8799}}
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Handling volume-extended event for volume 53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) extend_volume /opt/stack/nova/nova/compute/manager.py:8644}}
Sep 06 14:42:42 controller nova-compute[13260]: WARNING nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Extend volume failed, volume 1 is not attached to instance.: VolumeBDMNotFound_Remote: No volume Block Device Mapping with id 1.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1843008/+subscriptions
