yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #64639
[Bug 1690311] Re: Raise particular exception when cannot found instance in hypervisor layer
There is no active patch here
** Changed in: nova
Assignee: Zhenyu Zheng (zhengzhenyu) => (unassigned)
** Changed in: nova
Status: New => Opinion
** Changed in: nova
Importance: Undecided => Wishlist
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1690311
Title:
Raise particular exception when cannot found instance in hypervisor
layer
Status in OpenStack Compute (nova):
Opinion
Bug description:
When request vnc console using /servers/{server_id}/remote_console API,
if the server could not be found in the hypervisor layer,
exception.InstanceNotFound will raise and if the server's vm_state is
"BUILDING" it will be translated to exception.InstanceNotReady and raise,
otherwise exception.InstanceNotFound will raise to the API layer and
response to the users.
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4594
This could be confusing in scenarios as follows:
1. Boot a server with suitable image and flavor;
2. Rebuild the server with a qcow2 type image, and the image size in glance is smaller than the server's flavor but when we use "force_raw_image" in compute, so it will pass API layer image vs flavor check, but it will become larger and failed to spawn in compute, the detailed example that we used will be shown latter;
3. The server is now in ERROR state in Nova DB but gone in hypervisor;
4. List/Show the server, not problems;
5. Get VNC console for the server, exception.InstanceNotFound raised.
As for this kind of scenario, user can list/show the server, but it
raised exception.InstanceNotFound when try to get vnc console for this
server, this might be confusing, as exception.InstanceNotFound is
normally replied to user when the server is not exist in Nova DB;
It might be better if we can use an separate exception for the above
mentioned scenario.
detailed example:
ENV: devstack + nova master:
root@zhenyu-dev:/opt/stack/nova# git log
commit c2d3e80658801ca2ac91899479e525538f233fb1
Author: He Jie Xu <hejie.xu@xxxxxxxxx>
Date: Tue May 2 11:21:47 2017 +0800
Add description to policies in extended_status and
extended_volumes
This updates the policy doc for extended_status.py and
extended_volumes.py
Partial implement blueprint policy-docs
Change-Id: I980eff5fd118601ecdf61c558013f3abba56a6ce
Step 1:
Boot an server, using cirros image and flavor 1 from devstack (1G disk):
root@zhenyu-dev:/opt/stack/nova# nova boot --flavor 1 --image 2808c30b-1848-418f-bfb1-653e65c4e6f3 --nic net-id=0353454b-9fee-4943-a1bc-a3f6ca679667 test
Step2:
Rebuild the instance using a qcow2 image that is:
1) size is smaller than the flavor's disk;
2) when "force_raw_image" is set in compute, and this image transformed to raw, size is larger than flavor's disk;
In our case:
root@zhenyu-dev:/opt/stack/nova# glance image-list
+--------------------------------------+--------------------------+
| ID | Name |
+--------------------------------------+--------------------------+
| 2808c30b-1848-418f-bfb1-653e65c4e6f3 | cirros-0.3.5-x86_64-disk |
| 4ccc6dcc-cf66-438f-ab5d-1e8e115ba685 | mysql |
+--------------------------------------+--------------------------+
root@zhenyu-dev:/opt/stack/nova# glance image-show 4ccc6dcc-cf66-438f-ab5d-1e8e115ba685
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 5c34dc5993beb9068eb9f9b1b25def27 |
| container_format | bare |
| created_at | 2017-05-12T02:09:46Z |
| disk_format | qcow2 |
| id | 4ccc6dcc-cf66-438f-ab5d-1e8e115ba685 |
| min_disk | 0 |
| min_ram | 0 |
| name | mysql |
| owner | 3ed63cea5fe5433eaed39fae595d5a9d |
| protected | False |
| size | 517668864 |
| status | active |
| tags | [] |
| updated_at | 2017-05-12T02:09:48Z |
| virtual_size | None |
| visibility | public |
+------------------+--------------------------------------+
this is a ubuntu 16.04 image and its size is about 500MB which suits out flavor, but it will grow to 2.5G when transformed to RAW;
root@zhenyu-dev:/opt/stack/nova# nova list
/usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
warnings.warn(msg)
+--------------------------------------+------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+---------------------------------------------------------+
| 6eecb8aa-c431-446b-a450-e9663a9b29ec | test | ACTIVE | - | Running | private=10.0.0.10, fd32:6d3f:e1e6:0:f816:3eff:fefb:5419 |
DO THE REBUILD:
root@zhenyu-dev:/opt/stack/nova# nova rebuild 6eecb8aa-c431-446b-a450-e9663a9b29ec 4ccc6dcc-cf66-438f-ab5d-1e8e115ba685
root@zhenyu-dev:/opt/stack/nova# nova list
/usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
warnings.warn(msg)
+--------------------------------------+------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+---------------------------------------------------------+
| 6eecb8aa-c431-446b-a450-e9663a9b29ec | test | ERROR | - | Running | private=10.0.0.10, fd32:6d3f:e1e6:0:f816:3eff:fefb:5419 |
+--------------------------------------+------+--------+------------+-------------+---------------------------------------------------------+
instance in ERROR state due to:
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2668, in _rebuild_default_impl
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server block_device_info=new_block_device_info)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server block_device_info=block_device_info)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server fallback_from_host)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server instance, size, fallback_from_host)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server size=size)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server *args, **kwargs)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 574, in create_image
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server self.verify_base_size(base, size)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 283, in verify_base_size
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server flavor_size=size, image_size=base_size)
2017-05-12 15:34:05.995 TRACE oslo_messaging.rpc.server FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 1073741824 bytes, image is 2523463680 bytes.
And the domain in hypervisor is gone;
Last Step:
Try to get vnc console for the server:
root@zhenyu-dev:/opt/stack/nova# nova get-vnc-console 6eecb8aa-c431-446b-a450-e9663a9b29ec novnc
/usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
warnings.warn(msg)
ERROR (NotFound): Instance 6eecb8aa-c431-446b-a450-e9663a9b29ec could not be found. (HTTP 404) (Request-ID: req-46eb7721-1d18-48af-96ac-e8a521a99791)
A little bit confusing as I can list/show the server, but when I
calling for vnc-console, it shows InstanceNotFound.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1690311/+subscriptions
References
