yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #52485
[Bug 1593672] [NEW] Nova - Novncproxy - Console opens to an incorrect VM
Public bug reported:
Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance.
We tried to shut off all the instances and start them again, but the issue is still exists.
Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...
But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance.
We tried to shut off all the instances and start them again, but the issue is still exists.
Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...
But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there
Steps:
- Open OPenstack dashboard
- Click the instance name
- Change to console tab
- Open console
or
- nova get-vnc-console Windows_2k12R2_perf-26
Expected result
===============
We have a console to an another instance
Environment
===========
Control node:
root@openstack1:/etc/nova# dpkg -l | grep nova
ii nova-api 2:13.0.0-0ubuntu2 all OpenStack Compute - API frontend
ii nova-cells 2:13.0.0-0ubuntu2 all Openstack Compute - cells
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Compute node1:
root@openstackcompute:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node2:
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node3:
root@openstackcompute3:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Hypervisors:
KVM + Libvirt on all compute nodes
Storage:
CEPH, iSCSI and FC using Netapp
CEPH version - the same on all storage nodes
ii ceph 10.2.0-0ubuntu0.16.04.1 amd64 distributed storage and file system
ii ceph-common 10.2.0-0ubuntu0.16.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-deploy 1.5.32-0ubuntu1 all Deployment and configuration of Ceph.
ii ceph-mds 10.2.0-0ubuntu0.16.04.1 amd64 metadata server for the ceph distributed file system
ii libcephfs1 10.2.0-0ubuntu0.16.04.1 amd64 Ceph distributed file system client library
ii python-cephfs 10.2.0-0ubuntu0.16.04.1 amd64 Python libraries for the Ceph libcephfs library
Networking
Neutron networking with Openvswitch
----
Example process flow:
Instance information:
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstackcompute3 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstackcompute3 |
| OS-EXT-SRV-ATTR:instance_name | instance-000000f4 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-06-16T12:10:06.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | net-10.10.128=10.10.128.19 |
| config_drive | |
| created | 2016-06-16T12:09:09Z |
| flavor | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108) |
| hostId | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec |
| id | fe3f9980-76c5-4845-baaf-589d129ddfbb |
| image | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068) |
| key_name | CCLTests |
| name | Windows_2k12R2_perf-2 |
| os-extended-volumes:volumes_attached | [{u'id': u'd8a9ce45-dac0-42de-a496-db133f1d2a7f'}, {u'id': u'0050819f-3fe7-4ec8-b4d0-2a939ccc533b'}, {u'id': u'343d0841-501d-4a38-a5a5-31f86a7d5053'}] |
| progress | 0 |
| project_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| properties | |
| security_groups | [{u'name': u'CCL'}] |
| status | ACTIVE |
| updated | 2016-06-17T08:35:21Z |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Get the console link:
root@openstack1:/etc/nova# nova get-vnc-console Windows_2k12R2_perf-2 novnc
+-------+-----------------------------------------------------------------------------------+
| Type | Url |
+-------+-----------------------------------------------------------------------------------+
| novnc | http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3 |
+-------+-----------------------------------------------------------------------------------+
Consoleauth log:
2016-06-17 11:23:37.555 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Token: a33bbd35-1e39-4887-9eed-c86013e598d3 failed to save into memcached.
2016-06-17 11:23:37.557 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Instance: fe3f9980-76c5-4845-baaf-589d129ddfbb failed to save into memcached
2016-06-17 11:23:37.558 20844 INFO nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Received Token: a33bbd35-1e39-4887-9eed-c86013e598d3, {'instance_uuid': u'fe3f9980-76c5-4845-baaf-589d129ddfbb', 'access_url': u'http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3', 'token': u'a33bbd35-1e39-4887-9eed-c86013e598d3', 'last_activity_at': 1466159017.555781, 'internal_access_path': None, 'console_type': u'novnc', 'host': u'192.168.5.56', 'port': u'5914'}
If I check the IP of the host it is 192.168.5.56 which belongs to
openstackcompute2 and the instance is running on openstackcompute3
2016-06-17 11:23:35.834 20775 DEBUG keystoneauth.session [-] RESP: [200] Content-Length: 7168 X-Subject-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 17 Jun 2016 10:23:35 GMT x-openstack-request-id: req-d525aca5-8b85-47e2-9bc6-0fbddfa7fa24 Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"token": {"methods": ["password"], "roles": [{"id": "2588e57d4cb44ad3a95f0f8aaaffcf94", "name": "admin"}], "expires_at": "2016-06-17T11:23:35.295341Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "name": "admin"}, "catalog": "<removed>", "user": {"domain": {"id": "default", "name": "Default"}, "id": "75f412ba1601457a8415519bbdd3a932", "name": "admin"}, "audit_ids": ["uUYFFZ_PTte2PvD09pFusg"], "issued_at": "2016-06-17T10:23:35.000000Z"}}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:35.841 20775 INFO nova.osapi_compute.wsgi.server [req-35805c67-a652-4714-bec4-ac39d1ba82e4 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f HTTP/1.1" status: 404 len: 264 time: 0.5082760
2016-06-17 11:23:35.854 20775 DEBUG nova.api.openstack.wsgi [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7f11ccf1c290>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:35.856 20775 INFO nova.osapi_compute.wsgi.server [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/ HTTP/1.1" status: 200 len: 665 time: 0.0106051
2016-06-17 11:23:36.283 20775 DEBUG nova.api.openstack.wsgi [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.index of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.286 20775 DEBUG nova.compute.api [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Searching by: {'deleted': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', u'name': u'Windows_2k12R2_perf-2'} get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:2055
2016-06-17 11:23:36.402 20775 INFO nova.osapi_compute.wsgi.server [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers?name=Windows_2k12R2_perf-2 HTTP/1.1" status: 200 len: 4528 time: 0.1263618
2016-06-17 11:23:36.414 20775 DEBUG nova.api.openstack.wsgi [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.416 20775 DEBUG nova.compute.api [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:36.543 20775 DEBUG nova.policy [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'domain': None, 'project_name': u'admin', 'project_domain': None, 'timestamp': '2016-06-17T10:23:36.409062', 'remote_address': '192.168.5.70', 'quota_class': None, 'resource_uuid': None, 'is_admin': True, 'user': u'75f412ba1601457a8415519bbdd3a932', 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volume', u'name': u'cinder'}], 'tenant': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'read_only': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'user_id': u'75f412ba1601457a8415519bbdd3a932', 'show_deleted': False, 'roles': [u'admin'], 'user_identity': '75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -', 'read_deleted': 'no', 'request_id': 'req-2bf0aad5-ccd0-4727-9a2b-b276656c5632', 'instance_lock_checked': False, 'user_domain': None, 'user_name': u'admin'} enforce /usr/lib/python2.7/dist-packages/nova/policy.py:107
2016-06-17 11:23:36.557 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/ports.json?device_id=fe3f9980-76c5-4845-baaf-589d129ddfbb -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.269 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 887 X-Openstack-Request-Id: req-445709ad-1f43-470e-81c7-d3dbe679cd74
RESP BODY: {"ports": [{"status": "ACTIVE", "binding:host_id": "openstackcompute3", "description": "", "allowed_address_pairs": [], "extra_dhcp_opts": [], "updated_at": "2016-06-17T08:35:43", "device_owner": "compute:nova", "port_security_enabled": true, "binding:profile": {}, "fixed_ips": [{"subnet_id": "ab18e9de-f27b-4c57-a54e-4e4631a79ec9", "ip_address": "10.10.128.19"}], "id": "55a4f4ee-274b-4d2b-82c0-8c3f20050fd0", "security_groups": ["3b859d0c-3641-48f1-bbf6-3c52fb321bf0"], "device_id": "fe3f9980-76c5-4845-baaf-589d129ddfbb", "name": "", "admin_state_up": true, "network_id": "be93253f-ffa3-40e8-a8a9-93976da0cf5c", "dns_name": null, "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": false}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "mac_address": "fa:16:3e:64:17:71", "created_at": "2016-06-16T12:09:53"}]}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.272 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/security-groups.json?id=3b859d0c-3641-48f1-bbf6-3c52fb321bf0 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.342 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 1900 X-Openstack-Request-Id: req-e4a09d2e-fc4a-4d43-8542-a00a56b73a88
RESP BODY: {"security_groups": [{"tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "description": "", "id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "security_group_rules": [{"direction": "ingress", "protocol": "tcp", "description": null, "port_range_max": 22, "id": "1345c82e-34e3-443c-b39d-c38a06365529", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 22, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "49d12fbd-d580-4214-8f81-3cdae3638e3a", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "671b4a2f-c6a9-4450-b450-f56e6eaa356b", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv6"}, {"direction": "ingress", "protocol": "tcp", "description": "", "port_range_max": 5599, "id": "7d3d2a07-2675-446a-8182-8592dde6f64f", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 5500, "ethertype": "IPv4"}, {"direction": "ingress", "protocol": "icmp", "description": null, "port_range_max": null, "id": "bc1b57f4-c75f-4851-99c0-7b9b918897f5", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}], "name": "CCL"}]}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.347 20775 DEBUG nova.objects.instance [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Lazy-loading 'services' on Instance uuid fe3f9980-76c5-4845-baaf-589d129ddfbb obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:895
2016-06-17 11:23:37.430 20775 INFO nova.osapi_compute.wsgi.server [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb HTTP/1.1" status: 200 len: 2908 time: 1.0243011
2016-06-17 11:23:37.440 20775 DEBUG nova.api.openstack.wsgi [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Action: 'create', calling method: <function version_select at 0x7f11cc48b938>, body: {"remote_console": {"type": "novnc", "protocol": "vnc"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:696
2016-06-17 11:23:37.443 20775 DEBUG nova.compute.api [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:37.564 20775 INFO nova.osapi_compute.wsgi.server [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "POST /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb/remote-consoles HTTP/1.1" status: 200 len: 417 time: 0.1293130
If you need any other information I can provide.
** Affects: nova
Importance: Undecided
Status: New
** Attachment added: "nova.conf"
https://bugs.launchpad.net/bugs/1593672/+attachment/4685559/+files/nova.conf
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1593672
Title:
Nova - Novncproxy - Console opens to an incorrect VM
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance.
We tried to shut off all the instances and start them again, but the issue is still exists.
Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...
But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance.
We tried to shut off all the instances and start them again, but the issue is still exists.
Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...
But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there
Steps:
- Open OPenstack dashboard
- Click the instance name
- Change to console tab
- Open console
or
- nova get-vnc-console Windows_2k12R2_perf-26
Expected result
===============
We have a console to an another instance
Environment
===========
Control node:
root@openstack1:/etc/nova# dpkg -l | grep nova
ii nova-api 2:13.0.0-0ubuntu2 all OpenStack Compute - API frontend
ii nova-cells 2:13.0.0-0ubuntu2 all Openstack Compute - cells
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Compute node1:
root@openstackcompute:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node2:
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node3:
root@openstackcompute3:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Hypervisors:
KVM + Libvirt on all compute nodes
Storage:
CEPH, iSCSI and FC using Netapp
CEPH version - the same on all storage nodes
ii ceph 10.2.0-0ubuntu0.16.04.1 amd64 distributed storage and file system
ii ceph-common 10.2.0-0ubuntu0.16.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-deploy 1.5.32-0ubuntu1 all Deployment and configuration of Ceph.
ii ceph-mds 10.2.0-0ubuntu0.16.04.1 amd64 metadata server for the ceph distributed file system
ii libcephfs1 10.2.0-0ubuntu0.16.04.1 amd64 Ceph distributed file system client library
ii python-cephfs 10.2.0-0ubuntu0.16.04.1 amd64 Python libraries for the Ceph libcephfs library
Networking
Neutron networking with Openvswitch
----
Example process flow:
Instance information:
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstackcompute3 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstackcompute3 |
| OS-EXT-SRV-ATTR:instance_name | instance-000000f4 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-06-16T12:10:06.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | net-10.10.128=10.10.128.19 |
| config_drive | |
| created | 2016-06-16T12:09:09Z |
| flavor | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108) |
| hostId | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec |
| id | fe3f9980-76c5-4845-baaf-589d129ddfbb |
| image | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068) |
| key_name | CCLTests |
| name | Windows_2k12R2_perf-2 |
| os-extended-volumes:volumes_attached | [{u'id': u'd8a9ce45-dac0-42de-a496-db133f1d2a7f'}, {u'id': u'0050819f-3fe7-4ec8-b4d0-2a939ccc533b'}, {u'id': u'343d0841-501d-4a38-a5a5-31f86a7d5053'}] |
| progress | 0 |
| project_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| properties | |
| security_groups | [{u'name': u'CCL'}] |
| status | ACTIVE |
| updated | 2016-06-17T08:35:21Z |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Get the console link:
root@openstack1:/etc/nova# nova get-vnc-console Windows_2k12R2_perf-2 novnc
+-------+-----------------------------------------------------------------------------------+
| Type | Url |
+-------+-----------------------------------------------------------------------------------+
| novnc | http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3 |
+-------+-----------------------------------------------------------------------------------+
Consoleauth log:
2016-06-17 11:23:37.555 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Token: a33bbd35-1e39-4887-9eed-c86013e598d3 failed to save into memcached.
2016-06-17 11:23:37.557 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Instance: fe3f9980-76c5-4845-baaf-589d129ddfbb failed to save into memcached
2016-06-17 11:23:37.558 20844 INFO nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Received Token: a33bbd35-1e39-4887-9eed-c86013e598d3, {'instance_uuid': u'fe3f9980-76c5-4845-baaf-589d129ddfbb', 'access_url': u'http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3', 'token': u'a33bbd35-1e39-4887-9eed-c86013e598d3', 'last_activity_at': 1466159017.555781, 'internal_access_path': None, 'console_type': u'novnc', 'host': u'192.168.5.56', 'port': u'5914'}
If I check the IP of the host it is 192.168.5.56 which belongs to
openstackcompute2 and the instance is running on openstackcompute3
2016-06-17 11:23:35.834 20775 DEBUG keystoneauth.session [-] RESP: [200] Content-Length: 7168 X-Subject-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 17 Jun 2016 10:23:35 GMT x-openstack-request-id: req-d525aca5-8b85-47e2-9bc6-0fbddfa7fa24 Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"token": {"methods": ["password"], "roles": [{"id": "2588e57d4cb44ad3a95f0f8aaaffcf94", "name": "admin"}], "expires_at": "2016-06-17T11:23:35.295341Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "name": "admin"}, "catalog": "<removed>", "user": {"domain": {"id": "default", "name": "Default"}, "id": "75f412ba1601457a8415519bbdd3a932", "name": "admin"}, "audit_ids": ["uUYFFZ_PTte2PvD09pFusg"], "issued_at": "2016-06-17T10:23:35.000000Z"}}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:35.841 20775 INFO nova.osapi_compute.wsgi.server [req-35805c67-a652-4714-bec4-ac39d1ba82e4 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f HTTP/1.1" status: 404 len: 264 time: 0.5082760
2016-06-17 11:23:35.854 20775 DEBUG nova.api.openstack.wsgi [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7f11ccf1c290>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:35.856 20775 INFO nova.osapi_compute.wsgi.server [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/ HTTP/1.1" status: 200 len: 665 time: 0.0106051
2016-06-17 11:23:36.283 20775 DEBUG nova.api.openstack.wsgi [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.index of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.286 20775 DEBUG nova.compute.api [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Searching by: {'deleted': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', u'name': u'Windows_2k12R2_perf-2'} get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:2055
2016-06-17 11:23:36.402 20775 INFO nova.osapi_compute.wsgi.server [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers?name=Windows_2k12R2_perf-2 HTTP/1.1" status: 200 len: 4528 time: 0.1263618
2016-06-17 11:23:36.414 20775 DEBUG nova.api.openstack.wsgi [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.416 20775 DEBUG nova.compute.api [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:36.543 20775 DEBUG nova.policy [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'domain': None, 'project_name': u'admin', 'project_domain': None, 'timestamp': '2016-06-17T10:23:36.409062', 'remote_address': '192.168.5.70', 'quota_class': None, 'resource_uuid': None, 'is_admin': True, 'user': u'75f412ba1601457a8415519bbdd3a932', 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volume', u'name': u'cinder'}], 'tenant': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'read_only': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'user_id': u'75f412ba1601457a8415519bbdd3a932', 'show_deleted': False, 'roles': [u'admin'], 'user_identity': '75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -', 'read_deleted': 'no', 'request_id': 'req-2bf0aad5-ccd0-4727-9a2b-b276656c5632', 'instance_lock_checked': False, 'user_domain': None, 'user_name': u'admin'} enforce /usr/lib/python2.7/dist-packages/nova/policy.py:107
2016-06-17 11:23:36.557 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/ports.json?device_id=fe3f9980-76c5-4845-baaf-589d129ddfbb -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.269 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 887 X-Openstack-Request-Id: req-445709ad-1f43-470e-81c7-d3dbe679cd74
RESP BODY: {"ports": [{"status": "ACTIVE", "binding:host_id": "openstackcompute3", "description": "", "allowed_address_pairs": [], "extra_dhcp_opts": [], "updated_at": "2016-06-17T08:35:43", "device_owner": "compute:nova", "port_security_enabled": true, "binding:profile": {}, "fixed_ips": [{"subnet_id": "ab18e9de-f27b-4c57-a54e-4e4631a79ec9", "ip_address": "10.10.128.19"}], "id": "55a4f4ee-274b-4d2b-82c0-8c3f20050fd0", "security_groups": ["3b859d0c-3641-48f1-bbf6-3c52fb321bf0"], "device_id": "fe3f9980-76c5-4845-baaf-589d129ddfbb", "name": "", "admin_state_up": true, "network_id": "be93253f-ffa3-40e8-a8a9-93976da0cf5c", "dns_name": null, "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": false}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "mac_address": "fa:16:3e:64:17:71", "created_at": "2016-06-16T12:09:53"}]}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.272 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/security-groups.json?id=3b859d0c-3641-48f1-bbf6-3c52fb321bf0 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.342 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 1900 X-Openstack-Request-Id: req-e4a09d2e-fc4a-4d43-8542-a00a56b73a88
RESP BODY: {"security_groups": [{"tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "description": "", "id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "security_group_rules": [{"direction": "ingress", "protocol": "tcp", "description": null, "port_range_max": 22, "id": "1345c82e-34e3-443c-b39d-c38a06365529", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 22, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "49d12fbd-d580-4214-8f81-3cdae3638e3a", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "671b4a2f-c6a9-4450-b450-f56e6eaa356b", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv6"}, {"direction": "ingress", "protocol": "tcp", "description": "", "port_range_max": 5599, "id": "7d3d2a07-2675-446a-8182-8592dde6f64f", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 5500, "ethertype": "IPv4"}, {"direction": "ingress", "protocol": "icmp", "description": null, "port_range_max": null, "id": "bc1b57f4-c75f-4851-99c0-7b9b918897f5", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}], "name": "CCL"}]}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.347 20775 DEBUG nova.objects.instance [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Lazy-loading 'services' on Instance uuid fe3f9980-76c5-4845-baaf-589d129ddfbb obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:895
2016-06-17 11:23:37.430 20775 INFO nova.osapi_compute.wsgi.server [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb HTTP/1.1" status: 200 len: 2908 time: 1.0243011
2016-06-17 11:23:37.440 20775 DEBUG nova.api.openstack.wsgi [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Action: 'create', calling method: <function version_select at 0x7f11cc48b938>, body: {"remote_console": {"type": "novnc", "protocol": "vnc"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:696
2016-06-17 11:23:37.443 20775 DEBUG nova.compute.api [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:37.564 20775 INFO nova.osapi_compute.wsgi.server [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "POST /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb/remote-consoles HTTP/1.1" status: 200 len: 417 time: 0.1293130
If you need any other information I can provide.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1593672/+subscriptions
Follow ups
