← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1593672] [NEW] Nova - Novncproxy - Console opens to an incorrect VM

 

Public bug reported:

Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance. 
We tried to shut off all the instances and start them again, but the issue is still exists. 

Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...

But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there 

root@openstackcompute2:~# dpkg -l | grep nova
ii  nova-common                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - common files
ii  nova-compute                         2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node base
ii  nova-compute-kvm                     2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                 2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node libvirt support
ii  python-nova                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute Python libraries
ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7
Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance. 
We tried to shut off all the instances and start them again, but the issue is still exists. 

Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...

But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there 

Steps:
- Open OPenstack dashboard 
- Click the instance name
- Change to console  tab
- Open console
or
- nova get-vnc-console Windows_2k12R2_perf-26

Expected result
===============
We have a console to an another instance

Environment
===========
Control node:
root@openstack1:/etc/nova# dpkg -l | grep nova
ii  nova-api                               2:13.0.0-0ubuntu2                          all          OpenStack Compute - API frontend
ii  nova-cells                             2:13.0.0-0ubuntu2                          all          Openstack Compute - cells
ii  nova-cert                              2:13.0.0-0ubuntu2                          all          OpenStack Compute - certificate management
ii  nova-common                            2:13.0.0-0ubuntu2                          all          OpenStack Compute - common files
ii  nova-conductor                         2:13.0.0-0ubuntu2                          all          OpenStack Compute - conductor service
ii  nova-consoleauth                       2:13.0.0-0ubuntu2                          all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                        2:13.0.0-0ubuntu2                          all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                         2:13.0.0-0ubuntu2                          all          OpenStack Compute - virtual machine scheduler
ii  python-nova                            2:13.0.0-0ubuntu2                          all          OpenStack Compute Python libraries
ii  python-novaclient                      2:3.3.1-2                                  all          client library for OpenStack Compute API - Python 2.7

Compute node1:
root@openstackcompute:~# dpkg -l | grep nova
ii  nova-common                               2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - common files
ii  nova-compute                              2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - compute node base
ii  nova-compute-kvm                          2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                      2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - compute node libvirt support
ii  python-nova                               2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute Python libraries
ii  python-novaclient                         2:3.3.1-2~cloud0                                    all          client library for OpenStack Compute API - Python 2.7

Compute node2:
root@openstackcompute2:~# dpkg -l | grep nova
ii  nova-common                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - common files
ii  nova-compute                         2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node base
ii  nova-compute-kvm                     2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                 2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node libvirt support
ii  python-nova                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute Python libraries
ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7

Compute node3:
root@openstackcompute3:~# dpkg -l | grep nova
ii  nova-common                          2:13.0.0-0ubuntu2                   all          OpenStack Compute - common files
ii  nova-compute                         2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node base
ii  nova-compute-kvm                     2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                 2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
ii  python-nova                          2:13.0.0-0ubuntu2                   all          OpenStack Compute Python libraries
ii  python-novaclient                    2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7

Hypervisors:
KVM + Libvirt on all compute nodes

Storage:
CEPH, iSCSI and FC using Netapp

CEPH version - the same on all storage nodes
ii  ceph                                 10.2.0-0ubuntu0.16.04.1         amd64        distributed storage and file system
ii  ceph-common                          10.2.0-0ubuntu0.16.04.1         amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-deploy                          1.5.32-0ubuntu1                 all          Deployment and configuration of Ceph.
ii  ceph-mds                             10.2.0-0ubuntu0.16.04.1         amd64        metadata server for the ceph distributed file system
ii  libcephfs1                           10.2.0-0ubuntu0.16.04.1         amd64        Ceph distributed file system client library
ii  python-cephfs                        10.2.0-0ubuntu0.16.04.1         amd64        Python libraries for the Ceph libcephfs library

Networking
Neutron networking with Openvswitch

----
Example process flow:

Instance information:
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                                | Value                                                                                                                                                  |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                                                                                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                                                                                                                   |
| OS-EXT-SRV-ATTR:host                 | openstackcompute3                                                                                                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | openstackcompute3                                                                                                                                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-000000f4                                                                                                                                      |
| OS-EXT-STS:power_state               | 1                                                                                                                                                      |
| OS-EXT-STS:task_state                | None                                                                                                                                                   |
| OS-EXT-STS:vm_state                  | active                                                                                                                                                 |
| OS-SRV-USG:launched_at               | 2016-06-16T12:10:06.000000                                                                                                                             |
| OS-SRV-USG:terminated_at             | None                                                                                                                                                   |
| accessIPv4                           |                                                                                                                                                        |
| accessIPv6                           |                                                                                                                                                        |
| addresses                            | net-10.10.128=10.10.128.19                                                                                                                             |
| config_drive                         |                                                                                                                                                        |
| created                              | 2016-06-16T12:09:09Z                                                                                                                                   |
| flavor                               | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108)                                                                                                     |
| hostId                               | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec                                                                                               |
| id                                   | fe3f9980-76c5-4845-baaf-589d129ddfbb                                                                                                                   |
| image                                | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068)                                                                                             |
| key_name                             | CCLTests                                                                                                                                               |
| name                                 | Windows_2k12R2_perf-2                                                                                                                                  |
| os-extended-volumes:volumes_attached | [{u'id': u'd8a9ce45-dac0-42de-a496-db133f1d2a7f'}, {u'id': u'0050819f-3fe7-4ec8-b4d0-2a939ccc533b'}, {u'id': u'343d0841-501d-4a38-a5a5-31f86a7d5053'}] |
| progress                             | 0                                                                                                                                                      |
| project_id                           | 6d0fa2c2b45a4f6c8aac185b4592d44f                                                                                                                       |
| properties                           |                                                                                                                                                        |
| security_groups                      | [{u'name': u'CCL'}]                                                                                                                                    |
| status                               | ACTIVE                                                                                                                                                 |
| updated                              | 2016-06-17T08:35:21Z                                                                                                                                   |
| user_id                              | 75f412ba1601457a8415519bbdd3a932                                                                                                                       |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+

Get the console link:
root@openstack1:/etc/nova# nova get-vnc-console  Windows_2k12R2_perf-2 novnc
+-------+-----------------------------------------------------------------------------------+
| Type  | Url                                                                               |
+-------+-----------------------------------------------------------------------------------+
| novnc | http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3 |
+-------+-----------------------------------------------------------------------------------+

Consoleauth log:
2016-06-17 11:23:37.555 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Token: a33bbd35-1e39-4887-9eed-c86013e598d3 failed to save into memcached.
2016-06-17 11:23:37.557 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Instance: fe3f9980-76c5-4845-baaf-589d129ddfbb failed to save into memcached
2016-06-17 11:23:37.558 20844 INFO nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Received Token: a33bbd35-1e39-4887-9eed-c86013e598d3, {'instance_uuid': u'fe3f9980-76c5-4845-baaf-589d129ddfbb', 'access_url': u'http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3', 'token': u'a33bbd35-1e39-4887-9eed-c86013e598d3', 'last_activity_at': 1466159017.555781, 'internal_access_path': None, 'console_type': u'novnc', 'host': u'192.168.5.56', 'port': u'5914'}

If I check the IP of the host it is 192.168.5.56 which belongs to
openstackcompute2 and the instance is running on openstackcompute3

2016-06-17 11:23:35.834 20775 DEBUG keystoneauth.session [-] RESP: [200] Content-Length: 7168 X-Subject-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 17 Jun 2016 10:23:35 GMT x-openstack-request-id: req-d525aca5-8b85-47e2-9bc6-0fbddfa7fa24 Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"token": {"methods": ["password"], "roles": [{"id": "2588e57d4cb44ad3a95f0f8aaaffcf94", "name": "admin"}], "expires_at": "2016-06-17T11:23:35.295341Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "name": "admin"}, "catalog": "<removed>", "user": {"domain": {"id": "default", "name": "Default"}, "id": "75f412ba1601457a8415519bbdd3a932", "name": "admin"}, "audit_ids": ["uUYFFZ_PTte2PvD09pFusg"], "issued_at": "2016-06-17T10:23:35.000000Z"}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:35.841 20775 INFO nova.osapi_compute.wsgi.server [req-35805c67-a652-4714-bec4-ac39d1ba82e4 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f HTTP/1.1" status: 404 len: 264 time: 0.5082760
2016-06-17 11:23:35.854 20775 DEBUG nova.api.openstack.wsgi [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7f11ccf1c290>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:35.856 20775 INFO nova.osapi_compute.wsgi.server [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/ HTTP/1.1" status: 200 len: 665 time: 0.0106051
2016-06-17 11:23:36.283 20775 DEBUG nova.api.openstack.wsgi [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.index of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.286 20775 DEBUG nova.compute.api [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Searching by: {'deleted': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', u'name': u'Windows_2k12R2_perf-2'} get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:2055
2016-06-17 11:23:36.402 20775 INFO nova.osapi_compute.wsgi.server [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers?name=Windows_2k12R2_perf-2 HTTP/1.1" status: 200 len: 4528 time: 0.1263618
2016-06-17 11:23:36.414 20775 DEBUG nova.api.openstack.wsgi [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.416 20775 DEBUG nova.compute.api [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:36.543 20775 DEBUG nova.policy [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'domain': None, 'project_name': u'admin', 'project_domain': None, 'timestamp': '2016-06-17T10:23:36.409062', 'remote_address': '192.168.5.70', 'quota_class': None, 'resource_uuid': None, 'is_admin': True, 'user': u'75f412ba1601457a8415519bbdd3a932', 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volume', u'name': u'cinder'}], 'tenant': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'read_only': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'user_id': u'75f412ba1601457a8415519bbdd3a932', 'show_deleted': False, 'roles': [u'admin'], 'user_identity': '75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -', 'read_deleted': 'no', 'request_id': 'req-2bf0aad5-ccd0-4727-9a2b-b276656c5632', 'instance_lock_checked': False, 'user_domain': None, 'user_name': u'admin'} enforce /usr/lib/python2.7/dist-packages/nova/policy.py:107
2016-06-17 11:23:36.557 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/ports.json?device_id=fe3f9980-76c5-4845-baaf-589d129ddfbb -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.269 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 887 X-Openstack-Request-Id: req-445709ad-1f43-470e-81c7-d3dbe679cd74
RESP BODY: {"ports": [{"status": "ACTIVE", "binding:host_id": "openstackcompute3", "description": "", "allowed_address_pairs": [], "extra_dhcp_opts": [], "updated_at": "2016-06-17T08:35:43", "device_owner": "compute:nova", "port_security_enabled": true, "binding:profile": {}, "fixed_ips": [{"subnet_id": "ab18e9de-f27b-4c57-a54e-4e4631a79ec9", "ip_address": "10.10.128.19"}], "id": "55a4f4ee-274b-4d2b-82c0-8c3f20050fd0", "security_groups": ["3b859d0c-3641-48f1-bbf6-3c52fb321bf0"], "device_id": "fe3f9980-76c5-4845-baaf-589d129ddfbb", "name": "", "admin_state_up": true, "network_id": "be93253f-ffa3-40e8-a8a9-93976da0cf5c", "dns_name": null, "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": false}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "mac_address": "fa:16:3e:64:17:71", "created_at": "2016-06-16T12:09:53"}]}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.272 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/security-groups.json?id=3b859d0c-3641-48f1-bbf6-3c52fb321bf0 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.342 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 1900 X-Openstack-Request-Id: req-e4a09d2e-fc4a-4d43-8542-a00a56b73a88
RESP BODY: {"security_groups": [{"tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "description": "", "id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "security_group_rules": [{"direction": "ingress", "protocol": "tcp", "description": null, "port_range_max": 22, "id": "1345c82e-34e3-443c-b39d-c38a06365529", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 22, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "49d12fbd-d580-4214-8f81-3cdae3638e3a", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "671b4a2f-c6a9-4450-b450-f56e6eaa356b", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv6"}, {"direction": "ingress", "protocol": "tcp", "description": "", "port_range_max": 5599, "id": "7d3d2a07-2675-446a-8182-8592dde6f64f", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 5500, "ethertype": "IPv4"}, {"direction": "ingress", "protocol": "icmp", "description": null, "port_range_max": null, "id": "bc1b57f4-c75f-4851-99c0-7b9b918897f5", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}], "name": "CCL"}]}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.347 20775 DEBUG nova.objects.instance [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Lazy-loading 'services' on Instance uuid fe3f9980-76c5-4845-baaf-589d129ddfbb obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:895
2016-06-17 11:23:37.430 20775 INFO nova.osapi_compute.wsgi.server [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb HTTP/1.1" status: 200 len: 2908 time: 1.0243011
2016-06-17 11:23:37.440 20775 DEBUG nova.api.openstack.wsgi [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Action: 'create', calling method: <function version_select at 0x7f11cc48b938>, body: {"remote_console": {"type": "novnc", "protocol": "vnc"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:696
2016-06-17 11:23:37.443 20775 DEBUG nova.compute.api [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:37.564 20775 INFO nova.osapi_compute.wsgi.server [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "POST /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb/remote-consoles HTTP/1.1" status: 200 len: 417 time: 0.1293130


If you need any other information I can provide.

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "nova.conf"
   https://bugs.launchpad.net/bugs/1593672/+attachment/4685559/+files/nova.conf

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1593672

Title:
  Nova - Novncproxy - Console opens to an incorrect VM

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance. 
  We tried to shut off all the instances and start them again, but the issue is still exists. 

  Steps to reproduce
  ==================
  At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...

  But as we remember the following happened:
  - Deployed 33 Win2k3 instances with "Launch instance and Count 33"
  - Deployed 33 Win2k8 instances with "Launch instance and count 33"
  - Deployed 33 Win2k12 instances with heat template -> We noticed the problem
  - Stack deleted with all the instances
  - Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there 

  root@openstackcompute2:~# dpkg -l | grep nova
  ii  nova-common                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - common files
  ii  nova-compute                         2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                     2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                 2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node libvirt support
  ii  python-nova                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7
  Description
  ===========
  After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance. 
  We tried to shut off all the instances and start them again, but the issue is still exists. 

  Steps to reproduce
  ==================
  At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...

  But as we remember the following happened:
  - Deployed 33 Win2k3 instances with "Launch instance and Count 33"
  - Deployed 33 Win2k8 instances with "Launch instance and count 33"
  - Deployed 33 Win2k12 instances with heat template -> We noticed the problem
  - Stack deleted with all the instances
  - Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there 

  Steps:
  - Open OPenstack dashboard 
  - Click the instance name
  - Change to console  tab
  - Open console
  or
  - nova get-vnc-console Windows_2k12R2_perf-26

  Expected result
  ===============
  We have a console to an another instance

  Environment
  ===========
  Control node:
  root@openstack1:/etc/nova# dpkg -l | grep nova
  ii  nova-api                               2:13.0.0-0ubuntu2                          all          OpenStack Compute - API frontend
  ii  nova-cells                             2:13.0.0-0ubuntu2                          all          Openstack Compute - cells
  ii  nova-cert                              2:13.0.0-0ubuntu2                          all          OpenStack Compute - certificate management
  ii  nova-common                            2:13.0.0-0ubuntu2                          all          OpenStack Compute - common files
  ii  nova-conductor                         2:13.0.0-0ubuntu2                          all          OpenStack Compute - conductor service
  ii  nova-consoleauth                       2:13.0.0-0ubuntu2                          all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                        2:13.0.0-0ubuntu2                          all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                         2:13.0.0-0ubuntu2                          all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                            2:13.0.0-0ubuntu2                          all          OpenStack Compute Python libraries
  ii  python-novaclient                      2:3.3.1-2                                  all          client library for OpenStack Compute API - Python 2.7

  Compute node1:
  root@openstackcompute:~# dpkg -l | grep nova
  ii  nova-common                               2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - common files
  ii  nova-compute                              2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                          2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                      2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute - compute node libvirt support
  ii  python-nova                               2:13.0.0-0ubuntu2~cloud0                            all          OpenStack Compute Python libraries
  ii  python-novaclient                         2:3.3.1-2~cloud0                                    all          client library for OpenStack Compute API - Python 2.7

  Compute node2:
  root@openstackcompute2:~# dpkg -l | grep nova
  ii  nova-common                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - common files
  ii  nova-compute                         2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                     2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                 2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute - compute node libvirt support
  ii  python-nova                          2:13.0.0-0ubuntu2~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                    2:3.3.1-2~cloud0                      all          client library for OpenStack Compute API - Python 2.7

  Compute node3:
  root@openstackcompute3:~# dpkg -l | grep nova
  ii  nova-common                          2:13.0.0-0ubuntu2                   all          OpenStack Compute - common files
  ii  nova-compute                         2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                     2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                 2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
  ii  python-nova                          2:13.0.0-0ubuntu2                   all          OpenStack Compute Python libraries
  ii  python-novaclient                    2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7

  Hypervisors:
  KVM + Libvirt on all compute nodes

  Storage:
  CEPH, iSCSI and FC using Netapp

  CEPH version - the same on all storage nodes
  ii  ceph                                 10.2.0-0ubuntu0.16.04.1         amd64        distributed storage and file system
  ii  ceph-common                          10.2.0-0ubuntu0.16.04.1         amd64        common utilities to mount and interact with a ceph storage cluster
  ii  ceph-deploy                          1.5.32-0ubuntu1                 all          Deployment and configuration of Ceph.
  ii  ceph-mds                             10.2.0-0ubuntu0.16.04.1         amd64        metadata server for the ceph distributed file system
  ii  libcephfs1                           10.2.0-0ubuntu0.16.04.1         amd64        Ceph distributed file system client library
  ii  python-cephfs                        10.2.0-0ubuntu0.16.04.1         amd64        Python libraries for the Ceph libcephfs library

  Networking
  Neutron networking with Openvswitch

  ----
  Example process flow:

  Instance information:
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field                                | Value                                                                                                                                                  |
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | AUTO                                                                                                                                                   |
  | OS-EXT-AZ:availability_zone          | nova                                                                                                                                                   |
  | OS-EXT-SRV-ATTR:host                 | openstackcompute3                                                                                                                                      |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | openstackcompute3                                                                                                                                      |
  | OS-EXT-SRV-ATTR:instance_name        | instance-000000f4                                                                                                                                      |
  | OS-EXT-STS:power_state               | 1                                                                                                                                                      |
  | OS-EXT-STS:task_state                | None                                                                                                                                                   |
  | OS-EXT-STS:vm_state                  | active                                                                                                                                                 |
  | OS-SRV-USG:launched_at               | 2016-06-16T12:10:06.000000                                                                                                                             |
  | OS-SRV-USG:terminated_at             | None                                                                                                                                                   |
  | accessIPv4                           |                                                                                                                                                        |
  | accessIPv6                           |                                                                                                                                                        |
  | addresses                            | net-10.10.128=10.10.128.19                                                                                                                             |
  | config_drive                         |                                                                                                                                                        |
  | created                              | 2016-06-16T12:09:09Z                                                                                                                                   |
  | flavor                               | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108)                                                                                                     |
  | hostId                               | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec                                                                                               |
  | id                                   | fe3f9980-76c5-4845-baaf-589d129ddfbb                                                                                                                   |
  | image                                | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068)                                                                                             |
  | key_name                             | CCLTests                                                                                                                                               |
  | name                                 | Windows_2k12R2_perf-2                                                                                                                                  |
  | os-extended-volumes:volumes_attached | [{u'id': u'd8a9ce45-dac0-42de-a496-db133f1d2a7f'}, {u'id': u'0050819f-3fe7-4ec8-b4d0-2a939ccc533b'}, {u'id': u'343d0841-501d-4a38-a5a5-31f86a7d5053'}] |
  | progress                             | 0                                                                                                                                                      |
  | project_id                           | 6d0fa2c2b45a4f6c8aac185b4592d44f                                                                                                                       |
  | properties                           |                                                                                                                                                        |
  | security_groups                      | [{u'name': u'CCL'}]                                                                                                                                    |
  | status                               | ACTIVE                                                                                                                                                 |
  | updated                              | 2016-06-17T08:35:21Z                                                                                                                                   |
  | user_id                              | 75f412ba1601457a8415519bbdd3a932                                                                                                                       |
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+

  Get the console link:
  root@openstack1:/etc/nova# nova get-vnc-console  Windows_2k12R2_perf-2 novnc
  +-------+-----------------------------------------------------------------------------------+
  | Type  | Url                                                                               |
  +-------+-----------------------------------------------------------------------------------+
  | novnc | http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3 |
  +-------+-----------------------------------------------------------------------------------+

  Consoleauth log:
  2016-06-17 11:23:37.555 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Token: a33bbd35-1e39-4887-9eed-c86013e598d3 failed to save into memcached.
  2016-06-17 11:23:37.557 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Instance: fe3f9980-76c5-4845-baaf-589d129ddfbb failed to save into memcached
  2016-06-17 11:23:37.558 20844 INFO nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Received Token: a33bbd35-1e39-4887-9eed-c86013e598d3, {'instance_uuid': u'fe3f9980-76c5-4845-baaf-589d129ddfbb', 'access_url': u'http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3', 'token': u'a33bbd35-1e39-4887-9eed-c86013e598d3', 'last_activity_at': 1466159017.555781, 'internal_access_path': None, 'console_type': u'novnc', 'host': u'192.168.5.56', 'port': u'5914'}

  If I check the IP of the host it is 192.168.5.56 which belongs to
  openstackcompute2 and the instance is running on openstackcompute3

  2016-06-17 11:23:35.834 20775 DEBUG keystoneauth.session [-] RESP: [200] Content-Length: 7168 X-Subject-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 17 Jun 2016 10:23:35 GMT x-openstack-request-id: req-d525aca5-8b85-47e2-9bc6-0fbddfa7fa24 Content-Type: application/json X-Distribution: Ubuntu
  RESP BODY: {"token": {"methods": ["password"], "roles": [{"id": "2588e57d4cb44ad3a95f0f8aaaffcf94", "name": "admin"}], "expires_at": "2016-06-17T11:23:35.295341Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "name": "admin"}, "catalog": "<removed>", "user": {"domain": {"id": "default", "name": "Default"}, "id": "75f412ba1601457a8415519bbdd3a932", "name": "admin"}, "audit_ids": ["uUYFFZ_PTte2PvD09pFusg"], "issued_at": "2016-06-17T10:23:35.000000Z"}}
   _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
  2016-06-17 11:23:35.841 20775 INFO nova.osapi_compute.wsgi.server [req-35805c67-a652-4714-bec4-ac39d1ba82e4 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f HTTP/1.1" status: 404 len: 264 time: 0.5082760
  2016-06-17 11:23:35.854 20775 DEBUG nova.api.openstack.wsgi [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7f11ccf1c290>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
  2016-06-17 11:23:35.856 20775 INFO nova.osapi_compute.wsgi.server [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/ HTTP/1.1" status: 200 len: 665 time: 0.0106051
  2016-06-17 11:23:36.283 20775 DEBUG nova.api.openstack.wsgi [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.index of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
  2016-06-17 11:23:36.286 20775 DEBUG nova.compute.api [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Searching by: {'deleted': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', u'name': u'Windows_2k12R2_perf-2'} get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:2055
  2016-06-17 11:23:36.402 20775 INFO nova.osapi_compute.wsgi.server [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers?name=Windows_2k12R2_perf-2 HTTP/1.1" status: 200 len: 4528 time: 0.1263618
  2016-06-17 11:23:36.414 20775 DEBUG nova.api.openstack.wsgi [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
  2016-06-17 11:23:36.416 20775 DEBUG nova.compute.api [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
  2016-06-17 11:23:36.543 20775 DEBUG nova.policy [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'domain': None, 'project_name': u'admin', 'project_domain': None, 'timestamp': '2016-06-17T10:23:36.409062', 'remote_address': '192.168.5.70', 'quota_class': None, 'resource_uuid': None, 'is_admin': True, 'user': u'75f412ba1601457a8415519bbdd3a932', 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volume', u'name': u'cinder'}], 'tenant': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'read_only': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'user_id': u'75f412ba1601457a8415519bbdd3a932', 'show_deleted': False, 'roles': [u'admin'], 'user_identity': '75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -', 'read_deleted': 'no', 'request_id': 'req-2bf0aad5-ccd0-4727-9a2b-b276656c5632', 'instance_lock_checked': False, 'user_domain': None, 'user_name': u'admin'} enforce /usr/lib/python2.7/dist-packages/nova/policy.py:107
  2016-06-17 11:23:36.557 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/ports.json?device_id=fe3f9980-76c5-4845-baaf-589d129ddfbb -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
  2016-06-17 11:23:37.269 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 887 X-Openstack-Request-Id: req-445709ad-1f43-470e-81c7-d3dbe679cd74
  RESP BODY: {"ports": [{"status": "ACTIVE", "binding:host_id": "openstackcompute3", "description": "", "allowed_address_pairs": [], "extra_dhcp_opts": [], "updated_at": "2016-06-17T08:35:43", "device_owner": "compute:nova", "port_security_enabled": true, "binding:profile": {}, "fixed_ips": [{"subnet_id": "ab18e9de-f27b-4c57-a54e-4e4631a79ec9", "ip_address": "10.10.128.19"}], "id": "55a4f4ee-274b-4d2b-82c0-8c3f20050fd0", "security_groups": ["3b859d0c-3641-48f1-bbf6-3c52fb321bf0"], "device_id": "fe3f9980-76c5-4845-baaf-589d129ddfbb", "name": "", "admin_state_up": true, "network_id": "be93253f-ffa3-40e8-a8a9-93976da0cf5c", "dns_name": null, "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": false}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "mac_address": "fa:16:3e:64:17:71", "created_at": "2016-06-16T12:09:53"}]}
   _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
  2016-06-17 11:23:37.272 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/security-groups.json?id=3b859d0c-3641-48f1-bbf6-3c52fb321bf0 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
  2016-06-17 11:23:37.342 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 1900 X-Openstack-Request-Id: req-e4a09d2e-fc4a-4d43-8542-a00a56b73a88
  RESP BODY: {"security_groups": [{"tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "description": "", "id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "security_group_rules": [{"direction": "ingress", "protocol": "tcp", "description": null, "port_range_max": 22, "id": "1345c82e-34e3-443c-b39d-c38a06365529", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 22, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "49d12fbd-d580-4214-8f81-3cdae3638e3a", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "671b4a2f-c6a9-4450-b450-f56e6eaa356b", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv6"}, {"direction": "ingress", "protocol": "tcp", "description": "", "port_range_max": 5599, "id": "7d3d2a07-2675-446a-8182-8592dde6f64f", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 5500, "ethertype": "IPv4"}, {"direction": "ingress", "protocol": "icmp", "description": null, "port_range_max": null, "id": "bc1b57f4-c75f-4851-99c0-7b9b918897f5", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}], "name": "CCL"}]}
   _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
  2016-06-17 11:23:37.347 20775 DEBUG nova.objects.instance [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Lazy-loading 'services' on Instance uuid fe3f9980-76c5-4845-baaf-589d129ddfbb obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:895
  2016-06-17 11:23:37.430 20775 INFO nova.osapi_compute.wsgi.server [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb HTTP/1.1" status: 200 len: 2908 time: 1.0243011
  2016-06-17 11:23:37.440 20775 DEBUG nova.api.openstack.wsgi [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Action: 'create', calling method: <function version_select at 0x7f11cc48b938>, body: {"remote_console": {"type": "novnc", "protocol": "vnc"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:696
  2016-06-17 11:23:37.443 20775 DEBUG nova.compute.api [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
  2016-06-17 11:23:37.564 20775 INFO nova.osapi_compute.wsgi.server [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "POST /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb/remote-consoles HTTP/1.1" status: 200 len: 417 time: 0.1293130

  
  If you need any other information I can provide.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1593672/+subscriptions


Follow ups