← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1558384] [NEW] Unable to establish connection to http://controller:35357/v3/auth/tokens

 

Public bug reported:

I install Liberty on CentOS 7 according to
http://docs.openstack.org/liberty/install-guide-rdo/index.html, with one
controller node and one compute node.

1, Compute node can not connect to keystone on controller:

"[root@compute3 ~]# nova service-list
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
ERROR (ConnectionRefused): Unable to establish connection to http://controller:35357/v3/auth/tokens";


2, On controller, "openstack-status" shows "<class 'keystoneclient.exceptions.ConnectionRefused'> (HTTP 500) (Request-ID: req-2b2cd91c-3fbd-4baa-87f3-bec2c57039e7)":

"[root@controller ~]# openstack-status
== Nova services ==
openstack-nova-api:                     active
openstack-nova-compute:                 inactive  (disabled on boot)
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-cert:                    active
openstack-nova-conductor:               active
openstack-nova-console:                 inactive  (disabled on boot)
openstack-nova-consoleauth:             active
openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active
== Keystone service ==
openstack-keystone:                     inactive  (disabled on boot)
== Support services ==
mysqld:                                 inactive  (disabled on boot)
dbus:                                   active
rabbitmq-server:                        active
memcached:                              active
== Keystone users ==
/usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
WARNING: unsupported identity-api-version 3, falling back to 2.0
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/site-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:376: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-7b248df7-dbc6-4bc8-be72-862c2e622994)
== Glance images ==
+--------------------------------------+--------+
| ID                                   | Name   |
+--------------------------------------+--------+
| b90fc7b1-5f7e-4a72-87ce-0b7f21ca2afc | cirros |
+--------------------------------------+--------+
== Nova managed services ==
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host       | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-conductor   | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
| 2  | nova-consoleauth | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
| 3  | nova-scheduler   | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
| 4  | nova-cert        | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
== Nova networks ==
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneclient.exceptions.ConnectionRefused'> (HTTP 500) (Request-ID: req-2b2cd91c-3fbd-4baa-87f3-bec2c57039e7)
== Nova instance flavors ==
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1  | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
== Nova instances ==
+----+------+-----------+--------+------------+-------------+----------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+----+------+-----------+--------+------------+-------------+----------+
+----+------+-----------+--------+------------+-------------+----------+"


3, Seems port 5000 and 35357 is not opened by httpd:

"[root@controller ~]# systemctl status httpd.service 
鈼[0m httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-03-17 08:49:46 EDT; 1h 58min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 1366 (httpd)
   Status: "Total requests: 60; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           鈹溾攢1366 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢2696 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2697 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2698 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2699 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2700 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2701 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2702 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2703 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2704 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2705 (wsgi:keystone- -DFOREGROUND
           鈹溾攢2706 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢2707 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢2708 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢2709 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢2710 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢3307 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢3370 /usr/sbin/httpd -DFOREGROUND
           鈹溾攢3371 /usr/sbin/httpd -DFOREGROUND
           鈹斺攢3372 /usr/sbin/httpd -DFOREGROUND

Mar 17 08:49:44 controller systemd[1]: Starting The Apache HTTP Server...
Mar 17 08:49:46 controller systemd[1]: Started The Apache HTTP Server."


4, /var/log/keystone/keystone.log attached:

"2016-03-17 09:22:28.404 2702 INFO keystone.common.wsgi [req-e127518c-de2b-4484-8359-d1324f05dd51 - - - - -] GET http://controller:35357/v3/
2016-03-17 09:22:28.408 2701 INFO keystone.common.wsgi [req-ce27bdf1-254b-4dee-983e-0ae159231daf - - - - -] POST http://controller:35357/v3/auth/tokens
2016-03-17 09:22:28.483 2703 INFO keystone.common.wsgi [req-f56d0bb4-9468-458f-a591-52d9f74be7ba - - - - -] GET http://controller:35357/v3/auth/tokens
2016-03-17 09:22:28.734 2705 INFO keystone.common.wsgi [req-58077650-ee50-426d-a7b6-a666baf63602 - - - - -] GET http://controller:35357/v3/
2016-03-17 09:22:28.737 2702 INFO keystone.common.wsgi [req-f67b247f-3854-432e-aba7-8cd895501723 - - - - -] POST http://controller:35357/v3/auth/tokens
2016-03-17 09:22:28.780 2704 INFO keystone.common.wsgi [req-1d5125c4-3fe8-4e80-bbd6-8a191f65e285 - - - - -] GET http://controller:35357/v3/auth/tokens
2016-03-17 09:22:29.109 2705 INFO keystone.common.wsgi [req-2e5a3426-a7d9-4e01-87d1-5bbc9861c2f6 - - - - -] GET http://controller:35357/v3/
2016-03-17 09:22:29.113 2703 INFO keystone.common.wsgi [req-2a8f7eaf-243b-4da0-9e11-043e9529b682 - - - - -] POST http://controller:35357/v3/auth/tokens
2016-03-17 09:22:29.154 2701 INFO keystone.common.wsgi [req-56241137-d85c-4522-9a5c-1da4db689760 - - - - -] GET http://controller:35357/v3/auth/tokens
2016-03-17 09:22:29.471 2705 INFO keystone.common.wsgi [req-a19d48e5-8f43-4a83-bb5e-35960976383b - - - - -] GET http://controller:35357/v3/
2016-03-17 09:22:29.475 2703 INFO keystone.common.wsgi [req-262f60aa-cd8a-4056-b8bf-ed19452c7210 - - - - -] POST http://controller:35357/v3/auth/tokens
2016-03-17 09:22:29.516 2704 INFO keystone.common.wsgi [req-8b369ab3-f791-45b5-bfe5-142dde22bdb2 - - - - -] GET http://controller:35357/v3/auth/tokens
2016-03-17 09:22:29.841 2702 INFO keystone.common.wsgi [req-f93c2103-cb88-4e46-b21a-a71291c9ff13 - - - - -] GET http://controller:35357/v3/
2016-03-17 09:22:29.844 2703 INFO keystone.common.wsgi [req-7766d108-1671-4080-9d12-ebbcb01ec1f5 - - - - -] POST http://controller:35357/v3/auth/tokens
2016-03-17 09:22:29.885 2705 INFO keystone.common.wsgi [req-22c3d476-d439-4ef7-9b13-3f2989f02c7f - - - - -] GET http://controller:35357/v3/auth/tokens
2016-03-17 09:24:56.072 2701 INFO keystone.common.wsgi [req-9251220e-62ad-475a-ad75-8efbd9840abb - - - - -] GET http://controller:35357/v3/
2016-03-17 09:24:56.075 2703 INFO keystone.common.wsgi [req-2a59386a-09d3-46de-b1fb-7044082058e3 - - - - -] POST http://controller:35357/v3/auth/tokens
2016-03-17 09:24:56.116 2705 INFO keystone.common.wsgi [req-77bb05dd-c74d-4862-afee-d8afdb584082 - - - - -] GET http://controller:35357/v3/auth/tokens";


5, /var/log/httpd/error_log attached:

"[Thu Mar 17 07:52:48.247443 2016] [mpm_prefork:notice] [pid 1358] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Mar 17 07:55:01.758180 2016] [core:notice] [pid 1360] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Mar 17 07:55:02.045384 2016] [suexec:notice] [pid 1360] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 17 07:55:02.053827 2016] [auth_digest:notice] [pid 1360] AH01757: generating secret for digest authentication ...
[Thu Mar 17 07:55:02.054499 2016] [lbmethod_heartbeat:notice] [pid 1360] AH02282: No slotmem from mod_heartmonitor
[Thu Mar 17 07:55:02.058430 2016] [mpm_prefork:notice] [pid 1360] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Mar 17 07:55:02.058446 2016] [core:notice] [pid 1360] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Mar 17 08:27:39.730480 2016] [mpm_prefork:notice] [pid 1360] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Mar 17 08:29:55.642727 2016] [core:notice] [pid 1347] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Mar 17 08:29:55.708727 2016] [suexec:notice] [pid 1347] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 17 08:29:55.717232 2016] [auth_digest:notice] [pid 1347] AH01757: generating secret for digest authentication ...
[Thu Mar 17 08:29:55.717886 2016] [lbmethod_heartbeat:notice] [pid 1347] AH02282: No slotmem from mod_heartmonitor
[Thu Mar 17 08:29:55.721848 2016] [mpm_prefork:notice] [pid 1347] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Mar 17 08:29:55.721865 2016] [core:notice] [pid 1347] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Mar 17 08:47:32.275843 2016] [mpm_prefork:notice] [pid 1347] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Mar 17 08:49:46.323504 2016] [core:notice] [pid 1366] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Mar 17 08:49:46.344285 2016] [suexec:notice] [pid 1366] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 17 08:49:46.353395 2016] [auth_digest:notice] [pid 1366] AH01757: generating secret for digest authentication ...
[Thu Mar 17 08:49:46.354157 2016] [lbmethod_heartbeat:notice] [pid 1366] AH02282: No slotmem from mod_heartmonitor
[Thu Mar 17 08:49:46.358306 2016] [mpm_prefork:notice] [pid 1366] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Mar 17 08:49:46.358331 2016] [core:notice] [pid 1366] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'" 


6, /var/log/httpd/keystone-access.log attached:

"::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
135.251.157.109 - - [17/Mar/2016:08:50:11 -0400] "POST /v3/tokens HTTP/1.1" 404 93 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:13 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:13 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:14 -0400] "GET / HTTP/1.1" 300 591 "-" "glance/11.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:14 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2260 "-" "glance/11.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:15 -0400] "GET / HTTP/1.1" 300 591 "-" "glance/11.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:15 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2258 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2258 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:19 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:19 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:19 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:36 -0400] "POST /v3/tokens HTTP/1.1" 404 93 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "POST /v3/tokens HTTP/1.1" 404 93 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:24:56 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:24:56 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
135.251.157.109 - - [17/Mar/2016:09:24:56 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient""


7, /var/log/httpd/keystone-error.log attached:

"2016-03-17 08:50:12.323143 No handlers could be found for logger "oslo_config.cfg"
2016-03-17 08:50:13.490936 No handlers could be found for logger "oslo_config.cfg"
2016-03-17 08:50:13.955267 No handlers could be found for logger "oslo_config.cfg"
2016-03-17 08:50:14.726416 No handlers could be found for logger "oslo_config.cfg"
2016-03-17 08:50:15.159876 No handlers could be found for logger "oslo_config.cfg""

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1558384

Title:
  Unable to establish connection to
  http://controller:35357/v3/auth/tokens

Status in OpenStack Compute (nova):
  New

Bug description:
  I install Liberty on CentOS 7 according to
  http://docs.openstack.org/liberty/install-guide-rdo/index.html, with
  one controller node and one compute node.

  1, Compute node can not connect to keystone on controller:

  "[root@compute3 ~]# nova service-list
  No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
  ERROR (ConnectionRefused): Unable to establish connection to http://controller:35357/v3/auth/tokens";

  
  2, On controller, "openstack-status" shows "<class 'keystoneclient.exceptions.ConnectionRefused'> (HTTP 500) (Request-ID: req-2b2cd91c-3fbd-4baa-87f3-bec2c57039e7)":

  "[root@controller ~]# openstack-status
  == Nova services ==
  openstack-nova-api:                     active
  openstack-nova-compute:                 inactive  (disabled on boot)
  openstack-nova-network:                 inactive  (disabled on boot)
  openstack-nova-scheduler:               active
  openstack-nova-cert:                    active
  openstack-nova-conductor:               active
  openstack-nova-console:                 inactive  (disabled on boot)
  openstack-nova-consoleauth:             active
  openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
  == Glance services ==
  openstack-glance-api:                   active
  openstack-glance-registry:              active
  == Keystone service ==
  openstack-keystone:                     inactive  (disabled on boot)
  == Support services ==
  mysqld:                                 inactive  (disabled on boot)
  dbus:                                   active
  rabbitmq-server:                        active
  memcached:                              active
  == Keystone users ==
  /usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
    'python-keystoneclient.', DeprecationWarning)
  WARNING: unsupported identity-api-version 3, falling back to 2.0
  /usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
    'the 2.0.0 release.', DeprecationWarning)
  /usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
    super(Client, self).__init__(**kwargs)
  /usr/lib/python2.7/site-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
    return f(*args, **kwargs)
  /usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:376: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
    'the 2.0.0 release.', DeprecationWarning)
  Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-7b248df7-dbc6-4bc8-be72-862c2e622994)
  == Glance images ==
  +--------------------------------------+--------+
  | ID                                   | Name   |
  +--------------------------------------+--------+
  | b90fc7b1-5f7e-4a72-87ce-0b7f21ca2afc | cirros |
  +--------------------------------------+--------+
  == Nova managed services ==
  +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
  | Id | Binary           | Host       | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
  | 1  | nova-conductor   | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
  | 2  | nova-consoleauth | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
  | 3  | nova-scheduler   | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
  | 4  | nova-cert        | controller | internal | enabled | up    | 2016-03-17T13:22:27.000000 | -               |
  +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
  == Nova networks ==
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'keystoneclient.exceptions.ConnectionRefused'> (HTTP 500) (Request-ID: req-2b2cd91c-3fbd-4baa-87f3-bec2c57039e7)
  == Nova instance flavors ==
  +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
  | ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
  +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
  | 1  | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
  | 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
  | 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
  | 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
  | 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
  +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
  == Nova instances ==
  +----+------+-----------+--------+------------+-------------+----------+
  | ID | Name | Tenant ID | Status | Task State | Power State | Networks |
  +----+------+-----------+--------+------------+-------------+----------+
  +----+------+-----------+--------+------------+-------------+----------+"


  3, Seems port 5000 and 35357 is not opened by httpd:

  "[root@controller ~]# systemctl status httpd.service 
  鈼[0m httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2016-03-17 08:49:46 EDT; 1h 58min ago
       Docs: man:httpd(8)
             man:apachectl(8)
   Main PID: 1366 (httpd)
     Status: "Total requests: 60; Current requests/sec: 0; Current traffic:   0 B/sec"
     CGroup: /system.slice/httpd.service
             鈹溾攢1366 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢2696 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2697 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2698 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2699 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2700 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2701 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2702 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2703 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2704 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2705 (wsgi:keystone- -DFOREGROUND
             鈹溾攢2706 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢2707 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢2708 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢2709 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢2710 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢3307 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢3370 /usr/sbin/httpd -DFOREGROUND
             鈹溾攢3371 /usr/sbin/httpd -DFOREGROUND
             鈹斺攢3372 /usr/sbin/httpd -DFOREGROUND

  Mar 17 08:49:44 controller systemd[1]: Starting The Apache HTTP Server...
  Mar 17 08:49:46 controller systemd[1]: Started The Apache HTTP Server."

  
  4, /var/log/keystone/keystone.log attached:

  "2016-03-17 09:22:28.404 2702 INFO keystone.common.wsgi [req-e127518c-de2b-4484-8359-d1324f05dd51 - - - - -] GET http://controller:35357/v3/
  2016-03-17 09:22:28.408 2701 INFO keystone.common.wsgi [req-ce27bdf1-254b-4dee-983e-0ae159231daf - - - - -] POST http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:28.483 2703 INFO keystone.common.wsgi [req-f56d0bb4-9468-458f-a591-52d9f74be7ba - - - - -] GET http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:28.734 2705 INFO keystone.common.wsgi [req-58077650-ee50-426d-a7b6-a666baf63602 - - - - -] GET http://controller:35357/v3/
  2016-03-17 09:22:28.737 2702 INFO keystone.common.wsgi [req-f67b247f-3854-432e-aba7-8cd895501723 - - - - -] POST http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:28.780 2704 INFO keystone.common.wsgi [req-1d5125c4-3fe8-4e80-bbd6-8a191f65e285 - - - - -] GET http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:29.109 2705 INFO keystone.common.wsgi [req-2e5a3426-a7d9-4e01-87d1-5bbc9861c2f6 - - - - -] GET http://controller:35357/v3/
  2016-03-17 09:22:29.113 2703 INFO keystone.common.wsgi [req-2a8f7eaf-243b-4da0-9e11-043e9529b682 - - - - -] POST http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:29.154 2701 INFO keystone.common.wsgi [req-56241137-d85c-4522-9a5c-1da4db689760 - - - - -] GET http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:29.471 2705 INFO keystone.common.wsgi [req-a19d48e5-8f43-4a83-bb5e-35960976383b - - - - -] GET http://controller:35357/v3/
  2016-03-17 09:22:29.475 2703 INFO keystone.common.wsgi [req-262f60aa-cd8a-4056-b8bf-ed19452c7210 - - - - -] POST http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:29.516 2704 INFO keystone.common.wsgi [req-8b369ab3-f791-45b5-bfe5-142dde22bdb2 - - - - -] GET http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:29.841 2702 INFO keystone.common.wsgi [req-f93c2103-cb88-4e46-b21a-a71291c9ff13 - - - - -] GET http://controller:35357/v3/
  2016-03-17 09:22:29.844 2703 INFO keystone.common.wsgi [req-7766d108-1671-4080-9d12-ebbcb01ec1f5 - - - - -] POST http://controller:35357/v3/auth/tokens
  2016-03-17 09:22:29.885 2705 INFO keystone.common.wsgi [req-22c3d476-d439-4ef7-9b13-3f2989f02c7f - - - - -] GET http://controller:35357/v3/auth/tokens
  2016-03-17 09:24:56.072 2701 INFO keystone.common.wsgi [req-9251220e-62ad-475a-ad75-8efbd9840abb - - - - -] GET http://controller:35357/v3/
  2016-03-17 09:24:56.075 2703 INFO keystone.common.wsgi [req-2a59386a-09d3-46de-b1fb-7044082058e3 - - - - -] POST http://controller:35357/v3/auth/tokens
  2016-03-17 09:24:56.116 2705 INFO keystone.common.wsgi [req-77bb05dd-c74d-4862-afee-d8afdb584082 - - - - -] GET http://controller:35357/v3/auth/tokens";



  5, /var/log/httpd/error_log attached:

  "[Thu Mar 17 07:52:48.247443 2016] [mpm_prefork:notice] [pid 1358] AH00170: caught SIGWINCH, shutting down gracefully
  [Thu Mar 17 07:55:01.758180 2016] [core:notice] [pid 1360] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
  [Thu Mar 17 07:55:02.045384 2016] [suexec:notice] [pid 1360] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
  [Thu Mar 17 07:55:02.053827 2016] [auth_digest:notice] [pid 1360] AH01757: generating secret for digest authentication ...
  [Thu Mar 17 07:55:02.054499 2016] [lbmethod_heartbeat:notice] [pid 1360] AH02282: No slotmem from mod_heartmonitor
  [Thu Mar 17 07:55:02.058430 2016] [mpm_prefork:notice] [pid 1360] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
  [Thu Mar 17 07:55:02.058446 2016] [core:notice] [pid 1360] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
  [Thu Mar 17 08:27:39.730480 2016] [mpm_prefork:notice] [pid 1360] AH00170: caught SIGWINCH, shutting down gracefully
  [Thu Mar 17 08:29:55.642727 2016] [core:notice] [pid 1347] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
  [Thu Mar 17 08:29:55.708727 2016] [suexec:notice] [pid 1347] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
  [Thu Mar 17 08:29:55.717232 2016] [auth_digest:notice] [pid 1347] AH01757: generating secret for digest authentication ...
  [Thu Mar 17 08:29:55.717886 2016] [lbmethod_heartbeat:notice] [pid 1347] AH02282: No slotmem from mod_heartmonitor
  [Thu Mar 17 08:29:55.721848 2016] [mpm_prefork:notice] [pid 1347] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
  [Thu Mar 17 08:29:55.721865 2016] [core:notice] [pid 1347] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
  [Thu Mar 17 08:47:32.275843 2016] [mpm_prefork:notice] [pid 1347] AH00170: caught SIGWINCH, shutting down gracefully
  [Thu Mar 17 08:49:46.323504 2016] [core:notice] [pid 1366] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
  [Thu Mar 17 08:49:46.344285 2016] [suexec:notice] [pid 1366] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
  [Thu Mar 17 08:49:46.353395 2016] [auth_digest:notice] [pid 1366] AH01757: generating secret for digest authentication ...
  [Thu Mar 17 08:49:46.354157 2016] [lbmethod_heartbeat:notice] [pid 1366] AH02282: No slotmem from mod_heartmonitor
  [Thu Mar 17 08:49:46.358306 2016] [mpm_prefork:notice] [pid 1366] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
  [Thu Mar 17 08:49:46.358331 2016] [core:notice] [pid 1366] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'" 


  
  6, /var/log/httpd/keystone-access.log attached:

  "::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
  ::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
  ::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
  ::1 - - [17/Mar/2016:08:47:32 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)"
  135.251.157.109 - - [17/Mar/2016:08:50:11 -0400] "POST /v3/tokens HTTP/1.1" 404 93 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:13 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:13 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:14 -0400] "GET / HTTP/1.1" 300 591 "-" "glance/11.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:14 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2260 "-" "glance/11.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:15 -0400] "GET / HTTP/1.1" 300 591 "-" "glance/11.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:15 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2258 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:16 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:17 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2258 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET / HTTP/1.1" 300 591 "-" "nova/12.0.1 keystonemiddleware.auth_token/2.3.1"
  135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:18 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:19 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:19 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:19 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:36 -0400] "POST /v3/tokens HTTP/1.1" 404 93 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:37 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:08:50:38 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "POST /v3/tokens HTTP/1.1" 404 93 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:28 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:22:29 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:24:56 -0400] "GET /v3 HTTP/1.1" 200 250 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:24:56 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 2257 "-" "python-keystoneclient"
  135.251.157.109 - - [17/Mar/2016:09:24:56 -0400] "GET /v3/auth/tokens HTTP/1.1" 200 2257 "-" "python-keystoneclient""



  7, /var/log/httpd/keystone-error.log attached:

  "2016-03-17 08:50:12.323143 No handlers could be found for logger "oslo_config.cfg"
  2016-03-17 08:50:13.490936 No handlers could be found for logger "oslo_config.cfg"
  2016-03-17 08:50:13.955267 No handlers could be found for logger "oslo_config.cfg"
  2016-03-17 08:50:14.726416 No handlers could be found for logger "oslo_config.cfg"
  2016-03-17 08:50:15.159876 No handlers could be found for logger "oslo_config.cfg""

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1558384/+subscriptions


Follow ups