yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #68653
[Bug 1724686] [NEW] authentication code hangs when there are three or more admin keystone endpoints
Public bug reported:
I'm running stable/pike devstack, and I was playing around with what
happens when there are many endpoints in multiple regions, and I
stumbled over a scenario where the keystone authentication code hangs.
My original endpoint list looked like this:
ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 |
| 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement |
| 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity |
| 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image |
| 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 |
| 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 |
| 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 |
| 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity |
| a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s |
| b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s |
| bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s |
| e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ |
| f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
I was able to successfully run the following python code:
from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client
loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
keystone.services.list()
I then duplicated all of the endpoints in a new region "region2", and
was able to run the python code. When I duplicated all the endpoints
again in a new region "region3" (for a total of 39 endpoints) the python
code hung at the final line.
Removing all the "region3" endpoints allowed the python code to work
again.
During all of this the command "openstack endpoint list" worked fine.
Further testing seems to indicate that it is the third "admin" keystone
endpoint that is causing the problem. I can add multiple "public"
keystone endpoints, but three or more "admin" keystone endpoints cause
the python code to hang.
** Affects: keystone
Importance: Undecided
Status: New
** Summary changed:
- authentication code hangs when there are many endpoints
+ authentication code hangs when there are three or more admin keystone endpoints
** Description changed:
I'm running stable/pike devstack, and I was playing around with what
happens when there are many endpoints in multiple regions, and I
stumbled over a scenario where the keystone authentication code hangs.
My original endpoint list looked like this:
ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 |
| 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement |
| 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity |
| 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image |
| 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 |
| 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 |
| 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 |
| 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity |
| a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s |
| b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s |
| bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s |
| e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ |
| f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
I was able to successfully run the following python code:
from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client
loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
keystone.services.list()
-
- I then duplicated all of the endpoints in a new region "region2", and was able to run the python code. When I duplicated all the endpoints again in a new region "region3" (for a total of 39 endpoints) the python code hung at the final line.
+ I then duplicated all of the endpoints in a new region "region2", and
+ was able to run the python code. When I duplicated all the endpoints
+ again in a new region "region3" (for a total of 39 endpoints) the python
+ code hung at the final line.
Removing all the "region3" endpoints allowed the python code to work
again.
During all of this the command "openstack endpoint list" worked fine.
- Further testing seems to indicate that it is the third keystone endpoint
- that is causing the problem. I can add all the others, but if I add the
- third keystone endpoint (regardless of what order I add them in) then
- the python code hangs.
+ Further testing seems to indicate that it is the third "admin" keystone
+ endpoint that is causing the problem. I can add all the others, but if
+ I add the third keystone endpoint (regardless of what order I add them
+ in) then the python code hangs.
** Description changed:
I'm running stable/pike devstack, and I was playing around with what
happens when there are many endpoints in multiple regions, and I
stumbled over a scenario where the keystone authentication code hangs.
My original endpoint list looked like this:
ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 |
| 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement |
| 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity |
| 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image |
| 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 |
| 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 |
| 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 |
| 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity |
| a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s |
| b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s |
| bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s |
| e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ |
| f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
I was able to successfully run the following python code:
from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client
loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
keystone.services.list()
I then duplicated all of the endpoints in a new region "region2", and
was able to run the python code. When I duplicated all the endpoints
again in a new region "region3" (for a total of 39 endpoints) the python
code hung at the final line.
Removing all the "region3" endpoints allowed the python code to work
again.
During all of this the command "openstack endpoint list" worked fine.
Further testing seems to indicate that it is the third "admin" keystone
- endpoint that is causing the problem. I can add all the others, but if
- I add the third keystone endpoint (regardless of what order I add them
- in) then the python code hangs.
+ endpoint that is causing the problem. I can add multiple "public"
+ keystone endpoints, but three or more "admin" keystone endpoints cause
+ the python code to hang.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1724686
Title:
authentication code hangs when there are three or more admin keystone
endpoints
Status in OpenStack Identity (keystone):
New
Bug description:
I'm running stable/pike devstack, and I was playing around with what
happens when there are many endpoints in multiple regions, and I
stumbled over a scenario where the keystone authentication code hangs.
My original endpoint list looked like this:
ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
| 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 |
| 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement |
| 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity |
| 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image |
| 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 |
| 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 |
| 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 |
| 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity |
| a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s |
| b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s |
| bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s |
| e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ |
| f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+
I was able to successfully run the following python code:
from keystoneauth1 import loading
from keystoneauth1 import loading
from keystoneauth1 import session
from keystoneclient.v3 import client
loader = loading.get_plugin_loader("password")
auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)
keystone.services.list()
I then duplicated all of the endpoints in a new region "region2", and
was able to run the python code. When I duplicated all the endpoints
again in a new region "region3" (for a total of 39 endpoints) the
python code hung at the final line.
Removing all the "region3" endpoints allowed the python code to work
again.
During all of this the command "openstack endpoint list" worked fine.
Further testing seems to indicate that it is the third "admin"
keystone endpoint that is causing the problem. I can add multiple
"public" keystone endpoints, but three or more "admin" keystone
endpoints cause the python code to hang.
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1724686/+subscriptions
Follow ups
