yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #80562
[Bug 1850759] [NEW] Keystone performance degradation in stein comparing with queens
Public bug reported:
I install two ubuntu vms using the same resources (cpu/memory/disks) to
setup devstack. One devstack is using stein branch and the other
devstack is using queens branch.
I have run the same keystone test cases in the two devstack environments
using rally benchmark tool. And the result is keystone performance is
worse in stein comparing with queens (comparing average time for each
keystone api shown below in rally task report).
For Queens devstack env:
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.045 | 0.045 | 0.045 | 0.045 | 0.045 | 0.045 | 100.0% | 1 |
| keystone_v3.validate_token | 0.073 | 0.073 | 0.073 | 0.073 | 0.073 | 0.073 | 100.0% | 1 |
| total | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 100.0% | 1 |
| -> duration | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 100.0% | 1 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 1 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 0.117612
Full duration: 2.366941
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.04 | 0.043 | 0.046 | 0.048 | 0.127 | 0.046 | 100.0% | 30 |
| keystone_v3.validate_token | 0.063 | 0.07 | 0.075 | 0.078 | 0.082 | 0.071 | 100.0% | 30 |
| total | 0.107 | 0.113 | 0.122 | 0.126 | 0.194 | 0.117 | 100.0% | 30 |
| -> duration | 0.107 | 0.113 | 0.122 | 0.126 | 0.194 | 0.117 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 6.699078
Full duration: 8.961246
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.create_project | 0.112 | 0.119 | 0.126 | 0.128 | 0.131 | 0.12 | 100.0% | 30 |
| keystone_v3.create_user | 0.131 | 0.14 | 0.151 | 0.152 | 0.156 | 0.14 | 100.0% | 30 |
| -> keystone_v3.list_roles | 0.018 | 0.024 | 0.027 | 0.029 | 0.032 | 0.024 | 100.0% | 30 |
| -> keystone_v3.add_role | 0.039 | 0.043 | 0.046 | 0.049 | 0.051 | 0.043 | 100.0% | 30 |
| keystone_v3.create_role | 0.021 | 0.023 | 0.026 | 0.026 | 0.027 | 0.023 | 100.0% | 30 |
| keystone_v3.get_project | 0.015 | 0.018 | 0.02 | 0.02 | 0.02 | 0.018 | 100.0% | 30 |
| keystone_v3.get_user | 0.014 | 0.017 | 0.018 | 0.019 | 0.02 | 0.017 | 100.0% | 30 |
| keystone_v3.get_role | 0.015 | 0.018 | 0.019 | 0.02 | 0.021 | 0.018 | 100.0% | 30 |
| keystone_v3.list_services | 0.017 | 0.021 | 0.023 | 0.024 | 0.027 | 0.021 | 100.0% | 30 |
| keystone_v3.get_services | 0.013 | 0.017 | 0.02 | 0.021 | 0.022 | 0.017 | 100.0% | 30 |
| total | 0.368 | 0.392 | 0.412 | 0.416 | 0.421 | 0.393 | 100.0% | 30 |
| -> duration | 0.368 | 0.392 | 0.412 | 0.416 | 0.421 | 0.393 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 17.211363
Full duration: 31.490424
For Stein environment:
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.047 | 0.047 | 0.047 | 0.047 | 0.047 | 0.047 | 100.0% | 1 |
| keystone_v3.validate_token | 0.09 | 0.09 | 0.09 | 0.09 | 0.09 | 0.09 | 100.0% | 1 |
| total | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 100.0% | 1 |
| -> duration | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 100.0% | 1 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 1 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 0.137515
Full duration: 2.635841
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.043 | 0.052 | 0.056 | 0.058 | 0.063 | 0.051 | 100.0% | 30 |
| keystone_v3.validate_token | 0.083 | 0.103 | 0.11 | 0.112 | 0.242 | 0.104 | 100.0% | 30 |
| total | 0.13 | 0.156 | 0.164 | 0.171 | 0.295 | 0.155 | 100.0% | 30 |
| -> duration | 0.13 | 0.156 | 0.164 | 0.171 | 0.295 | 0.155 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 9.240253
Full duration: 12.131328
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.create_project | 0.153 | 0.161 | 0.168 | 0.169 | 0.235 | 0.163 | 100.0% | 30 |
| keystone_v3.create_user | 0.2 | 0.217 | 0.222 | 0.225 | 0.29 | 0.218 | 100.0% | 30 |
| -> keystone_v3.list_roles | 0.025 | 0.03 | 0.033 | 0.034 | 0.036 | 0.03 | 100.0% | 30 |
| -> keystone_v3.add_role | 0.043 | 0.047 | 0.048 | 0.05 | 0.057 | 0.047 | 100.0% | 30 |
| keystone_v3.create_role | 0.03 | 0.032 | 0.033 | 0.034 | 0.043 | 0.032 | 100.0% | 30 |
| keystone_v3.get_project | 0.022 | 0.026 | 0.027 | 0.027 | 0.028 | 0.025 | 100.0% | 30 |
| keystone_v3.get_user | 0.021 | 0.024 | 0.026 | 0.027 | 0.028 | 0.024 | 100.0% | 30 |
| keystone_v3.get_role | 0.022 | 0.024 | 0.026 | 0.026 | 0.027 | 0.024 | 100.0% | 30 |
| keystone_v3.list_services | 0.023 | 0.027 | 0.029 | 0.03 | 0.03 | 0.027 | 100.0% | 30 |
| keystone_v3.get_services | 0.02 | 0.024 | 0.027 | 0.028 | 0.031 | 0.025 | 100.0% | 30 |
| total | 0.579 | 0.597 | 0.609 | 0.641 | 0.673 | 0.601 | 100.0% | 30 |
| -> duration | 0.579 | 0.597 | 0.609 | 0.641 | 0.673 | 0.601 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 22.954598
Full duration: 39.454973
** Affects: keystone
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1850759
Title:
Keystone performance degradation in stein comparing with queens
Status in OpenStack Identity (keystone):
New
Bug description:
I install two ubuntu vms using the same resources (cpu/memory/disks)
to setup devstack. One devstack is using stein branch and the other
devstack is using queens branch.
I have run the same keystone test cases in the two devstack
environments using rally benchmark tool. And the result is keystone
performance is worse in stein comparing with queens (comparing average
time for each keystone api shown below in rally task report).
For Queens devstack env:
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.045 | 0.045 | 0.045 | 0.045 | 0.045 | 0.045 | 100.0% | 1 |
| keystone_v3.validate_token | 0.073 | 0.073 | 0.073 | 0.073 | 0.073 | 0.073 | 100.0% | 1 |
| total | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 100.0% | 1 |
| -> duration | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 0.118 | 100.0% | 1 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 1 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 0.117612
Full duration: 2.366941
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.04 | 0.043 | 0.046 | 0.048 | 0.127 | 0.046 | 100.0% | 30 |
| keystone_v3.validate_token | 0.063 | 0.07 | 0.075 | 0.078 | 0.082 | 0.071 | 100.0% | 30 |
| total | 0.107 | 0.113 | 0.122 | 0.126 | 0.194 | 0.117 | 100.0% | 30 |
| -> duration | 0.107 | 0.113 | 0.122 | 0.126 | 0.194 | 0.117 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 6.699078
Full duration: 8.961246
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.create_project | 0.112 | 0.119 | 0.126 | 0.128 | 0.131 | 0.12 | 100.0% | 30 |
| keystone_v3.create_user | 0.131 | 0.14 | 0.151 | 0.152 | 0.156 | 0.14 | 100.0% | 30 |
| -> keystone_v3.list_roles | 0.018 | 0.024 | 0.027 | 0.029 | 0.032 | 0.024 | 100.0% | 30 |
| -> keystone_v3.add_role | 0.039 | 0.043 | 0.046 | 0.049 | 0.051 | 0.043 | 100.0% | 30 |
| keystone_v3.create_role | 0.021 | 0.023 | 0.026 | 0.026 | 0.027 | 0.023 | 100.0% | 30 |
| keystone_v3.get_project | 0.015 | 0.018 | 0.02 | 0.02 | 0.02 | 0.018 | 100.0% | 30 |
| keystone_v3.get_user | 0.014 | 0.017 | 0.018 | 0.019 | 0.02 | 0.017 | 100.0% | 30 |
| keystone_v3.get_role | 0.015 | 0.018 | 0.019 | 0.02 | 0.021 | 0.018 | 100.0% | 30 |
| keystone_v3.list_services | 0.017 | 0.021 | 0.023 | 0.024 | 0.027 | 0.021 | 100.0% | 30 |
| keystone_v3.get_services | 0.013 | 0.017 | 0.02 | 0.021 | 0.022 | 0.017 | 100.0% | 30 |
| total | 0.368 | 0.392 | 0.412 | 0.416 | 0.421 | 0.393 | 100.0% | 30 |
| -> duration | 0.368 | 0.392 | 0.412 | 0.416 | 0.421 | 0.393 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 17.211363
Full duration: 31.490424
For Stein environment:
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.047 | 0.047 | 0.047 | 0.047 | 0.047 | 0.047 | 100.0% | 1 |
| keystone_v3.validate_token | 0.09 | 0.09 | 0.09 | 0.09 | 0.09 | 0.09 | 100.0% | 1 |
| total | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 100.0% | 1 |
| -> duration | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 0.138 | 100.0% | 1 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 1 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 0.137515
Full duration: 2.635841
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.fetch_token | 0.043 | 0.052 | 0.056 | 0.058 | 0.063 | 0.051 | 100.0% | 30 |
| keystone_v3.validate_token | 0.083 | 0.103 | 0.11 | 0.112 | 0.242 | 0.104 | 100.0% | 30 |
| total | 0.13 | 0.156 | 0.164 | 0.171 | 0.295 | 0.155 | 100.0% | 30 |
| -> duration | 0.13 | 0.156 | 0.164 | 0.171 | 0.295 | 0.155 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 9.240253
Full duration: 12.131328
+-------------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.create_project | 0.153 | 0.161 | 0.168 | 0.169 | 0.235 | 0.163 | 100.0% | 30 |
| keystone_v3.create_user | 0.2 | 0.217 | 0.222 | 0.225 | 0.29 | 0.218 | 100.0% | 30 |
| -> keystone_v3.list_roles | 0.025 | 0.03 | 0.033 | 0.034 | 0.036 | 0.03 | 100.0% | 30 |
| -> keystone_v3.add_role | 0.043 | 0.047 | 0.048 | 0.05 | 0.057 | 0.047 | 100.0% | 30 |
| keystone_v3.create_role | 0.03 | 0.032 | 0.033 | 0.034 | 0.043 | 0.032 | 100.0% | 30 |
| keystone_v3.get_project | 0.022 | 0.026 | 0.027 | 0.027 | 0.028 | 0.025 | 100.0% | 30 |
| keystone_v3.get_user | 0.021 | 0.024 | 0.026 | 0.027 | 0.028 | 0.024 | 100.0% | 30 |
| keystone_v3.get_role | 0.022 | 0.024 | 0.026 | 0.026 | 0.027 | 0.024 | 100.0% | 30 |
| keystone_v3.list_services | 0.023 | 0.027 | 0.029 | 0.03 | 0.03 | 0.027 | 100.0% | 30 |
| keystone_v3.get_services | 0.02 | 0.024 | 0.027 | 0.028 | 0.031 | 0.025 | 100.0% | 30 |
| total | 0.579 | 0.597 | 0.609 | 0.641 | 0.673 | 0.601 | 100.0% | 30 |
| -> duration | 0.579 | 0.597 | 0.609 | 0.641 | 0.673 | 0.601 | 100.0% | 30 |
| -> idle_duration | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0% | 30 |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Load duration: 22.954598
Full duration: 39.454973
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1850759/+subscriptions
Follow ups
