← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1587777] Re: Mitaka: dashboard performance

 

I changed the status back to new because the docs for Ocata still don't
contain the required information to configure memcached correctly,
please note comment #8.

** Changed in: keystone
       Status: Invalid => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1587777

Title:
  Mitaka: dashboard performance

Status in OpenStack Identity (keystone):
  New

Bug description:
  Environment: Openstack Mitaka on top of Leap 42.1, 1 control node, 2
  compute nodes, 3-node-Ceph-cluster.

  Issue: Since switching to Mitaka, we're experiencing severe delays
  when accessing the dashboard - i.e. switching between "Compute -
  Overview" and "Compute - Instances" takes 15+ seconds, even after
  multiple invocations.

  Steps to reproduce:
  1. Install Openstack Mitaka, incl. dashboard & navigate through the dashboard.

  Expected result:
  Browsing through the dashboard with reasonable waiting times.

  Actual result:
  Refreshing the dashboard can take up to 30 secs, switching between views (e.g. volumes to instances) takes about 15 secs in average.

  Additional information:
  I've had a look at the requests, the Apache logs and our control node's stats and noticed that it's a single call that's taking all the time... I see no indications of any error, it seems that once WSGI is invoked, that call simply takes its time. Intermediate curl requests are logged, so I see it's doing its work. Looking at "vmstat" I can see that it's user space taking all the load (Apache / mod_wsgi drives its CPU to 100%, while other CPUs are idle - and no i/o wait, no system space etc.).

  ---cut here---
  control1:/var/log # top
  top - 10:51:35 up 8 days, 18:16,  2 users,  load average: 2,17, 1,65, 1,48
  Tasks: 383 total,   2 running, 381 sleeping,   0 stopped,   0 zombie
  %Cpu0  : 31,7 us,  2,9 sy,  0,0 ni, 65,0 id,  0,3 wa,  0,0 hi,  0,0 si,  0,0 st
  %Cpu1  : 13,1 us,  0,7 sy,  0,0 ni, 86,2 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
  %Cpu2  : 17,2 us,  0,7 sy,  0,0 ni, 81,2 id,  1,0 wa,  0,0 hi,  0,0 si,  0,0 st
  %Cpu3  : 69,4 us, 12,6 sy,  0,0 ni, 17,9 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
  %Cpu4  : 14,6 us,  1,0 sy,  0,0 ni, 84,4 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
  %Cpu5  : 16,9 us,  0,7 sy,  0,0 ni, 81,7 id,  0,7 wa,  0,0 hi,  0,0 si,  0,0 st
  %Cpu6  : 17,3 us,  1,3 sy,  0,0 ni, 81,0 id,  0,3 wa,  0,0 hi,  0,0 si,  0,0 st
  %Cpu7  : 21,2 us,  1,3 sy,  0,0 ni, 77,5 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
  KiB Mem:  65943260 total, 62907676 used,  3035584 free,     1708 buffers
  KiB Swap:  2103292 total,        0 used,  2103292 free. 53438560 cached Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6776 wwwrun    20   0  565212 184504  13352 S 100,3 0,280   0:07.83 httpd-prefork
   1130 root      20   0  399456  35760  22508 S 5,980 0,054 818:13.17 X
   1558 sddm      20   0  922744 130440  72148 S 5,316 0,198 966:03.82 sddm-greeter
  20999 nova      20   0  285888 116292   5696 S 2,658 0,176 164:27.08 nova-conductor
  21030 nova      20   0  758752 182644  16512 S 2,658 0,277  58:20.40 nova-api
  18757 heat      20   0  273912  73740   4612 S 2,326 0,112  50:48.72 heat-engine
  18759 heat      20   0  273912  73688   4612 S 2,326 0,112   4:27.54 heat-engine
  20995 nova      20   0  286236 116644   5696 S 2,326 0,177 164:38.89 nova-conductor
  21027 nova      20   0  756204 180752  16980 S 2,326 0,274  58:20.09 nova-api
  21029 nova      20   0  756536 180644  16496 S 2,326 0,274 139:46.29 nova-api
  21031 nova      20   0  756888 180920  16512 S 2,326 0,274  58:36.37 nova-api
  24771 glance    20   0 2312152 139000  17360 S 2,326 0,211  24:47.83 glance-api
  24772 glance    20   0  631672 111248   4848 S 2,326 0,169  22:59.77 glance-api
  28424 cinder    20   0  720972 108536   4968 S 2,326 0,165  28:31.42 cinder-api
  28758 neutron   20   0  317708 101812   4472 S 2,326 0,154 153:45.55 neutron-server

  #####################################

  control1:/var/log # vmstat 1
  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
   1  0      0 2253144   1708 53440472    0    0     4    60    4    4 11  1 88  0  0
   0  0      0 2255588   1708 53440476    0    0     0   568 3063 7627 15  1 83  0  0
   1  0      0 2247596   1708 53440476    0    0     0   144 3066 6803 14  2 83  0  0
   1  0      0 2156008   1708 53440476    0    0     0    72 3474 7193 25  3 72  0  0
   2  0      0 2131968   1708 53440484    0    0     0   652 3497 8565 28  2 70  0  0
   3  1      0 2134000   1708 53440512    0    0     0 14340 3629 10644 25  2 71  2  0
   2  0      0 2136956   1708 53440580    0    0     0    12 3483 10620 25  2 70  3  0
   9  1      0 2138164   1708 53440596    0    0     0   248 3442 9980 27  1 72  0  0
   4  0      0 2105160   1708 53440628    0    0     0   428 3617 22791 27  2 70  1  0
   3  0      0 2093416   1708 53440644    0    0     0  1216 3502 7917 25  2 72  1  0
   3  0      0 2096344   1708 53440636    0    0     0    72 3555 9216 25  2 73  0  0
   6  0      0 2073564   1708 53440636    0    0     0    72 3587 11160 28  2 70  0  0
   2  0      0 2070236   1708 53440636    0    0     0   432 3854 8160 26  4 70  0  0
   1  0      0 2103628   1708 53440640    0    0     0    76 3407 7492 25  3 73  0  0
   3  0      0 2100320   1708 53440636    0    0     0  1384 3383 7955 24  2 73  1  0
   3  0      0 2100648   1708 53440644    0    0     0   216 3571 12276 33  2 65  0  0
   1  0      0 2091808   1708 53440672    0    0     0    72 3734 12848 36  2 62  0  0
   1  0      0 2092400   1708 53440700    0    0     0     0 4029 16556 23  1 77  0  0
   1  0      0 2092400   1708 53440696    0    0     0    72 2938 6621 14  1 85  0  0
   0  0      0 2092080   1708 53440696    0    0     0   704 3067 6958 15  1 83  1  0
   0  0      0 2092048   1708 53440696    0    0     0   360 3222 7279 14  1 85  1  0

  ---cut here---

  Current test, switching to "Volumes" took 7 secs loading
  http://control1/project/volumes/. Switching back to "Instances" took
  14.5 secs for http://control1/project/instances/. Here is the output
  from /var/log/apache2/openstack-dashboard-access_log, we added the
  duration to the log file.

  ---cut here---
  [01/Jun/2016:09:50:13 +0200] "GET /project/volumes/ HTTP/1.1" 200 42683 "http://control1/project/instances/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 6905 ms
  [01/Jun/2016:09:50:20 +0200] "GET /static/dashboard/css/b66201fdb62c.css HTTP/1.1" 200 856329 "http://control1/project/volumes/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 25 ms
  [01/Jun/2016:09:50:20 +0200] "GET /static/dashboard/js/63c5c82dd695.js HTTP/1.1" 200 1194379 "http://control1/project/volumes/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 270 ms
  [01/Jun/2016:09:50:20 +0200] "GET /i18n/js/horizon+openstack_dashboard/ HTTP/1.1" 200 60400 "http://control1/project/volumes/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 1622 ms
  [01/Jun/2016:09:50:38 +0200] "GET /project/instances/ HTTP/1.1" 200 43547 "http://control1/project/volumes/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 14576 ms
  [01/Jun/2016:09:50:53 +0200] "GET /static/dashboard/css/b66201fdb62c.css HTTP/1.1" 200 856329 "http://control1/project/instances/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 22 ms
  [01/Jun/2016:09:50:53 +0200] "GET /static/dashboard/js/63c5c82dd695.js HTTP/1.1" 200 1194379 "http://control1/project/instances/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 32 ms
  [01/Jun/2016:09:50:53 +0200] "GET /i18n/js/horizon+openstack_dashboard/ HTTP/1.1" 200 60400 "http://control1/project/instances/"; "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" Duration 46 ms
  ---cut here---

  We did not see this with previous releases, we used Liberty before and upgraded to Mitaka. 
  Just for reference, we're running the following version of Horizon:

  --- cut here ---
  control1:/etc/apache2/conf.d # rpm -qi openstack-dashboard-9.0.1~a0~dev12-2.1.noarch
  Name        : openstack-dashboard
  Version     : 9.0.1~a0~dev12
  Release     : 2.1
  Architecture: noarch
  Install Date: Tue May 10 12:39:04 2016
  Group       : Development/Languages/Python
  Size        : 51400386
  License     : Apache-2.0
  Signature   : RSA/SHA256, Mon May  9 11:07:15 2016, Key ID 893a90dad85f9316
  Source RPM  : openstack-dashboard-9.0.1~a0~dev12-2.1.src.rpm
  Build Date  : Mon May  9 11:06:31 2016
  Build Host  : build81
  Relocations : (not relocatable)
  Vendor      : obs://build.opensuse.org/Cloud:OpenStack
  URL         : http://wiki.openstack.org/OpenStackDashboard
  --- cut here ---

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1587777/+subscriptions


References