← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1412961] [NEW] storing multiple heat snapshots causing significant memory consumption

 

You have been subscribed to a public bug:

Running a randomized snapshot + restore workload on a single heat stack
(single vm) with 2 concurrent users causes significant memory
consumption that continues even after all tests are completed.

KiB Mem:  37066812 total,  6883732 used, 30183080 free,   244800 buffers
KiB Mem:  37066812 total, 35717836 used,  1348976 free,    29860 buffers <- post workload

Tests:
clone https://github.com/pcrews/rannsaka
cd rannsaka/rannsaka
python rannsaka.py --host=http://192.168.0.5 --requests 500 -w 2 --test-file=locust_files/heat_basic_stress.py

# PRE STRESS TEST
top - 12:01:42 up  1:04,  2 users,  load average: 0.87, 0.87, 0.58
Tasks: 368 total,   2 running, 366 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.3 us,  0.3 sy,  0.3 ni, 96.5 id,  1.6 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  37066812 total,  6883732 used, 30183080 free,   244800 buffers
KiB Swap: 37738492 total,        0 used, 37738492 free.  2516732 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                  
14687 stack     20   0  221808   8092   4596 R  99.3  0.0   0:05.49 qemu-img                                                                                                                                 
 2208 rabbitmq  20   0 2349296  66804   2516 S   6.2  0.2   0:28.53 beam.smp                                                                                                                                 
12106 stack     20   0  173852  73228   5084 S   6.2  0.2   0:03.91 glance-api                                                                                                                               
14705 stack     20   0   25212   1728   1088 R   6.2  0.0   0:00.01 top                                                                                                                                      


# POST STRESS TEST
top - 12:46:17 up  1:49,  3 users,  load average: 0.34, 0.53, 1.39
Tasks: 376 total,   2 running, 374 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.2 sy,  0.0 ni, 98.5 id,  0.5 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  37066812 total, 35717836 used,  1348976 free,    29860 buffers
KiB Swap: 37738492 total,   251000 used, 37487492 free. 30435868 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                  
21654 libvirt+  20   0 4812112 491872   9416 S   0.3  1.3   0:15.80 qemu-system-x86                                                                                                                          
21402 libvirt+  20   0 4743248 481652   9392 S   0.3  1.3   0:36.78 qemu-system-x86                                                                                                                          
12281 stack     20   0  340540 145080   3732 S   0.0  0.4   0:12.59 nova-api                                                                                                                                 
12282 stack     20   0  340524 144968   3760 S   0.0  0.4   0:12.53 nova-api                                                                                                                                 
12280 stack     20   0  339660 143952   3736 S   0.0  0.4   0:12.08 nova-api                                                                                                                                 
12279 stack     20   0  337568 141988   3732 S   0.0  0.4   0:13.15 nova-api                                                                                                                                 
 9784 mysql     20   0 4099628 121912   7984 S   0.3  0.3   0:34.01 mysqld                                                                                                                                   
12423 stack     20   0  276484  89216   3608 S   0.0  0.2   0:12.66 nova-conductor                                                                                                                           
12428 stack     20   0  276584  89184   3604 S   0.3  0.2   0:12.86 nova-conductor                                                                                                                           
12424 stack     20   0  276372  89148   3608 S   0.0  0.2   0:12.27 nova-conductor                                                                                                                           
12422 stack     20   0  276112  88764   3604 S   0.7  0.2   0:12.51 nova-conductor                                                                                                                           
12425 stack     20   0  275296  87968   3604 S   0.3  0.2   0:12.79 nova-conductor                                                                                                                           
12429 stack     20   0  275232  87776   3600 S   0.3  0.2   0:12.76 nova-conductor                                                                                                                           
12262 stack     20   0  190240  87700   5472 S   0.7  0.2   0:29.42 nova-api                                                                                                                                 
12426 stack     20   0  273992  86596   3608 S   0.0  0.2   0:11.64 nova-conductor                                                                                                                           
12427 stack     20   0  272976  85724   3592 S   0.3  0.2   0:12.00 nova-conductor                                                                                                                           
12289 stack     20   0  190240  83728   1568 S   0.0  0.2   0:00.01 nova-api                                                                                                                                 
12290 stack     20   0  190240  83728   1568 S   0.0  0.2   0:00.00 nova-api                                                                                                                                 
12291 stack     20   0  190240  83728   1568 S   0.0  0.2   0:00.01 nova-api                                                                                                                                 
12292 stack     20   0  190240  83728   1568 S   0.0  0.2   0:00.01 nova-api

** Affects: glance
     Importance: Undecided
         Status: New

-- 
storing multiple heat snapshots causing significant memory consumption
https://bugs.launchpad.net/bugs/1412961
You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance.