yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1615502] [NEW] LBAAS - housekeeping serive does not cleanup stale amphora VMs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: chandrasekaran natarajan <chandrasekaran.natarajan@xxxxxx>
Date: Mon, 22 Aug 2016 06:49:56 -0000
Reply-to: Bug 1615502 <1615502@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

1.	Initially there were no spare VMs since the “spare_amphora_pool_size = 0”
 .

 [house_keeping]
# Pool size for the spare pool
spare_amphora_pool_size = 0


stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                      |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.5                    |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.6                    |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+

2.      Change the spare pool size to 1 and restart Octavia-housekeeping
service. Spare Amphora VM gets created as below.


stack@hlm:~/scratch/ansible/next/hos/ansible$  nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                      |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b | amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec | ACTIVE | -          | Running     | OCTAVIA-MGMT-NET=100.74.25.13 |
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.5                    |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.6                    |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+

3.      Now change the spare pool size to 0 and restart Octavia-
housekeeping service. Spare Amphora VM does not get deleted.

stack@hlm:~/scratch/ansible/next/hos/ansible$  nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                      |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b | amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec | ACTIVE | -          | Running     | OCTAVIA-MGMT-NET=100.74.25.13 |
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.5                    |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.6                    |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
stack@hlm:~/scratch/ansible/next/hos/ansible$

4.      I see the requirement is not satisfied as per the config so
hosue keeping should take care of cleaning up the spare amphora VM.But
in the logs I see the statement “ requirement is satisfied “.

In the logs

2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating the cleanup of old amphora... db_cleanup /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating spare amphora check... spare_amphora_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count : 0 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count : 1 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current spare amphora count satisfies the requirement spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60 2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating the cleanup of old amphora... db_cleanup /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating spare amphora check... spare_amphora_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count : 0 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count : 1 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current spare amphora count satisfies the requirement spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1615502

Title:
  LBAAS - housekeeping serive does not cleanup stale amphora VMs

Status in neutron:
  New

Bug description:
  1.	Initially there were no spare VMs since the “spare_amphora_pool_size = 0”
   .

   [house_keeping]
  # Pool size for the spare pool
  spare_amphora_pool_size = 0

  
  stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
  WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
  | ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                      |
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
  | 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.5                    |
  | 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.6                    |
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+

  2.      Change the spare pool size to 1 and restart Octavia-
  housekeeping service. Spare Amphora VM gets created as below.

  
  stack@hlm:~/scratch/ansible/next/hos/ansible$  nova list --all
  WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
  | ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                      |
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
  | 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b | amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec | ACTIVE | -          | Running     | OCTAVIA-MGMT-NET=100.74.25.13 |
  | 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.5                    |
  | 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.6                    |
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+

  3.      Now change the spare pool size to 0 and restart Octavia-
  housekeeping service. Spare Amphora VM does not get deleted.

  stack@hlm:~/scratch/ansible/next/hos/ansible$  nova list --all
  WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
  | ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                      |
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
  | 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b | amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec | ACTIVE | -          | Running     | OCTAVIA-MGMT-NET=100.74.25.13 |
  | 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.5                    |
  | 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2                                          | d15f2abc106740499a453260ae6522f3 | ACTIVE | -          | Running     | n1=4.5.6.6                    |
  +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
  stack@hlm:~/scratch/ansible/next/hos/ansible$

  4.      I see the requirement is not satisfied as per the config so
  hosue keeping should take care of cleaning up the spare amphora VM.But
  in the logs I see the statement “ requirement is satisfied “.

  In the logs

  2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating the cleanup of old amphora... db_cleanup /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
  2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating spare amphora check... spare_amphora_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
  2016-08-22 06:37:57.203 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count : 0 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
  2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count : 1 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
  2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current spare amphora count satisfies the requirement spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60 2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating the cleanup of old amphora... db_cleanup /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
  2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating spare amphora check... spare_amphora_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
  2016-08-22 06:37:57.203 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count : 0 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
  2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count : 1 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
  2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current spare amphora count satisfies the requirement spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1615502/+subscriptions
Follow ups

[Bug 1615502] Re: LBAAS - housekeeping serive does not cleanup stale amphora VMs
From: Gregory Thiemonge, 2023-03-31
[Bug 1615502] Re: LBAAS - housekeeping serive does not cleanup stale amphora VMs
From: Elena Ezhova, 2016-08-24