yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95591
[Bug 2103868] [NEW] slave_connection is never used
Public bug reported:
Context
=======
Neutron 2024.2
oslo.db==16.0.0
neutron-lib==3.15.0
Neutron, like any other OpenStack service rely on oslo.db for SQL statements.
oslo.db provides two parameters to connect to the DB:
connection
slave_connection
Slave connection is supposed to be used for read requests.
Olso.db enginefacade provides a way to have both reader and writter context.
Neutron use that, e.g. taken from neutron-lib:
CONTEXT_READER = get_context_manager().reader
CONTEXT_WRITER = get_context_manager().writer
In my test env, I created two mariadb users:
root
neutronro
I configured the connection info like this:
[database]
connection = mysql+pymysql://root:pass@mysql-neutron/neutron
slave_connection = mysql+pymysql://neutronro:pass@mysql-neutron/neutron
I am following the connection to the db with:
MariaDB [neutron]> show processlist;
Issue
=====
Despite the fact that neutron does use the CONTEXT_READER on a lot of code that do not need any writer context, the slave_connection is never used, thus the connection is always perform to the same server:
MariaDB [neutron]> show processlist;
+-----+------+------------------+---------+---------+------+----------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+-----+------+------------------+---------+---------+------+----------+------------------+----------+
| 598 | root | localhost | neutron | Query | 0 | starting | show processlist | 0.000 |
| 876 | root | 10.42.0.61:41336 | neutron | Sleep | 8 | | NULL | 0.000 |
| 877 | root | 10.42.0.61:41338 | neutron | Sleep | 0 | | NULL | 0.000 |
| 878 | root | 10.42.0.61:41348 | neutron | Sleep | 0 | | NULL | 0.000 |
| 879 | root | 10.42.0.61:44508 | neutron | Sleep | 0 | | NULL | 0.000 |
| 880 | root | 10.42.0.61:44522 | neutron | Sleep | 0 | | NULL | 0.000 |
| 881 | root | 10.42.0.61:44524 | neutron | Sleep | 0 | | NULL | 0.000 |
| 882 | root | 10.42.0.61:44534 | neutron | Sleep | 0 | | NULL | 0.000 |
| 883 | root | 10.42.0.61:44542 | neutron | Sleep | 0 | | NULL | 0.000 |
| 884 | root | 10.42.0.61:44548 | neutron | Sleep | 0 | | NULL | 0.000 |
| 885 | root | 10.42.0.61:44560 | neutron | Sleep | 0 | | NULL | 0.000 |
| 886 | root | 10.42.0.61:44562 | neutron | Sleep | 0 | | NULL | 0.000 |
| 887 | root | 10.42.0.61:44574 | neutron | Sleep | 0 | | NULL | 0.000 |
| 888 | root | 10.42.0.61:44582 | neutron | Sleep | 0 | | NULL | 0.000 |
| 889 | root | 10.42.0.61:44594 | neutron | Sleep | 0 | | NULL | 0.000 |
| 890 | root | 10.42.0.61:44606 | neutron | Sleep | 0 | | NULL | 0.000 |
| 891 | root | 10.42.0.61:44622 | neutron | Sleep | 0 | | NULL | 0.000 |
| 892 | root | 10.42.0.61:44632 | neutron | Sleep | 0 | | NULL | 0.000 |
| 893 | root | 10.42.0.61:44640 | neutron | Sleep | 0 | | NULL | 0.000 |
| 894 | root | 10.42.0.61:44642 | neutron | Sleep | 0 | | NULL | 0.000 |
| 895 | root | 10.42.0.61:44646 | neutron | Sleep | 0 | | NULL | 0.000 |
| 896 | root | 10.42.0.61:44656 | neutron | Sleep | 0 | | NULL | 0.000 |
| 897 | root | 10.42.0.61:44664 | neutron | Sleep | 0 | | NULL | 0.000 |
| 898 | root | 10.42.0.61:44668 | neutron | Sleep | 16 | | NULL | 0.000 |
| 899 | root | 10.42.0.61:44682 | neutron | Sleep | 0 | | NULL | 0.000 |
| 900 | root | 10.42.0.61:44684 | neutron | Sleep | 0 | | NULL | 0.000 |
+-----+------+------------------+---------+---------+------+----------+------------------+----------+
Expected behavior
=================
To reduce the load on the writer db node, an operator may want to redirect all read requests to a second synchronous node (like in a galera cluster).
What I would expect is:
MariaDB [neutron]> show processlist;
+-----+-----------+------------------+---------+---------+------+----------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+-----+-----------+------------------+---------+---------+------+----------+------------------+----------+
| 598 | root | localhost | neutron | Query | 0 | starting | show processlist | 0.000 |
| 822 | neutronro | 10.42.0.61:48804 | neutron | Sleep | 25 | | NULL | 0.000 |
| 823 | neutronro | 10.42.0.61:48818 | neutron | Sleep | 25 | | NULL | 0.000 |
| 824 | neutronro | 10.42.0.61:48834 | neutron | Sleep | 20 | | NULL | 0.000 |
| 825 | root | 10.42.0.61:48844 | neutron | Sleep | 3 | | NULL | 0.000 |
| 826 | root | 10.42.0.61:49036 | neutron | Sleep | 2 | | NULL | 0.000 |
| 827 | neutronro | 10.42.0.61:49052 | neutron | Sleep | 242 | | NULL | 0.000 |
| 828 | root | 10.42.0.61:49068 | neutron | Sleep | 1 | | NULL | 0.000 |
| 829 | neutronro | 10.42.0.61:49076 | neutron | Sleep | 242 | | NULL | 0.000 |
| 830 | root | 10.42.0.61:49084 | neutron | Sleep | 7 | | NULL | 0.000 |
| 831 | root | 10.42.0.61:49088 | neutron | Sleep | 7 | | NULL | 0.000 |
| 832 | neutronro | 10.42.0.61:49090 | neutron | Sleep | 3 | | NULL | 0.000 |
| 833 | neutronro | 10.42.0.61:49100 | neutron | Sleep | 254 | | NULL | 0.000 |
| 834 | neutronro | 10.42.0.61:49114 | neutron | Sleep | 242 | | NULL | 0.000 |
| 835 | root | 10.42.0.61:38028 | neutron | Sleep | 42 | | NULL | 0.000 |
+-----+-----------+------------------+---------+---------+------+----------+------------------+----------+
** Affects: neutron
Importance: Undecided
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2103868
Title:
slave_connection is never used
Status in neutron:
In Progress
Bug description:
Context
=======
Neutron 2024.2
oslo.db==16.0.0
neutron-lib==3.15.0
Neutron, like any other OpenStack service rely on oslo.db for SQL statements.
oslo.db provides two parameters to connect to the DB:
connection
slave_connection
Slave connection is supposed to be used for read requests.
Olso.db enginefacade provides a way to have both reader and writter context.
Neutron use that, e.g. taken from neutron-lib:
CONTEXT_READER = get_context_manager().reader
CONTEXT_WRITER = get_context_manager().writer
In my test env, I created two mariadb users:
root
neutronro
I configured the connection info like this:
[database]
connection = mysql+pymysql://root:pass@mysql-neutron/neutron
slave_connection = mysql+pymysql://neutronro:pass@mysql-neutron/neutron
I am following the connection to the db with:
MariaDB [neutron]> show processlist;
Issue
=====
Despite the fact that neutron does use the CONTEXT_READER on a lot of code that do not need any writer context, the slave_connection is never used, thus the connection is always perform to the same server:
MariaDB [neutron]> show processlist;
+-----+------+------------------+---------+---------+------+----------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+-----+------+------------------+---------+---------+------+----------+------------------+----------+
| 598 | root | localhost | neutron | Query | 0 | starting | show processlist | 0.000 |
| 876 | root | 10.42.0.61:41336 | neutron | Sleep | 8 | | NULL | 0.000 |
| 877 | root | 10.42.0.61:41338 | neutron | Sleep | 0 | | NULL | 0.000 |
| 878 | root | 10.42.0.61:41348 | neutron | Sleep | 0 | | NULL | 0.000 |
| 879 | root | 10.42.0.61:44508 | neutron | Sleep | 0 | | NULL | 0.000 |
| 880 | root | 10.42.0.61:44522 | neutron | Sleep | 0 | | NULL | 0.000 |
| 881 | root | 10.42.0.61:44524 | neutron | Sleep | 0 | | NULL | 0.000 |
| 882 | root | 10.42.0.61:44534 | neutron | Sleep | 0 | | NULL | 0.000 |
| 883 | root | 10.42.0.61:44542 | neutron | Sleep | 0 | | NULL | 0.000 |
| 884 | root | 10.42.0.61:44548 | neutron | Sleep | 0 | | NULL | 0.000 |
| 885 | root | 10.42.0.61:44560 | neutron | Sleep | 0 | | NULL | 0.000 |
| 886 | root | 10.42.0.61:44562 | neutron | Sleep | 0 | | NULL | 0.000 |
| 887 | root | 10.42.0.61:44574 | neutron | Sleep | 0 | | NULL | 0.000 |
| 888 | root | 10.42.0.61:44582 | neutron | Sleep | 0 | | NULL | 0.000 |
| 889 | root | 10.42.0.61:44594 | neutron | Sleep | 0 | | NULL | 0.000 |
| 890 | root | 10.42.0.61:44606 | neutron | Sleep | 0 | | NULL | 0.000 |
| 891 | root | 10.42.0.61:44622 | neutron | Sleep | 0 | | NULL | 0.000 |
| 892 | root | 10.42.0.61:44632 | neutron | Sleep | 0 | | NULL | 0.000 |
| 893 | root | 10.42.0.61:44640 | neutron | Sleep | 0 | | NULL | 0.000 |
| 894 | root | 10.42.0.61:44642 | neutron | Sleep | 0 | | NULL | 0.000 |
| 895 | root | 10.42.0.61:44646 | neutron | Sleep | 0 | | NULL | 0.000 |
| 896 | root | 10.42.0.61:44656 | neutron | Sleep | 0 | | NULL | 0.000 |
| 897 | root | 10.42.0.61:44664 | neutron | Sleep | 0 | | NULL | 0.000 |
| 898 | root | 10.42.0.61:44668 | neutron | Sleep | 16 | | NULL | 0.000 |
| 899 | root | 10.42.0.61:44682 | neutron | Sleep | 0 | | NULL | 0.000 |
| 900 | root | 10.42.0.61:44684 | neutron | Sleep | 0 | | NULL | 0.000 |
+-----+------+------------------+---------+---------+------+----------+------------------+----------+
Expected behavior
=================
To reduce the load on the writer db node, an operator may want to redirect all read requests to a second synchronous node (like in a galera cluster).
What I would expect is:
MariaDB [neutron]> show processlist;
+-----+-----------+------------------+---------+---------+------+----------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+-----+-----------+------------------+---------+---------+------+----------+------------------+----------+
| 598 | root | localhost | neutron | Query | 0 | starting | show processlist | 0.000 |
| 822 | neutronro | 10.42.0.61:48804 | neutron | Sleep | 25 | | NULL | 0.000 |
| 823 | neutronro | 10.42.0.61:48818 | neutron | Sleep | 25 | | NULL | 0.000 |
| 824 | neutronro | 10.42.0.61:48834 | neutron | Sleep | 20 | | NULL | 0.000 |
| 825 | root | 10.42.0.61:48844 | neutron | Sleep | 3 | | NULL | 0.000 |
| 826 | root | 10.42.0.61:49036 | neutron | Sleep | 2 | | NULL | 0.000 |
| 827 | neutronro | 10.42.0.61:49052 | neutron | Sleep | 242 | | NULL | 0.000 |
| 828 | root | 10.42.0.61:49068 | neutron | Sleep | 1 | | NULL | 0.000 |
| 829 | neutronro | 10.42.0.61:49076 | neutron | Sleep | 242 | | NULL | 0.000 |
| 830 | root | 10.42.0.61:49084 | neutron | Sleep | 7 | | NULL | 0.000 |
| 831 | root | 10.42.0.61:49088 | neutron | Sleep | 7 | | NULL | 0.000 |
| 832 | neutronro | 10.42.0.61:49090 | neutron | Sleep | 3 | | NULL | 0.000 |
| 833 | neutronro | 10.42.0.61:49100 | neutron | Sleep | 254 | | NULL | 0.000 |
| 834 | neutronro | 10.42.0.61:49114 | neutron | Sleep | 242 | | NULL | 0.000 |
| 835 | root | 10.42.0.61:38028 | neutron | Sleep | 42 | | NULL | 0.000 |
+-----+-----------+------------------+---------+---------+------+----------+------------------+----------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2103868/+subscriptions
