maria-discuss team mailing list archive

[Pak Chan <pakchan9000@xxxxxxxxx>]

Hi Markus,

I can confirm that Galeramon is connecting to the Galera database instances
via unencrypted connections. In fact, Galeramon never loses connection to
the master node, constantly pinging it and querying it for
"wsrep_local_state" and "wsrep_local_index" status.

Having decrypted the packet, the last packet emitted by MaxScale to the
client prior to closing the connection to both the client and the master
node is a MySQL 2003 response. However, I have replicated it for a non-TLS
client connection.

I have tracked it down to a specific circumstance: it occurs when the
master node has a HIGHER 'wsrep_local_index' value than the slave when the
slave is brought back online. It does NOT occur when the master node has a
LOWER  'wsrep_local_index' value than the slave. In that circumstance
MaxScale/Galeramon reports (in /etc/syslog):

maxscale[10675]: [galeramon] There are no cluster members
maxscale[10675]: Server changed state: dbnode1[172.100.1.22:3306]:
lost_master. [Master, Synced, Running] -> [Running]
maxscale[10675]: [galeramon] Found cluster members
maxscale[10675]: Server changed state: dbnode1[172.100.1.22:3306]:
new_master. [Running] -> [Master, Synced, Running]
maxscale[10675]: Server changed state: dbnode2[172.100.1.23:3306]:
slave_up. [Down] -> [Slave, Synced, Running]


Is that a known issue, or should I report it as a new issue?


On 12 Oct 2017 11:08 p.m., "Pak Chan" <pakchan9000@xxxxxxxxx> wrote:

> Thanks for the response. I won't be able to get back to it until Monday,
> so I'll see about decrypting the packet then and report the bug with the
> extra information.
>
> On 12 Oct 2017 22:49, "Markus Mäkelä" <markus.makela@xxxxxxxxxxx> wrote:
>
>> Hi,
>>
>> Thank you for testing MaxScale and reporting your findings. We haven't
>> seen the behavior you described and it is certainly not expected. One
>> possibility I can think of is that for some reason, a TLS connection is
>> created between the monitor and the Galera when a normal one should be
>> used. Unless the servers are configured to use TLS, the monitor should
>> create normal, plain connections.
>>
>> I think you can decrypt the packets with Wireshark as long as you have
>> the private key and the certificate. This might be a helpful link in
>> determining what the encrypted packet was: https://wiki.wireshark.org/SSL
>>
>> As this is not expected behavior, I would like to ask you to file a bug
>> report on the MariaDB Jira under the MaxScale project. Hopefully we'll get
>> to the bottom of this soon.
>>
>> Markus
>>
>> On 12/10/17 19:39, Pak Chan wrote:
>>
>> Hi,
>>
>> I'm (still) testing MaxScale, set up to connect only to the master node,
>> on Ubuntu 16.04 fronting a Galera cluster. The cluster comprises of 2
>> MariaDB 10.1 instances on different servers, with a Galera Arbitrator
>> instance running on the MaxScale server. I'm testing this using the MariaDB
>> client (mysql) from a fourth machine.
>>
>> My test scenario is to see what the client experiences if I stop and
>> restart a MariaDB node part-way through a transaction. I start with the
>> "slave" node, to give me a baseline for comparison, before doing it with
>> the master node. However, the baseline case gives me different results,
>> depending on whether the MySQL client is connecting via TLS or not.
>>
>> If I connect to MaxScale via TLS, the connection disappears when the
>> "slave" node comes back (but not when it goes down). Watching it via a
>> network packet trace, I can see that, just after the slave comes back up,
>> MaxScale sends the client an encrypted packet to the mysql client, then
>> sends a MySQL Quit command to the master node before disconnecting. The
>> syslog contains the line "[galeramon] There are no cluster members".
>>
>> If I connect to MaxScale without TLS, the connection remains stable
>> regardless of the number of times the slave node goes down and up, and the
>> "galeramon" line doesn't appear in the syslog. (I discovered this when I
>> disabled the TLS in order to see what the encrypted packet being sent to
>> the client was...I still don't know what it is!)
>>
>> Has anyone else come across this behaviour?
>>
>> MaxScale is configured as follows (the commented-out configuration is
>> uncommented when connecting via TLS):
>>
>> [dbnode1]
>> type=server
>> address=172.16.1.22
>> port=3306
>> protocol=MySQLBackend
>>
>> priority=1
>>
>>
>> [dbnode2]
>> type=server
>> address=172.16.1.23
>> port=3306
>> protocol=MySQLBackend
>>
>> priority=2
>>
>>
>> [Galera Monitor]
>> type=monitor
>> module=galeramon
>> servers=dbnode1,dbnode2
>> user=galeramon
>> passwd=galeramon
>> monitor_interval=1000
>> available_when_donor=true
>> use_priority=true
>>
>> [Galera Service]
>> type=service
>> router=readrouteconn
>>
>> router_options=master
>>
>> servers=dbnode1,dbnode2
>> user=galeramon
>> passwd=galeramon
>>
>> [MaxAdmin Service]
>> type=service
>> router=cli
>>
>> [Galera Listener]
>> type=listener
>> service=Galera Service
>> protocol=MySQLClient
>> port=3306
>> #ssl=required
>> #ssl_version=TLSv12
>> #ssl_cert=/etc/mysql/ssl/server-cert.pem
>> #ssl_key=/etc/mysql/ssl/server-key.pem
>> #ssl_ca_cert=/etc/mysql/ssl/ca-cert.pem
>> #ssl_cert_verify_depth=1
>>
>> [MaxAdmin Listener]
>> type=listener
>> service=MaxAdmin Service
>> protocol=maxscaled
>> socket=default
>>
>>
>> PC
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~maria-discuss
>> Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~maria-discuss
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>> --
>> Markus Mäkelä, Software Engineer
>> MariaDB Corporation
>> t: +358 40 7740484 <+358%2040%207740484> | Skype: markus.j.makela
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~maria-discuss
>> Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~maria-discuss
>> More help   : https://help.launchpad.net/ListHelp
>>
>>