maria-discuss team mailing list archive

[Cédric Counotte <cedric.counotte@xxxxxxxxxx>]

I’m already in pain 😉

The config sent is a little old and I have since removed the dups, however those were the results of trying to fiddle with some of the settings in hope for improvements.

No idea what is PMM ?

Do you have a bug tracking this behavior, so that maybe I can follow it or apply work-around ?

I’m not quite keen on downgrading (which I’ll do using the spare servers I have and a little replication in between), but if it turns out to provide a more reliable alternative I’m eager to do it then!


De : Gordan Bobic <gordan.bobic@xxxxxxxxx>
Envoyé : mercredi 27 juillet 2022 13:27
À : Cédric Counotte <cedric.counotte@xxxxxxxxxx>
Cc : William Edwards <wedwards@xxxxxxxxxxxxxx>; Mailing-List mariadb <maria-discuss@xxxxxxxxxxxxxxxxxxx>
Objet : Re: [Maria-discuss] MariaDB server horribly slow on start

There is no supported downgrade path other than logical dump+restore.
There are also no packages built for distros where the major version is older than what ships with the distro.

Since your queries seem to end up stuck in commit stage, it could be related to redo log flushing, which behaves very erratically on 10.5+. If it leaves the log to fill up to 90% and the state transfer hits, it could be that with the checkpoint age already high, there just isn't enough headroom to avoid a massive stall. Purely guessing here without any telemetry.

So it seems you are, most unfortunately, in for an awful lot of pain.

You seem to have multiple instances of some settings in your config, that is never a good sign.

Do you have PMM set up on this cluster and fully instrumented?


On Wed, 27 Jul 2022, 14:04 Cédric Counotte, <cedric.counotte@xxxxxxxxxx<mailto:cedric.counotte@xxxxxxxxxx>> wrote:
I started with 10.4.x back in January, exact same issue.

Now on Ubuntu 22.04 it comes with MariaDB 10.6 and I was forced to upgrade!

I’ll test downgrading somehow, but on mariadb downloads I can’t find 22.04 version?!
Download MariaDB Server - MariaDB.org<https://mariadb.org/download/?t=repo-config&d=21.10+%22impish%22&v=10.5&r_m=mva>


De : Gordan Bobic <gordan.bobic@xxxxxxxxx<mailto:gordan.bobic@xxxxxxxxx>>
Envoyé : mercredi 27 juillet 2022 13:02
À : Cédric Counotte <cedric.counotte@xxxxxxxxxx<mailto:cedric.counotte@xxxxxxxxxx>>
Cc : William Edwards <wedwards@xxxxxxxxxxxxxx<mailto:wedwards@xxxxxxxxxxxxxx>>; maria-discuss@xxxxxxxxxxxxxxxxxxx<mailto:maria-discuss@xxxxxxxxxxxxxxxxxxx>
Objet : Re: [Maria-discuss] MariaDB server horribly slow on start

IMO your biggest cause of problems is chasing the bleeding edge.
You are on 10.6.x. I generally don't even deploy 10.5.x in production yet except in very exceptional circumstances (simple async replication, no expectation of extreme performance).
Migrate to 10.4.x and stick with it until it goes EOL. Then maybe think about upgrading to 10.5.x in the 6 months following 10.4.x EOL.

On Wed, Jul 27, 2022 at 1:50 PM Cédric Counotte <cedric.counotte@xxxxxxxxxx<mailto:cedric.counotte@xxxxxxxxxx>> wrote:
I must add that the slowness occurs when the SST or IST are completed, and the server becomes available. Donor is also entirely blocked (it seems) when the queue is being processed.

While the SST is performed, there is no issue.

De : William Edwards <wedwards@xxxxxxxxxxxxxx<mailto:wedwards@xxxxxxxxxxxxxx>>
Envoyé : mercredi 27 juillet 2022 11:58
À : Cédric Counotte <cedric.counotte@xxxxxxxxxx<mailto:cedric.counotte@xxxxxxxxxx>>
Cc : maria-discuss@xxxxxxxxxxxxxxxxxxx<mailto:maria-discuss@xxxxxxxxxxxxxxxxxxx>
Objet : Re: [Maria-discuss] MariaDB server horribly slow on start


Op 27 jul. 2022 om 11:46 heeft Cédric Counotte <cedric.counotte@xxxxxxxxxx<mailto:cedric.counotte@xxxxxxxxxx>> het volgende geschreven:


Hello all. I hope I’m at the right place to ask this question.

I opened a bug here: https://jira.mariadb.org/browse/MDEV-28969, however I was told to use this mailing list.



We have 4 MariaDB servers in a Galera Cluster and it happens that a server has to be restarted (be it for a crash which I have to open a bug for) or maintenance.



When that happens, the restarted server is causing huge slow down on the whole cluster, and it lasts for 10 to 30 minutes at the very least!



And by huge, I mean huge, we end up with 500 to 800 pending queries on all servers as you can see on attached screenshots

I’ve attached the configuration of any server for reference in case this is the source of the issue.



Any way to solve this would be greatly appreciated.

You seem to be focusing on effect. What is the cause? SST?


Regards,

3C.
[cid:image001.png@01D8A1B9.6DA3C260]
_______________________________________________
Mailing list: https://launchpad.net/~maria-discuss
Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx<mailto:maria-discuss@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~maria-discuss
Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx<mailto:maria-discuss@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp