monasca team mailing list archive

[<Yuan.Pen@xxxxxxxxxxxxx>]

Hi Craig,
It is indeed that the alarm was dropped because of error to write to kafka. Once we started debug mode in storm, we saw the error. Thanks for patching it up so next time we can see what happened in log file. By the way, the error is "Invalid partition given with record".Because the threshold engine writes alarm to alarm-state-transitions topic with specific partition number [0-7), it will fail if topic is configured with less than 8 partitions - that is the case we have. This may happen to anybody because the threshold engine hard coded the number of alarm-state-transitions topic to which may not be the case when the topic is created by separate process. I am suggesting to change the number of partition in Threshold Engine code to be configurable so that can be kept consistent with number of partitions parameter when creating alarm-state-transitions topic.
Thanks a lot for looking at the matter.
Yuan

From: Bryant, Craig W (HP Cloud Service) [mailto:craig.bryant@xxxxxxx]
Sent: Thursday, January 18, 2018 12:43 PM
To: Pen, Yuan; Hochmuth, Roland M
Cc: bradley.klein@xxxxxxxxxxx; monasca@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Monasca] alarm transition events are missing in kafka queue - mysql alarm state is updated properly

Hi Yuan,
I have not seen an issue like this before and have never had it reported, either. Unfortunately, the code only logs an exception on send to kafka as debug so it won't show up in the standard configuration. I will submit a patch to change that, but you will have to upgrade to get that change. I'm sorry, but I have no suggestions on how to make this more reliable except ensure you kafka is running in a high availability configuration.

Craig Bryant
HPE

From: Monasca <monasca-bounces+craig.bryant=hpe.com@xxxxxxxxxxxxxxxxxxx<mailto:monasca-bounces+craig.bryant=hpe.com@xxxxxxxxxxxxxxxxxxx>> on behalf of "Yuan.Pen@xxxxxxxxxxxxx<mailto:Yuan.Pen@xxxxxxxxxxxxx>" <Yuan.Pen@xxxxxxxxxxxxx<mailto:Yuan.Pen@xxxxxxxxxxxxx>>
Date: Friday, January 12, 2018 at 11:36 AM
To: "Hochmuth, Roland M" <roland.hochmuth@xxxxxxx<mailto:roland.hochmuth@xxxxxxx>>
Cc: "bradley.klein@xxxxxxxxxxx<mailto:bradley.klein@xxxxxxxxxxx>" <bradley.klein@xxxxxxxxxxx<mailto:bradley.klein@xxxxxxxxxxx>>, "monasca@xxxxxxxxxxxxxxxxxxx<mailto:monasca@xxxxxxxxxxxxxxxxxxx>" <monasca@xxxxxxxxxxxxxxxxxxx<mailto:monasca@xxxxxxxxxxxxxxxxxxx>>
Subject: [Monasca] alarm transition events are missing in kafka queue - mysql alarm state is updated properly

Hi Roland,
This is Yuan Pen from Deutsche Telekom. I am sending this email to the monasca community asking for help on monasca threshold engine. We have found that when sometime alarm state transitions happened, the threshold engine updated mysql alarm state properly, but failed to put  state transition events  in kafka queue (alarm-state-transitions).  Does this ring a bell to anyone in the community? If this is a real problem, is there anything we can do to make sure the event in transition queue and state in mysql is synched? Any comments or help are greatly appreciated.
Best Regard,

Yuan Pen

571-594-6155