← Back to team overview

maria-discuss team mailing list archive

Streaming telemetry and ColumnStore - viable or not

 

Hi all,

I am investigating the viability of using MariaDB as a datastore for
streaming telemetry data sourced via gRPC. The data ingestion rate will be
quite high (write once, read many with no change to the data), but the data
can be buffered and bulk loaded in 10-60sec intervals. We are currently
using MariaDB for OLTP purposes (Innodb) and also graphs (OQGRAPH) with no
issues.

My journey so far has been mainly focused on Kafka but the more I dig the
more future trouble I uncover. So many components required to solve even a
simple problem. Now I am investigating if using our current tooling that
our team knows already will get the job done.

Reading up on ColumnStore has also uncovered more gotchas and deprectations
that I am comfortable with. Things that concern me are:


   1. Is ColumnStore going to go the way of TokuDB?
   2. The docs warn that data needs to be bulk inserted - not super ideal
   for streaming applications. Is streaming via cpimport viable?
   3. The bulk write API's have been deprecated (
   https://mariadb.com/kb/en/columnstore-bulk-write-sdk/) which then
   implies that streaming is not advised (Kafka connectors are thus also not
   viable). This slideshow talks about streaming with ColumnStore 1.1
   https://www.slideshare.net/MariaDB/big-data-analytics-with-mariadb-columnstore-81661174
   4. Streaming data is a combination of fixed columns and key / value
   pairs. How to deal with the dynamic nature of the key / value pairs (see
   below for sample data that was sourced via gRPC and converted to OpenTSDB
   format - we can convert the data into any required format, this is just one
   example).


So some questions:

What is the current recommended streaming telemetry data store solution
from MariaDB, or should I rather look at a custom solution (different
tooling, more skills/re-skilling required)?
Is there a writeup or reference design?

Any insights will be helpful.

Thanks.

put eos.intflagpktcounter.counter.pkt 1611032330 547181
intf=Port-Channel553 direction=in type=Multicast host=30.7.253.145
put eos.intflagcounter.counter.octets 1611032330 121113118
intf=Port-Channel553 direction=in host=30.7.253.145
put eos.intflagpktcounter.counter.pkt 1611032330 3649031 type=Ucast
host=30.7.253.145 intf=Port-Channel553 direction=out
put eos.intflagpktcounter.counter.pkt 1611032330 297600 host=30.7.253.145
intf=Port-Channel553 direction=in type=Ucast
put eos.intflagcounter.counter.errors 1611032330 0 intf=Port-Channel553
direction=out host=30.7.253.145
put eos.totalcpu.cpu.utilization.total 1611032330 21 type=util
host=30.7.253.145
put eos.totalcpu.cpu.utilization.total 1611032330 225275556
host=30.7.253.145 type=user
put eos.corecpu.cpu.utilization.total 1611032330 42929390 type=system
host=30.7.253.145
put eos.corecpu.cpu.utilization.total 1611032330 4005894390 type=idle
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.0 1611032330 55264287 type=user
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.0 1611032330 10865672 host=30.7.253.145
type=system
put eos.corecpu.cpu.utilization.cpu.0 1611032330 1002366784 type=idle
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.1 1611032330 20 type=util
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.1 1611032330 50596732 type=user
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.1 1611032330 10883378 type=system
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.1 1611032330 1006988024 type=idle
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.2 1611032330 21 type=util
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.2 1611032330 58921103 type=user
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.2 1611032330 10405231 type=system
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.2 1611032330 999355549 type=idle
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.3 1611032330 21 type=util
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.3 1611032330 60493433 type=user
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.3 1611032330 10775108 type=system
host=30.7.253.145
put eos.corecpu.cpu.utilization.cpu.3 1611032330 997184032 type=idle
host=30.7.253.145
put eos.tempsensor.environment.temperature 1611032330 45.526831652731964
sensor=cell/1/TempSensor1 host=30.7.253.145
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
fan=Fan2\/1 host=30.7.253.145
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
host=30.7.253.145 fan=Fan2\/1
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
fan=Fan1\/1 host=30.7.253.145
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
fan=Fan1\/1 host=30.7.253.145
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
fan=Fan3\/1 host=30.7.253.145
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
fan=Fan3\/1 host=30.7.253.145
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
fan=Fan4\/1 host=30.7.253.145
put eos.fanspeed.environment.cooling 1611032330 60.57142857142857
fan=Fan4\/1 host=30.7.253.145
put eos.sfpfixedtemperature.xcvr.temperature 1611032330 38.0390625
host=30.7.253.145 intf=Ethernet3
put eos.sfpfixedrxtxpower.xcvr.power 1611032330 -2.0398103065285085
intf=Ethernet3 powerdirection=tx host=30.7.253.145
put eos.sfpfixedrxtxpower.xcvr.power 1611032330 -1.92938760082761
powerdirection=rx host=30.7.253.145 intf=Ethernet3
put eos.sfpfixedrxtxpower.xcvr.power 1611032330 -1.92938760082761
intf=Ethernet3 powerdirection=rx host=30.7.253.145

-- 
Richard Ellerbrock