maria-discuss team mailing list archive

[Cloud Foundry Services <cf-services-eng@xxxxxxxxxx>]

Apologies for the cross-post; we submitted this question to the Codership
forum a week ago and haven't received a response.
https://groups.google.com/forum/#!topic/codership-team/OSnQ1FLloBI

Hello,

Hoping to get some perspective on an issue we saw yesterday. While scaling
up from a single node cluster, we inserted into a table on node 0 and
selected the row count for that table on all subsequent nodes (scripts
below). For each node that was brought online, we observed that connections
to node 0 were suspended during SST, and that there was an additional IST.
At this point the nodes reported that they were synced. However, sometimes
replication did not occur correctly - the additional nodes reported fewer
rows than node 0. Each new node was added by itself, and all other nodes
restarted during each scale-up. Nodes that became out-of-sync continued to
have an incorrect row count even after restart, and sometimes got further
out of sync (i.e. the discrepancy between the number of rows in node 0 and
the other nodes increased)

We were conducting this experiment to see whether* innodb_disallow_writes *was
functioning correctly, but aren't sure whether this issue is related. Has
anyone seen similar behavior while scaling up?

Insert:

#!/bin/bash
mysql -u$db_user -p$db_password -e"create database if not exists $db_name;"
mysql -u$db_user -p$db_password -e"use $db_name; create table if not exists
$table_name (val int);"


for i in `seq $start_val $end_val`; do
  echo "inserting $i"
  mysql -u$db_user -p$db_password -e"use $db_name; insert into $table_name
VALUES ($i); select count(*) from $table_name;"
  sleep 0.1
done

Select:

 #!/bin/bash


for i in `seq $start_val $end_val`; do
  echo "statement $i"
  mysql -u$db_user -p$db_password -e"use replication_test; select count(*)
from vals;"
  sleep 0.1
done

Thank you,
Shannon

Cloud Foundry Services, Pivotal