← Back to team overview

randgen team mailing list archive

Re: Issue with MAX_ROWS_THRESHOLD, KILL QUERY and transformers



Thanks for the excellent analysis. I agree that there is a window of opportunity between the time the RQG detects a KILL is needed and the time it is actually issued.

So, there are two approaches I guess:

- replace the mechanism for killing memory and time-consuming queries with something else. I do think however that such a mechanism is needed. Previously, it was very easy to get the RQG to consume massive ammounts of memory, especially on unattended test runs.

- make Transforms gracefully handle the "unknown prepared statement" error and keep the current mechanism.

Let me ponder on it for a day or two.

----- Original Message ----- From: "John Embretsen" <johnemb@xxxxxxxxx>
To: <philip@xxxxxxxxx>
Cc: <randgen@xxxxxxxxxxxxxxxxxxx>
Sent: Tuesday, January 10, 2012 12:01 PM
Subject: Issue with MAX_ROWS_THRESHOLD, KILL QUERY and transformers

Hi Philip,

Some time ago (November 2011) you added MAX_ROWS_THRESHOLD to the MySQL
executor in the RQG. With that change, the executor would send "KILL
query" whenever a result set contained more than MAX_ROWS_THRESHOLD rows.

"* Terminate queries that have returned more than 10K rows via an
off-band KILL QUERY command"

After some subsequent changes to the Transformer validator's handling of
various statuses, specifically:

"make handling of interrupted queries more robust"


"properly handle STATUS_WONT_HANDLE from Transforms"

I have started seeing some weird behavior. The issue is that a
combination of the transformers "Distinct" and "ExecuteAsPreparedTwice"
results in STATUS_ENVIRONMENT_FAILURE. This happens because of the
following order of events, it seems:

1) Distinct transformer removes DISTINCT from query; transformed query
returns more than MAX_ROWS_THRESHOLD rows.
2) Executor detects the threshold being hit and sends
KILL QUERY to the server.
3) Executor/transformer continues with next transform:
PREPARE query; EXECUTE prep_stmt; EXECUTE prep_stmt;
4) "PREPARE query" is killed because of the KILL QUERY command
issued earlier.
5) "EXECUTE prep_stmt" fails because there is no prepared
statement, as the PREPARE was killed.
6) Transformer returns STATUS_ENVIRONMENT_FAILURE due to getting
"1243 Unknown prepared statement handler (%s) given to EXECUTE."

It seems to me that the "KILL QUERY" issued by the executor does not
work as intended. It kills the next query (or statement, to be precise)
instead of the original query.

I guess this is because the original query has already finished when the
executor counts the number of rows, and the next query is executed with
the same connection_id as the first one, thus matching the KILL QUERY
command sent earlier? Note that this is all with --threads=1.

I have attached some log files where you can see this issue (master.log
excerpt and part of RQG output). You can reproduce it using the
following grammar contents (referred to below as envFail.yy):

    SELECT DISTINCT   alias1 . `col_int_key` AS field1 FROM ( C AS
alias1 , ( C AS alias2 , D AS alias3 ) )    ORDER BY field1, field1  ;

and the following RQG command:

perl runall.pl \
  --queries=1 \
  --seed=1 \
  --threads=1 \
  --duration=60 \
  --transformer=Distinct,ExecuteAsPreparedTwice \
  --grammar=envFail.yy \
  --basedir=$CODE_OPT \

where $CODE is an environment variable pointing to a valid MySQL

Any comments / suggestion for fix?