maria-developers team mailing list archive

[worklog-noreply@xxxxxxxxxxxx]

-----------------------------------------------------------------------
                              WORKLOG TASK
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
TASK...........: Add Sphinx storage engine to MariaDB
CREATION DATE..: Mon, 10 Aug 2009, 23:57
SUPERVISOR.....: Monty
IMPLEMENTOR....: Knielsen
COPIES TO......: 
CATEGORY.......: Server-Sprint
TASK ID........: 42 (http://askmonty.org/worklog/?tid=42)
VERSION........: Server-5.2
STATUS.........: Assigned
PRIORITY.......: 60
WORKED HOURS...: 9
ESTIMATE.......: 7 (hours remain)
ORIG. ESTIMATE.: 16

PROGRESS NOTES:

-=-=(Knielsen - Mon, 07 Jun 2010, 07:12)=-=-
Help Andrew with the integration.

Worked 2 hours and estimate 7 hours remain (original estimate unchanged).

-=-=(Knielsen - Mon, 31 May 2010, 06:49)=-=-
Wrote patch that allows to test SphinxSE in mysql-test-run, using external Sphinx daemon.

Worked 7 hours and estimate 9 hours remain (original estimate unchanged).

-=-=(Knielsen - Fri, 28 May 2010, 07:49)=-=-
High-Level Specification modified.
--- /tmp/wklog.42.old.4369      2010-05-28 07:49:13.000000000 +0000
+++ /tmp/wklog.42.new.4369      2010-05-28 07:49:13.000000000 +0000
@@ -49,6 +49,10 @@
 might be possible to pre-generate the necessary data/index files and store
 them in the source tree.
 
+I pushed a proof-of-concept patch for this here:
+
+    lp:~knielsen/maria/5.2-sphinxse
+
 Here is a sample test case using this:
 
 --source include/have_sphinx.inc

-=-=(Knielsen - Fri, 28 May 2010, 06:31)=-=-
High-Level Specification modified.
--- /tmp/wklog.42.old.32746     2010-05-28 06:31:24.000000000 +0000
+++ /tmp/wklog.42.new.32746     2010-05-28 06:31:24.000000000 +0000
@@ -1 +1,63 @@
+Code
+----
+
+Andrew Aksyonoff from Sphinx is helping to integrate the SphinxSE plugin into
+the MariaDB tree.
+
+It is a plugin, so it can be added to the tree just by including the
+sub-directory storage/sphinx/.
+
+The Sphinx plugin is already of some maturity, having been used with MySQL for
+some time.
+
+
+Testing
+-------
+
+To get testing in the mysql-test-run framework, some extensions are needed.
+
+To use the Sphinx storage engine, the external Sphinx search daemon needs to
+be running with some data directory containing indexed data. It also needs to
+be allocated a port.
+
+This is the indended approach:
+
+1. Testing will use an external Sphinx setup installed on the machine. Sphinx
+binaries will be searched in typical locations (eg. /usr/bin, /usr/local/bin),
+or can be specified explicitly in the environment with SPHINXSEARCH_INDEXER
+and SPHINXSEARCH_SEARCHD for the two required binaries. If the external Sphinx
+binaries can not be found, then Sphinx tests will be disabled (using some
+--source include/have_sphinx.inc in the test cases).
+
+2. The mysql-test-run framework will install Sphinx search data and start/stop
+the Sphinx search daemon for the test cases, similarly how it is done for the
+other servers mysqld, ndbd, etc. We will run the Sphinx search daemon with
+options --console, --config, and --pidfile.
+
+3. The mysql-test-run framework will generate a Sphinx config file from a
+template in mysql-test/suite/sphinx/my.cnf. This config file will allocate
+ports and data directories appropriate for avoiding conflicts between multiple
+simultaneous mysql-test-run executions. The Sphinx config file is sufficiently
+similar to MySQL my.cnf that we can use the existing framework for generating
+config file, with just a slightly modified variant of the code writing the
+file to disk.
+
+4. The mysql-test-run framework will pre-load the mysql database with tables
+and data for Sphinx to index. It will then run the `indexer` program to
+generate the indexes, and then start the `searchd` daemon. These three steps
+must be done in order, as each step depends on the previous. ALTERNATIVE: it
+might be possible to pre-generate the necessary data/index files and store
+them in the source tree.
+
+Here is a sample test case using this:
+
+--source include/have_sphinx.inc
+--source include/have_sphinxse.inc
+
+--replace_result $SPHINXSEARCH_PORT SPHINXSEARCH_PORT
+eval create table ts ( id int unsigned not null, w int not null, q varchar(255)
+not null, index(q) ) engine=sphinx
+connection="sphinx://127.0.0.1:$SPHINXSEARCH_PORT/*";
+select * from ts where q='test';
+drop table ts;
 

-=-=(Knielsen - Fri, 28 May 2010, 06:07)=-=-
Version updated.
--- /tmp/wklog.42.old.32184     2010-05-28 06:07:00.000000000 +0000
+++ /tmp/wklog.42.new.32184     2010-05-28 06:07:00.000000000 +0000
@@ -1 +1 @@
-9.x
+Server-5.2

-=-=(Knielsen - Fri, 28 May 2010, 06:06)=-=-
Category updated.
--- /tmp/wklog.42.old.32171     2010-05-28 06:06:23.000000000 +0000
+++ /tmp/wklog.42.new.32171     2010-05-28 06:06:23.000000000 +0000
@@ -1 +1 @@
-Maria-BackLog
+Server-Sprint

-=-=(Knielsen - Fri, 28 May 2010, 06:06)=-=-
Version updated.
--- /tmp/wklog.42.old.32171     2010-05-28 06:06:23.000000000 +0000
+++ /tmp/wklog.42.new.32171     2010-05-28 06:06:23.000000000 +0000
@@ -1 +1 @@
-Maria-2.0
+9.x

-=-=(Knielsen - Fri, 28 May 2010, 06:06)=-=-
Status updated.
--- /tmp/wklog.42.old.32171     2010-05-28 06:06:23.000000000 +0000
+++ /tmp/wklog.42.new.32171     2010-05-28 06:06:23.000000000 +0000
@@ -1 +1 @@
-Un-Assigned
+Assigned

-=-=(Guest - Tue, 15 Sep 2009, 02:25)=-=-
no

Reported zero hours worked. Estimate unchanged.

-=-=(Guest - Tue, 15 Sep 2009, 02:24)=-=-
Version updated.
--- /tmp/wklog.42.old.13241     2009-09-15 02:24:07.000000000 +0300
+++ /tmp/wklog.42.new.13241     2009-09-15 02:24:07.000000000 +0300
@@ -1 +1 @@
-Connector/.NET-5.1
+Maria-2.0



DESCRIPTION:

Add the Sphinx storage engine to the MariaDB tree


HIGH-LEVEL SPECIFICATION:



Code
----

Andrew Aksyonoff from Sphinx is helping to integrate the SphinxSE plugin into
the MariaDB tree.

It is a plugin, so it can be added to the tree just by including the
sub-directory storage/sphinx/.

The Sphinx plugin is already of some maturity, having been used with MySQL for
some time.


Testing
-------

To get testing in the mysql-test-run framework, some extensions are needed.

To use the Sphinx storage engine, the external Sphinx search daemon needs to
be running with some data directory containing indexed data. It also needs to
be allocated a port.

This is the indended approach:

1. Testing will use an external Sphinx setup installed on the machine. Sphinx
binaries will be searched in typical locations (eg. /usr/bin, /usr/local/bin),
or can be specified explicitly in the environment with SPHINXSEARCH_INDEXER
and SPHINXSEARCH_SEARCHD for the two required binaries. If the external Sphinx
binaries can not be found, then Sphinx tests will be disabled (using some
--source include/have_sphinx.inc in the test cases).

2. The mysql-test-run framework will install Sphinx search data and start/stop
the Sphinx search daemon for the test cases, similarly how it is done for the
other servers mysqld, ndbd, etc. We will run the Sphinx search daemon with
options --console, --config, and --pidfile.

3. The mysql-test-run framework will generate a Sphinx config file from a
template in mysql-test/suite/sphinx/my.cnf. This config file will allocate
ports and data directories appropriate for avoiding conflicts between multiple
simultaneous mysql-test-run executions. The Sphinx config file is sufficiently
similar to MySQL my.cnf that we can use the existing framework for generating
config file, with just a slightly modified variant of the code writing the
file to disk.

4. The mysql-test-run framework will pre-load the mysql database with tables
and data for Sphinx to index. It will then run the `indexer` program to
generate the indexes, and then start the `searchd` daemon. These three steps
must be done in order, as each step depends on the previous. ALTERNATIVE: it
might be possible to pre-generate the necessary data/index files and store
them in the source tree.

I pushed a proof-of-concept patch for this here:

    lp:~knielsen/maria/5.2-sphinxse

Here is a sample test case using this:

--source include/have_sphinx.inc
--source include/have_sphinxse.inc

--replace_result $SPHINXSEARCH_PORT SPHINXSEARCH_PORT
eval create table ts ( id int unsigned not null, w int not null, q varchar(255)
not null, index(q) ) engine=sphinx
connection="sphinx://127.0.0.1:$SPHINXSEARCH_PORT/*";
select * from ts where q='test';
drop table ts;


ESTIMATED WORK TIME

ESTIMATED COMPLETION DATE
-----------------------------------------------------------------------
WorkLog (v3.5.9)