maria-developers team mailing list archive

[Henrik Ingo <henrik.ingo@xxxxxxxxxxxxx>]

So now would probably be a good time to blog something about our
MariaDB release. As agreed, I will blog something about how the
release came to be, Monty's will be more announcement like.

Is the below somewhat close to what really happened?

******************************************************
Producing a MariaDB release: It isn't over until the fat lady sings...

When I was younger and had lots of free time, I used to do video
editing as a hobby. At that time I developed a rule that is true for
many projects in general (it was also true for writing a book some
years later). The rule is: <em>When you think you are 90% done, you
are only 50% done.</em> With video-editing, this meant that when the
video was more or less ready, you are still 50% away from the final
goal of actually having a master copy on tape. The latter 50% would be
spent on checking ending credits, watching through the video a couple
of times, and in those time, rendering even simplest of effects. Using
a Windows PC for video editing was in those times a shaky effort in
itself, so even when mastering you had to sit there and watch through
the whole tape to make sure there were no glitches.

Producing a MariaDB release has been a similar process. In our company
meeting in August we were discussing "final steps" to produce a final
Beta, then Release Candidate, then production release. As I blogged
then, the progress has been documented on a daily basis on <a
href="http://askmonty.org/wiki/index.php/MariaDB_5.1_TODO";>the
askmonty.org wiki</a>.

<h2>Prioritizing features</h2>

We thought the list of features to include into the release was short.
Even so, we eventually pushed about half of them to a future release.
For instance the <a
href="http://datacharmer.blogspot.com/2008/09/mysql-virtual-columns.html";>Virtual
Columns feature was done in itself, but seemed to expose a memory
overrun in other parts of the server. Tracking this was difficult,
because it did not happen in a debug build. Similarly a customer
sponsored an optimization that <a
href="http://askmonty.org/worklog/Server-Sprint/?tid=53";>avoids
unnecesary flushing of the keycache</a> and while this was done in
time, pending final review it missed the train too. Some other
features were pushed into the future just due to lack of time.

We did however get in 2 Percona patches as planned, so MariaDB does
include some great additions that have been out in the MySQL community
for some time now. Virtual Columns will go into the 5.2 branch, and in
Monty's inbox we've found some further performance improvements done
by MySQL users that we'll start looking at too.

<h2>Just one final merge...</h2>

One innocent bullet point on the todo list was to merge in the newest
MySQL version, as well as XtraDB and PBXT. This was not a one day
job...

Somehow we managed to start the merge from MySQL from a more or less
random point in the <a
href="https://launchpad.net/mysql-server/5.1";>MySQL 5.1 Launchpad
tree</a>. This led to a long list of merge failures that needed to be
fixed. Quite soon we realized that we were merging something that was
between MySQL 5.1.37 and 5.1.38, and the missing changesets to 5.1.38
were then added.

I'm told the delta between MySQL and MariaDB is already 11kB, a lot of
it is in whitespace or comments. A good target for cleanup work in our
next release cycle...

The bigger problem was that XtraDB 6 was developed against MySQL
5.1.36, so merging it against a MariaDB with 5.1.38 was not easy.

All in all we spent at least 4 man weeks just fixing merge issues. But
we are learning from our mistakes, and most importantly, <a
href="http://askmonty.org/wiki/index.php/MariaDB::MergingMySQL";>documented
them</a>.

<h2>PBXT</h2>

If I remember correctly, merging the PBXT was relatively painless and
the Primebase team was quick to respond to small issues that came up.
We did notice that it did not compile on PowerPC (Mac OS X), and Paul
McCullagh confirmed that this was not a supported platform for PBXT.

<h2>Testing, testing... and first it has to build.</h2>

It turns out that <a
href="http://dev.mysql.com/doc/refman/5.1/en/news-5-1-38.html";>MySQL
5.1.38 happened to be a release with quite a lot of changes</a>. Most
notably Sun has suddenly included InnoDB plugin in the midst of a
stable release series. (Considering the major benefit to MySQL users,
I actually understand the decision, it just happened to create
additional surprises to us.) 5.1.38 also changed how it builds all
storage engines, so we had to apply the same fix both to XtraDB and
PBXT too.

Of course Buildbot runs did report all kinds of test-suite errors, so
another batch of man weeks was spent fixing various test failures.
Many test failures happened on Windows only. To our surprise they also
happened in the stock MySQL 5.1.38 (which is GA quality). We were
however able to find patches for most of those, until we were down to
a small number of test failures which were considered safe to ignore.
Developers told me that Windows always got less attention in MySQL, so
for instance replication on Windows may never have been that heavily
tested.

<h2>Installation packages</h2>

Personally I had completely underestimated the complexity of actually
building TAR, ZIP, RPM, DEB and Windows installer packages. After all,
developers build and do test runs every day. In any case, the Sun
MySQL team produces builds steadily on a monthly basis. It turns out
the knowledge how to do that doesn't really exist outside of Sun.

Lots of technical MySQL users know how to checkout the latest source
code from launchpad, build it and play with it. Also lots of users
know how to download the source code tar balls of MySQL releases,
build it and play with it. What turned out to be non-trivial is 1) the
step to create the source tar ball, from the repository and 2) even
more the step to create a DEB or RPM from the source tar ball. Maybe
the scripts to do this exist in the MySQL source codes, maybe they
don't, but at least we did not know how to do that.

Thankfully <a href="http://ourdelta.org/";>OurDelta</a> has been doing
this task for the 5.0 series of MySQL. They had so far not produced
5.1 builds, but thanks to close co-operation with Kristian from Monty
Program and Arjen from OurDelta, we were able to remove all problems
so that we know have produced builds for MariaDB 5.1.38.

This situation reminds me of other "open in license only" discussions,
such as seems to be the case for <a
href="http://lwn.net/Articles/331908/";>Google's Android</a>. The
producers of MySQL and Android didn't put lots of effort into actually
making it easy for others to excercise their GPL rights to build their
own versions of these Open Source products. This is in contrast to
such projects like Linux, where every kid knows how to compile their
own kernel, even if they wouldn't know how to actually write a single
line of C.

Anyway, thanks to the experience of OurDelta, this didn't block us for
more than a week or two. For MariaDB, we will of course make sure that
there are both scripts and instructions readily available so that
anyone can produce their own builds if they'd want to.

And we of course welcome feedback and contributions also in this area,
there are some popular Linuxes that OurDelta scripts don't yet cover.

<h2>Caring about Windows users</h2>

I personally believe that a key reason to MySQL's success (for
instance compared to PostgreSQL) was that it also supported Windows
(which PostgreSQL does as of recently). After all, many web developers
- especially in the 90's - would use Windows for their desktop, while
deploying on Linux servers. So they needed both. So even if we won't
initially support all of the MySQL supported platforms, MariaDB
certainly needs to support Windows if it wants to be a MySQL
replacement. <a
href="/blogs/2009/august/mariadb-release-plan-and-other-news-mp-company-meeting">This
topic was widely discussed in my previous MariaDB related blog
post</a>.

With this background in mind, our Windows support started from a
relatively low point: After merging XtraDB, MariaDB crashed on startup
for Windows. It seems that nodoby had ever used XtraDB on Windows
before that. Also in MariaDB, we just hadn't gotten around to run a
Buildbot slave on Windows.

While we did fix most of the Windows issues, one thing we don't have
is a Windows installer. (The one Sun has for MySQL is not Open Source,
it turns out...) So while we are still committed to supporting
Windows, we have now decided that it is not proper to delay releasing
the Linux binaries anymore and we have now decided to release the
Linux binaries as soon as possible, while still continuing to create
the Windows binaries.

Luckily there are people who care also about Windows users, and we are
thankful that <a
href="https://lists.launchpad.net/maria-developers/msg01210.html";>Webyog
has offered to help</a> in creating an Open Source MariaDB installer
for Windows.

<h2>Is there a happy end...</h2>

So this was a long story. (apologies) Does it end well?

It seems so. We now have debs and rpms and they are undergoing final
smoke testing and sanity checking. If there are no surprises, you can
look for a "recommended for public testing" release on <a
href="http://monty-says.blogspot.com";>Monty's blog</a> any day now.


PS: Just in case you're wondering: <a
href="http://www.theanswerbank.co.uk/Phrases-and-Sayings/article/it-isnt-over-until-the-fat-lady-sings/";>http://www.theanswerbank.co.uk/Phrases-and-Sayings/article/it-isnt-over-until-the-fat-lady-sings/</a>

-- 
email: henrik.ingo@xxxxxxxxxxxxx
tel:   +358-40-5697354
www:   www.avoinelama.fi/~hingo
book:  www.openlife.cc