maria-developers team mailing list archive

[Kristian Nielsen <knielsen@xxxxxxxxxxxxxxx>]

I have been doing a bit of preparation/planning for the upcoming merge of
MySQL 5.6.

I think the 5.6 merge (and 5.5 merge) is very important for MariaDB. There is
much stuff in there that we really need to make available to our users (and
probably some better left out). If we fail to keep the merges going, it will
leave the community in a very sorry situation; I do not see anyone else
stepping up to this task.

My idea is that we could merge MySQL 5.6 piece-by-piece. Several other
developers mentioned this idea at the Athens meeting:

 - It makes it much easier to selectively merge or not merge stuff. For
   example there are probably large chunks of optimiser changes that we do not
   want to merge (or maybe merge only the testcases for). On the other hand we
   probably want all InnoDB stuff.

 - I am hopeful it makes the merge much more managable when done in smaller
   pieces. Large reorganisations of code (header refactoring, copyright
   changes) should be easier when not intermixed with other changes. Merges
   from MySQL 5.5 -> MySQL 5.6 are probably also easier to deal with when done
   in isolation (I assume we can get conflicts for these, though we already
   have the equivalent changes in MariaDB 5.5).

 - Easier to share the task between developers. One developer can do a single
   piece in a few days, then pass the baton when he needs to switch to other
   tasks (or just gets tired). Complex merges of large feature pushes can be
   null-merged to allow main merge to progress, and in parallel another
   developer can work on reworking the patch into the MariaDB code. For some
   changes I think we will even be able to do the actual merge in parallel.

 - Easier to plan and understand how far the merge is progressing. Eg. if we
   merge M weeks worth of 5.6 pushes every N days, that allows to estimate
   when we will catch up with Oracle development

 - Easier debugging of test failures; by keeping the trees green between
   pieces merged, new test failures will be known to be caused by relatively
   smaller sets of changes, simplifying debugging.

I have done some work to facilitate this kind of piecewise merge. As it turns
out, it is not trivial to determine suitable pieces to merge. Oracle
developers use no discipline for the direction they do merges in, so the
sequence of pushes into lp:mysql-server ("trunk") is not immediately
apparent. So I took the time to reconstruct the list of pushes from reading
through the history; this was not overly hard to do.

I next wrote a script that scans the history based on this list of pushes, and
further annotates each push with some useful information:

 - Size of diff for the patch

 - Number of changesets merged (if a merge)

 - Mark if this is a merge from MySQL 5.5

 - One-line short comment for the push, either hand-crafted for selected
   items, or else extracted from the commit message.

 - Revision ID (including developer info)

It should be fairly simple to scan this list from the bottom, collecting
chunks of related or manageable pushes and merging them as one piece. Eg. A
sequence of small pushes could be merged as one, or a sequence of pushes by
one developer, or a single large feature push.

Here is an example extract of such an annotated list:

     0    2*     tor.didriksen@xxxxxxxxxx-20111101065343-ha3d79h1ayk9j7x9
             [5.5] Null merge from 5.1/5.5
    13           andrei.elkin@xxxxxxxxxx-20111031135232-q13pfsaieo8ujekj
             Minor test adjustment
   791           alexander.nozdrin@xxxxxxxxxx-20111031115220-01la8vj7ssmcm0mc
             Bug#61392
     0    2*     tor.didriksen@xxxxxxxxxx-20111031091354-8p6s03s1zxerrp8p
             [5.5] Null merge Bug#12406055 from 5.1/5.5
  5380   15      jorgen.loland@xxxxxxxxxx-20111031081122-dnufl9r4li0i3ni6
             [opt] Larger merge from optimiser tree
   134    2*     andrei.elkin@xxxxxxxxxx-20111028212219-cdcuh37i70k9o4ms
             [5.5] Merge Bug#56299 (?) from 5.5
   451    2      andrei.elkin@xxxxxxxxxx-20111028130851-mz4p2mg28y14wb7k
             merge from trunk to local tree.
    43           sergey.glukhov@xxxxxxxxxx-20111028094528-enre87osw9u4qjy7
             [opt] Bug#32254 post-merge fix (optimiser)
    48           tor.didriksen@xxxxxxxxxx-20111028092548-a3gjubgmwuhdunfj
             Bug#11747423 post-merge fix
   114           tor.didriksen@xxxxxxxxxx-20111028091835-xqavfrxkc50gbsxq
             "Default value for sort_buffer_size changed from 2M to 256K"

The first column is diff size, the second is number of changesets merged (or
empty for non-merge changesets). A "*" marks pushes that are only merges from
MySQL 5.5.

I pushed the script to lp:mariadb-tools, in /mysql-5.6-merge-stuff/. It also
includes the full output in 5.6-annotated.txt (it takes 15-20 minutes for bzr
to chew through the history on a fast multi-core machine).

I hope this will be useful for the merge. Comments/suggestions/questions very
welcome!

 - Kristian.