[Ayatana-dev] autolanding of unity and linked components branches ready and activated!

[Didier Roche <didrocks@xxxxxxxxxx>]

Hey everyone,
since last UDS, we planned to have autolanding of unity and related 
components branch to trunk. This has nice side-effects of enabling us 
doing extra checking, running autotests before the branch is landed, and 
pushing every commits to a ppa for users to get latest crack before it 
lands into precise. This short cycle of feedback will enable everyone to 
spot regressions way easier and create some rapid error/fix loop 
compared than what we had in the past.

Jean-Baptiste helped a lot on the infrastructure side to get a vm 
running and a jenkins slave called "unity-merger" to support this. We 
created as well a launchpad "unity-merger" account bot which has gpg and 
ssh access to the unity-team ppa and trunk. If you see merges and 
messages from a bot callled "unity-merger", don't run away! ;)

So, what happens?
1. From now on, you should not, at any mean, merging yourself a branch 
to trunk. Commit access will be removed if we see abuse there, but we 
try to trust people and that won't be necessary, isn't it? :)
2. For every merge request, once someone is happy with the branch and is 
sure it can be safely merged to trunk, he can change the *global* status 
of the merge request from "Needs Review" to "Approve".
3. A fire-tarmac cron job is run every 15 minutes and will look at every 
projects in parallel (but only one merge per project at a time to avoid 
conflicts) and run a special tarmac branch that I setup (more on that below)
4. Tarmac will basically ensure that the branch is mergeable, try to 
merge it, check some criterias which will be extended in the future, and 
then runs a jenkins job
5. Tarmac will then monitor the generic jenkins job (this one doesn't 
depend on the project which will be run), and from it:
5.0 copy all the branches to other dirs and handling the fact that 
jenkins is runned by another user without spoiling the upstream tree.
5.1 take the proposed branch, apply the packaging, figuring a correct 
versionning for the ppa, run autoreconf/libtoolize to take the latest 
dependencies (if you are using autotools, cmake project and projects 
without any separate packaging branches are also supported). 5.2 Then, 
it will spawn a pbuilder chroot, take all needed dependencies as well as 
a local repository with latest nux, dee, bamf… from previous builds, 
adds some specific test dependencies like google-tests and build from it.
5.3 Finally, it runs a make check on the project to ensure tests are passing
Note that the packaging rules are a little bit relaxed in the pbuilder: 
no need for exact symbol checking and such. However, it needs the 
packaging to be in synced with the upstream branch, but packaging heavy 
changes like soname bumps should only happen once a cycle (and that was 
what happened on the past), there will be no automatically ABI checks 
though.
6. If all of this is successfull, it will then take a commit message 
from the merge description, adds a "By <author>, reviewed by <reviewer>. 
Fixes: <bugs_fixed>.", commit and push to trunk. If a bug is attached to 
the branch, this one is automatically set to "Fix committed"
7. It dput a packaged version in the unity-team daily ppa: 
https://launchpad.net/~unity-team/+archive/ppa for broader audience 
(which will then build on the three architectures: i386, amd64 and 
finally armel). It won't wait on the end of it building though are we 
are already ensured it builds on amd64 through steps 5.
Then… it will go to the next branch of the same project

If jenkins exits in error or that other criterias are not met, the 
branch will be rejected, then, the result of this private jenkins is 
copied automatically on a public jenkins mirror 
(https://jenkins.qa.ubuntu.com) and a comment is added to the merge 
request while reverting the status to "needs review".
Example:
The Jenkins job 
https://jenkins.qa.ubuntu.com/job/unity-merger/37/console reported an 
error when processing this lp:~didrocks/unity-merger-test/foo10 branch.
Not merging it.

If jenkins doesn't respond (let's say it's under maintainenance), the 
branch is not rejected, just ignored for now and no more merge will be 
processed until jenkins is up again.

Limitations
As we are running all projects in parallels to avoid having to wait too 
much before a branch from a project is merged, we need to have in mind 
that there is no dependency automagic check, like:
1. you add a new API to nux
2. you use this API in unity.
1. needs to be merged (and so built) successfully. Please, do not 
approve 2. until 1. status is "merged". Then, the local repository in 
the pbuilder will automatically take the right Nux version with the 
additional API. If you set it to "approved" before the nux branch is 
merged, you can have great changes seeing your branch rejected because 
it will fail to build. Be patient before approving the second branch, as 
the merger is running every 15 minutes, you should get a merge soon. :-)

Future improvments
I will add additional checks soon in the future:
- ensure that every branches have at least a bug attached. There will be 
the possibility to add "NOBUGNEEDED" to bypass it, but we will have a 
report to avoid abuses (and that will enable us to have the bug 
automatically set and components added). No more unclosed bugs! Having 
real metrics of number of bugs closed, and such…
- ensure that every branches have tests (with a test directory of file 
changes). There will be the possibility to add "NOTESTNEEDED", with the 
same report to avoid abuses.
- we can even ensuring that the contributor signed the CLA checking for 
the right team if the team is put up to date again on launchpad.
- we can also imagining that after UIF or FF, if the bug linked to the 
branch authenticated as a UIFe or FFe is not acked by the release or 
documentation team, it is rejected automatically.
- finally, we can get a "ready to release" time, where no approved 
branch are merged automatically, when set in this mode, we can directly 
choose which branch to merge and pick only those that are helping 
getting closer to the release (a freeze-like mode in some way).


For developers, in a nutshell, no more direct merge, think to set the 
"approved" status in the merge request and all should be smoothed.
Of course, as this has been tested on a test project in the infra, we 
are enabling projects one after another.



Changes that were needed
This is more for interested technical people wanting to know what was 
needed to be done

Tarmac branch
I have made a bunch of modification to tarmac so that it can support 
this workflow, those are:
1. a jenkins plugin to communicate with a jenkins instance from tarmac. 
This support monitoring a job, waiting on the result, and launching new 
jobs. Of course, a full test suite is provided.
2. ensure that tarmac can be run by project, but that only one tarmac 
instance can exist at some point (to avoid trying to get two merges at 
the same time)
3. add a new tarmac exception type to not refuse the merge but ignore it 
for now and go to the next branch. This is used for plugin which can 
have transient errors like http connexion (like the jenkins one)
4. enabling merge description to be used as commit message for the branch
I'll push all those fixes and enhancements upstream.

fire-tarmac script
This script is the one piloting tarmac jobs, it basically uses a 
directory and run one tarmac for each project in parallel.
Consequently, adding a new project is basically:
- adding a jenkins job using the above generic script
- creating a directory with the corresponding tarmac.conf file

Jenkins job
There is a new shell script which is project independant for doing all 
what is described in step 5. It also uses some pbuilderrc tweaks and 
pbuilder hooks that are installed on the same vm.


Projects impacted by this workflow:
Every projects I'm directly maintaining: bamf, bamf-qt, dee, dee-qt, 
libunity, libunity-misc, nux, unity, unity-2d, unity-asset-pool, 
unity-lens-applications, unity-lens-files, unity-lens-music.

All projects are not activated on purpose right now, do not hesitate to 
ping me if you have an approved merge request that isn't triggered 
automatically. I prefer to monitor the first run myself as we know we 
will have rough edges in the coming days. :)

Jean-Baptiste is writing a documentation from our notes while setting up 
the infrastructure so that we can share with other projects we want to 
have the same workflow.

Thanks everyone for your cooperation,
Cheers,
Didier