← Back to team overview

unity-dev team mailing list archive

[Ayatana-dev] autolanding of unity and linked components branches ready and activated!

 

Hey everyone,
since last UDS, we planned to have autolanding of unity and related components branch to trunk. This has nice side-effects of enabling us doing extra checking, running autotests before the branch is landed, and pushing every commits to a ppa for users to get latest crack before it lands into precise. This short cycle of feedback will enable everyone to spot regressions way easier and create some rapid error/fix loop compared than what we had in the past.

Jean-Baptiste helped a lot on the infrastructure side to get a vm running and a jenkins slave called "unity-merger" to support this. We created as well a launchpad "unity-merger" account bot which has gpg and ssh access to the unity-team ppa and trunk. If you see merges and messages from a bot callled "unity-merger", don't run away! ;)

So, what happens?
1. From now on, you should not, at any mean, merging yourself a branch to trunk. Commit access will be removed if we see abuse there, but we try to trust people and that won't be necessary, isn't it? :) 2. For every merge request, once someone is happy with the branch and is sure it can be safely merged to trunk, he can change the *global* status of the merge request from "Needs Review" to "Approve". 3. A fire-tarmac cron job is run every 15 minutes and will look at every projects in parallel (but only one merge per project at a time to avoid conflicts) and run a special tarmac branch that I setup (more on that below) 4. Tarmac will basically ensure that the branch is mergeable, try to merge it, check some criterias which will be extended in the future, and then runs a jenkins job 5. Tarmac will then monitor the generic jenkins job (this one doesn't depend on the project which will be run), and from it: 5.0 copy all the branches to other dirs and handling the fact that jenkins is runned by another user without spoiling the upstream tree. 5.1 take the proposed branch, apply the packaging, figuring a correct versionning for the ppa, run autoreconf/libtoolize to take the latest dependencies (if you are using autotools, cmake project and projects without any separate packaging branches are also supported). 5.2 Then, it will spawn a pbuilder chroot, take all needed dependencies as well as a local repository with latest nux, dee, bamf… from previous builds, adds some specific test dependencies like google-tests and build from it.
5.3 Finally, it runs a make check on the project to ensure tests are passing
Note that the packaging rules are a little bit relaxed in the pbuilder: no need for exact symbol checking and such. However, it needs the packaging to be in synced with the upstream branch, but packaging heavy changes like soname bumps should only happen once a cycle (and that was what happened on the past), there will be no automatically ABI checks though. 6. If all of this is successfull, it will then take a commit message from the merge description, adds a "By <author>, reviewed by <reviewer>. Fixes: <bugs_fixed>.", commit and push to trunk. If a bug is attached to the branch, this one is automatically set to "Fix committed" 7. It dput a packaged version in the unity-team daily ppa: https://launchpad.net/~unity-team/+archive/ppa for broader audience (which will then build on the three architectures: i386, amd64 and finally armel). It won't wait on the end of it building though are we are already ensured it builds on amd64 through steps 5.
Then… it will go to the next branch of the same project

If jenkins exits in error or that other criterias are not met, the branch will be rejected, then, the result of this private jenkins is copied automatically on a public jenkins mirror (https://jenkins.qa.ubuntu.com) and a comment is added to the merge request while reverting the status to "needs review".
Example:
The Jenkins job https://jenkins.qa.ubuntu.com/job/unity-merger/37/console reported an error when processing this lp:~didrocks/unity-merger-test/foo10 branch.
Not merging it.

If jenkins doesn't respond (let's say it's under maintainenance), the branch is not rejected, just ignored for now and no more merge will be processed until jenkins is up again.

Limitations
As we are running all projects in parallels to avoid having to wait too much before a branch from a project is merged, we need to have in mind that there is no dependency automagic check, like:
1. you add a new API to nux
2. you use this API in unity.
1. needs to be merged (and so built) successfully. Please, do not approve 2. until 1. status is "merged". Then, the local repository in the pbuilder will automatically take the right Nux version with the additional API. If you set it to "approved" before the nux branch is merged, you can have great changes seeing your branch rejected because it will fail to build. Be patient before approving the second branch, as the merger is running every 15 minutes, you should get a merge soon. :-)

Future improvments
I will add additional checks soon in the future:
- ensure that every branches have at least a bug attached. There will be the possibility to add "NOBUGNEEDED" to bypass it, but we will have a report to avoid abuses (and that will enable us to have the bug automatically set and components added). No more unclosed bugs! Having real metrics of number of bugs closed, and such… - ensure that every branches have tests (with a test directory of file changes). There will be the possibility to add "NOTESTNEEDED", with the same report to avoid abuses. - we can even ensuring that the contributor signed the CLA checking for the right team if the team is put up to date again on launchpad. - we can also imagining that after UIF or FF, if the bug linked to the branch authenticated as a UIFe or FFe is not acked by the release or documentation team, it is rejected automatically. - finally, we can get a "ready to release" time, where no approved branch are merged automatically, when set in this mode, we can directly choose which branch to merge and pick only those that are helping getting closer to the release (a freeze-like mode in some way).


For developers, in a nutshell, no more direct merge, think to set the "approved" status in the merge request and all should be smoothed. Of course, as this has been tested on a test project in the infra, we are enabling projects one after another.



Changes that were needed
This is more for interested technical people wanting to know what was needed to be done

Tarmac branch
I have made a bunch of modification to tarmac so that it can support this workflow, those are: 1. a jenkins plugin to communicate with a jenkins instance from tarmac. This support monitoring a job, waiting on the result, and launching new jobs. Of course, a full test suite is provided. 2. ensure that tarmac can be run by project, but that only one tarmac instance can exist at some point (to avoid trying to get two merges at the same time) 3. add a new tarmac exception type to not refuse the merge but ignore it for now and go to the next branch. This is used for plugin which can have transient errors like http connexion (like the jenkins one)
4. enabling merge description to be used as commit message for the branch
I'll push all those fixes and enhancements upstream.

fire-tarmac script
This script is the one piloting tarmac jobs, it basically uses a directory and run one tarmac for each project in parallel.
Consequently, adding a new project is basically:
- adding a jenkins job using the above generic script
- creating a directory with the corresponding tarmac.conf file

Jenkins job
There is a new shell script which is project independant for doing all what is described in step 5. It also uses some pbuilderrc tweaks and pbuilder hooks that are installed on the same vm.


Projects impacted by this workflow:
Every projects I'm directly maintaining: bamf, bamf-qt, dee, dee-qt, libunity, libunity-misc, nux, unity, unity-2d, unity-asset-pool, unity-lens-applications, unity-lens-files, unity-lens-music.

All projects are not activated on purpose right now, do not hesitate to ping me if you have an approved merge request that isn't triggered automatically. I prefer to monitor the first run myself as we know we will have rough edges in the coming days. :)

Jean-Baptiste is writing a documentation from our notes while setting up the infrastructure so that we can share with other projects we want to have the same workflow.

Thanks everyone for your cooperation,
Cheers,
Didier



Follow ups