ubuntu-phone team mailing list archive

[Olivier Tilloy <olivier.tilloy@xxxxxxxxxxxxx>]

On Tue, Sep 22, 2015 at 5:38 AM, Robert Park <robert.park@xxxxxxxxxxxxx> wrote:
> Hello!
>
> So today has been an exciting and terrifying day for the train, as I've
> attempted to fix some issues discovered during my vacation, and created more
> problems, and then worked to fix those, too.
>
> First, the bad news: I've made a horrible, horrible mistake that resulted in
> at least 10 silos being deleted. I've done my best to restore & rebuild them
> but as usual I'm only human. Everybody should check over their silos, and if
> possible do a quick smoketest to make sure nothing has exploded. Most of the
> silos I rebuilt succeeded but in particular there are these 4 failures:
>
> timo-jyrinki wellark charles dobey, ubuntu/landing-026: Build failed:
> pay-service failed to build:
> https://requests.ci-train.ubuntu.com/#/ticket/284
>
> oSoMoN, ubuntu/landing-035: Build failed: webbrowser-app failed to build:
> https://requests.ci-train.ubuntu.com/#/ticket/386

This horrible, horrible mistake is unfortunate indeed. Builds in silo
35 (and other webbrowser-app silos like silo 17) started failing
randomly yesterday. What fails is a unit test that is being run at
build time, which had previously been rock-solid. It fails only on
i386 and amd64, never on armhf, and only randomly (but the failure has
been observed both on vivid and wily). The output of unit tests in
package builds is too sparse to be useful, and I haven’t had time to
take a closer look at the issue yet. Mirv suggested this might be
https://launchpadlibrarian.net/218473113/qtbase-opensource-src_5.4.1%2Bdfsg-2ubuntu8_5.4.1%2Bdfsg-2ubuntu9.diff.gz,
I’ll be verifying this later today.


> mandel, ubuntu/landing-030: Build failed: ubuntu-download-manager failed to
> build. unity-scope-click failed to build:
> https://requests.ci-train.ubuntu.com/#/ticket/394
>
> bzoltan, ubuntu/landing-038: Build failed: ubuntu-ui-toolkit failed to
> build: https://requests.ci-train.ubuntu.com/#/ticket/369
>
> Since all of the failures are PPA build failures, this means most likely
> they were broken before I accidentally deleted them and you're already aware
> of these failures and are already working on them anyway, but still best to
> double-check that I didn't make things worse.
>
> There is also a possibility that I've resurrected extra silos by mistake, so
> everybody should check their requests to make sure that things you thought
> were landed/abandoned aren't back from the dead.
>
> Here is the full list of rebuilds I performed:
> http://paste.ubuntu.com/12518903/
>
> Sorry for the inconvenience! Now with that out of the way, there is some
> good news:
>
> * The prepare-silo job now can no longer fail/explode when somebody else's
> request is malformed, so silo assignments should be bulletproof at this
> point. The only failure mode should be if we're out of silos, if you see
> anything else please inform me. Actually, inform me if we're out of silos as
> well and I'll try to free some up ;-)
>
> * When assigning silos, instead of always getting the lowest-numbered
> available one, you will get a random one. This isn't perfect, but can be
> helpful in the situation where you've uploaded an incorrectly high version
> number to a PPA and want to downgrade your version number in a rebuild. PPAs
> don't allow this, so in order to do this you need to free your silo then
> reassign. Previously since assignment always gave you the lexically-first
> silo, you'd find yourself freeing & reassigning and ending up back in the
> same silo. Now there's at least a chance you'll get a different silo.
>
> * There is a new 'abandon' job that is more along the lines of what people
> have been expecting. If you want to free a silo, you no longer run the
> "Merge & Clean" job with ONLY_FREE_SILO, there's now a separate "Abandon"
> link in Bileto. And this works whether or not the request has a silo
> assigned or not. So if you're done with a request, just click Abandon then
> Build on the jenkins form, if you had a silo it will be released, and then
> the status will unconditionally be set to 'Abandoned' which hides the
> request.
>
>
> Any questions or concerns please let me know, that's all for now! The
> iterations continue!!
>
> --
> Mailing list: https://launchpad.net/~ubuntu-phone
> Post to     : ubuntu-phone@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~ubuntu-phone
> More help   : https://help.launchpad.net/ListHelp
>