← Back to team overview

ubuntu-phone team mailing list archive

CI Train rollout issues update

 

Hi everybody,

First I just want to apologize, this rollout has had a number of
issues that I failed to anticipate, and I'm working on solutions.
Here's the status so far:

* The thing about values being double-urlencoded when running jenkins
jobs was quite the surprise to me, it didn't behave that way in
staging before the rollout, but now it does behave that way in staging
and I didn't change anything. Something must have changed in SSO to
break this.

I can't not urlencode the values when submitting from bileto to
jenkins, as URLs become corrupted this way and things break. Instead
I've programmed jenkins to de-urlencode the values. So you should
expect to continue seeing double-urlencoded values in the jenkins
form, but this no longer causes issues, the values are decoded and
interpreted successfully.


* There are intermittent HTTP 500 errors coming from the bileto API,
however the frontend web page does not report this in any way, so you
may experience some intermittent symptoms of the page not updating, or
trying to save a change and it mysteriously fails.

The logs indicate there is some kind of disk IO issue causing this,
I've notified #webops and they're investigating. It seems to be
intermittent so if you're having problems just try again, maybe a
couple times, and it should get through eventually.


* The spreadsheet soft-transition didn't work out as well as I'd
thought, there's lots of missing pieces here. I'll be working today to
migrate spreadsheet requests into bileto, and if all goes well, by the
end of my shift today we can begin just being 100% bileto and get rid
of the spreadsheet completely.


Apologies for the inconvenience!