ubuntu-phone team mailing list archive

[Robert Park <robert.park@xxxxxxxxxxxxx>]

Hi everybody,

Just a quick status update regarding the CI Train Spreadsheet.

This issue has been known to bite us from time to time, we've been
fortunate not to run into this for a while now but since yesterday
it's back with a vengeance.

What's happening is that the spreadsheet is just overloaded, too many
formulas are trying to run and too many people are trying to edit it
at the same time and google's servers just can't keep up with it all,
so it panics and reverts to the last "known-good" state, which
unfortunately is quite old, so we've lost all the rows entered since
Sunday.

Before you panic, please refer to the dashboard[0]. It is
authoritative in the event of discrepancies. If you had a silo
assigned since Sunday and your spreadsheet row is now missing, the
silo will still appear in the dashboard in order for you to trigger
builds. No actual silo packages have been lost, just the raw
spreadsheet rows are missing*. If you need QA for your silo urgently,
you'll have to ping QA directly until the spreadsheet issues resolve
themselves.


I've taken a few desperate measures to try and mitigate the load on the sheet:

1. I deleted ~4,000 rows from the 'Archive' sheet, and moved them into
a new doc[1]. I don't *expect* this to have much impact, because the
Archive sheet is largely dormant (no formulas running on it, not many
people reading it). However 4k rows is a lot so if deleting them
speeds up the spreadsheet, it's worth trying.

2. I disabled the 'Reconfigure' links. The formula that powers those
links is a bit doggish, so hopefully with that disabled for a little
while, the spreadsheet can recover and we can re-enable those later.
Unfortunately for now this means that only us trainguards can
reconfigure silos. I'll investigate an alternate solution for this
that may be more sustainable/scalable.


In the meantime, if you added a row to the spreadsheet and it got lost
*before* we could assign you a silo, that's just gone. Please email me
personally with the details and I'll manually assign a silo for you.

As usual, if you need anything, please ping 'trainguards' in
#ubuntu-ci-eng and we're happy to help!


[0] http://people.canonical.com/~platform/citrain_dashboard/#?distro=ubuntu&q=
[1] https://docs.google.com/a/canonical.com/spreadsheets/d/1ywAY1mBfW1vGtLdO9KCEVC6yW0CDupgIssXRNBRPuZA/edit#gid=679369919


* (just remember, "the map is not the territory", in this case the
spreadsheet is like a map, but the dashboard is the real territory)