ubuntu-phone team mailing list archive

[Łukasz 'sil2100' Zemczak <lukasz.zemczak@xxxxxxxxxxxxx>]

Hey Robert!

It seems to have somewhat helped, probably the Reconfigure links took
some load off from the spreadsheet. I noticed a decrease in the number
of error reports from the spreadsheet itself, so it should be once again
safe to use (for now). Thanks for that! Once I feel better I'll also try
to add some optimizations to the old cow.

Also, please remember that I'm doing backups of the spreadsheet every 15
minutes (but only keeping history from around 24 hours). So if your
landing disappeared, there's a chance it'll be somewhere in the backups.
Backups are saved as csv files of the Pending sheet:

http://sil2100.vexillium.org/citrain-backups/

Thanks!

W dniu 18.03.2015 o 01:58, Robert Park pisze:
> Hi everybody,
> 
> Just a quick status update regarding the CI Train Spreadsheet.
> 
> This issue has been known to bite us from time to time, we've been
> fortunate not to run into this for a while now but since yesterday
> it's back with a vengeance.
> 
> What's happening is that the spreadsheet is just overloaded, too many
> formulas are trying to run and too many people are trying to edit it
> at the same time and google's servers just can't keep up with it all,
> so it panics and reverts to the last "known-good" state, which
> unfortunately is quite old, so we've lost all the rows entered since
> Sunday.
> 
> Before you panic, please refer to the dashboard[0]. It is
> authoritative in the event of discrepancies. If you had a silo
> assigned since Sunday and your spreadsheet row is now missing, the
> silo will still appear in the dashboard in order for you to trigger
> builds. No actual silo packages have been lost, just the raw
> spreadsheet rows are missing*. If you need QA for your silo urgently,
> you'll have to ping QA directly until the spreadsheet issues resolve
> themselves.
> 
> 
> I've taken a few desperate measures to try and mitigate the load on the sheet:
> 
> 1. I deleted ~4,000 rows from the 'Archive' sheet, and moved them into
> a new doc[1]. I don't *expect* this to have much impact, because the
> Archive sheet is largely dormant (no formulas running on it, not many
> people reading it). However 4k rows is a lot so if deleting them
> speeds up the spreadsheet, it's worth trying.
> 
> 2. I disabled the 'Reconfigure' links. The formula that powers those
> links is a bit doggish, so hopefully with that disabled for a little
> while, the spreadsheet can recover and we can re-enable those later.
> Unfortunately for now this means that only us trainguards can
> reconfigure silos. I'll investigate an alternate solution for this
> that may be more sustainable/scalable.
> 
> 
> In the meantime, if you added a row to the spreadsheet and it got lost
> *before* we could assign you a silo, that's just gone. Please email me
> personally with the details and I'll manually assign a silo for you.
> 
> As usual, if you need anything, please ping 'trainguards' in
> #ubuntu-ci-eng and we're happy to help!
> 
> 
> [0] http://people.canonical.com/~platform/citrain_dashboard/#?distro=ubuntu&q=
> [1] https://docs.google.com/a/canonical.com/spreadsheets/d/1ywAY1mBfW1vGtLdO9KCEVC6yW0CDupgIssXRNBRPuZA/edit#gid=679369919
> 
> 
> * (just remember, "the map is not the territory", in this case the
> spreadsheet is like a map, but the dashboard is the real territory)
> 

Best regards,

-- 
Łukasz 'sil2100' Zemczak
 lukasz.zemczak@xxxxxxxxxxxxx
 www.canonical.com