ubuntu-phone team mailing list archive

[Timo Jyrinki <timo.jyrinki@xxxxxxxxxxxxx>]

Thanks for the link, Łukasz! Indeed the errors seem to have
disappeared so far. I've restored all missing landings from the
dashboards at the end of the spreadsheet with "description lost". The
link to the dashboard however works for all of them so status is
updated, the silo can be marked as tested (to move to QA's queue
automatically), etc. But you may want to re-fill the missing
information anyhow, especially if you want a reconfigure or such.

-Timo

On Wed, Mar 18, 2015 at 11:19 AM, Łukasz 'sil2100' Zemczak
<lukasz.zemczak@xxxxxxxxxxxxx> wrote:
> Hey Robert!
>
> It seems to have somewhat helped, probably the Reconfigure links took
> some load off from the spreadsheet. I noticed a decrease in the number
> of error reports from the spreadsheet itself, so it should be once again
> safe to use (for now). Thanks for that! Once I feel better I'll also try
> to add some optimizations to the old cow.
>
> Also, please remember that I'm doing backups of the spreadsheet every 15
> minutes (but only keeping history from around 24 hours). So if your
> landing disappeared, there's a chance it'll be somewhere in the backups.
> Backups are saved as csv files of the Pending sheet:
>
> http://sil2100.vexillium.org/citrain-backups/
>
> Thanks!
>
> W dniu 18.03.2015 o 01:58, Robert Park pisze:
>> Hi everybody,
>>
>> Just a quick status update regarding the CI Train Spreadsheet.
>>
>> This issue has been known to bite us from time to time, we've been
>> fortunate not to run into this for a while now but since yesterday
>> it's back with a vengeance.
>>
>> What's happening is that the spreadsheet is just overloaded, too many
>> formulas are trying to run and too many people are trying to edit it
>> at the same time and google's servers just can't keep up with it all,
>> so it panics and reverts to the last "known-good" state, which
>> unfortunately is quite old, so we've lost all the rows entered since
>> Sunday.
>>
>> Before you panic, please refer to the dashboard[0]. It is
>> authoritative in the event of discrepancies. If you had a silo
>> assigned since Sunday and your spreadsheet row is now missing, the
>> silo will still appear in the dashboard in order for you to trigger
>> builds. No actual silo packages have been lost, just the raw
>> spreadsheet rows are missing*. If you need QA for your silo urgently,
>> you'll have to ping QA directly until the spreadsheet issues resolve
>> themselves.
>>
>>
>> I've taken a few desperate measures to try and mitigate the load on the sheet:
>>
>> 1. I deleted ~4,000 rows from the 'Archive' sheet, and moved them into
>> a new doc[1]. I don't *expect* this to have much impact, because the
>> Archive sheet is largely dormant (no formulas running on it, not many
>> people reading it). However 4k rows is a lot so if deleting them
>> speeds up the spreadsheet, it's worth trying.
>>
>> 2. I disabled the 'Reconfigure' links. The formula that powers those
>> links is a bit doggish, so hopefully with that disabled for a little
>> while, the spreadsheet can recover and we can re-enable those later.
>> Unfortunately for now this means that only us trainguards can
>> reconfigure silos. I'll investigate an alternate solution for this
>> that may be more sustainable/scalable.
>>
>>
>> In the meantime, if you added a row to the spreadsheet and it got lost
>> *before* we could assign you a silo, that's just gone. Please email me
>> personally with the details and I'll manually assign a silo for you.
>>
>> As usual, if you need anything, please ping 'trainguards' in
>> #ubuntu-ci-eng and we're happy to help!
>>
>>
>> [0] http://people.canonical.com/~platform/citrain_dashboard/#?distro=ubuntu&q=
>> [1] https://docs.google.com/a/canonical.com/spreadsheets/d/1ywAY1mBfW1vGtLdO9KCEVC6yW0CDupgIssXRNBRPuZA/edit#gid=679369919
>>
>>
>> * (just remember, "the map is not the territory", in this case the
>> spreadsheet is like a map, but the dashboard is the real territory)
>>
>
> Best regards,
>
> --
> Łukasz 'sil2100' Zemczak
>  lukasz.zemczak@xxxxxxxxxxxxx
>  www.canonical.com
>
> --
> Mailing list: https://launchpad.net/~ubuntu-phone
> Post to     : ubuntu-phone@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~ubuntu-phone
> More help   : https://help.launchpad.net/ListHelp