launchpad-dev team mailing list archive

[Graham Binns <graham@xxxxxxxxxxxxx>]

On 9 March 2010 21:16, Gavin Panella <gavin.panella@xxxxxxxxxxxxx> wrote:
> On 9 March 2010 18:33, Jonathan Lange <jml@xxxxxxxxxxxxx> wrote:
> ...
>>> Generally bug imports run quite quickly, on the order of a few minutes
>>> for several hundred bugs. Is that too long? The bug importer also
>>> creates almost all new data, rather than updates, so won't be holding
>>> locks that are going to affect other people, I assume.
>>
>> You say "generally". How would this work in the exceptional cases
>> where it takes much more than a few minutes?
>
> Good question. Erm... a few possible answers:
>
>  * I think there's scope to improve the bug import script performance
>   a lot, perhaps enough that only massive imports would cause
>   concern.
>
>  * Depending on stub's response, it may not be an issue at all.
>
>  * If it's okay to keep a transaction open for X minutes, then we
>   could have a policy of committing every X minutes, between bugs. If
>   there have been no errors, commit, else rollback. If the broken
>   bugs are distributed evenly (which they won't be, natch) then this
>   will approximate an all-or-nothing solution.

It occurs to me that we don't need to be able to keep the transaction
open for X minutes on production, just on staging. I.e., we don't want
to be doing dry runs in the production environment anyway, do we?
(Well, I suppose it would be nice to check that it's going to work,
but read on...)

Instead we could use a DBLoopTuner to handle the batching. That gives
us a fairly robust import process (i.e. one bad bug might mess up the
batch it's in but the logging should be good enough to allow us to
track that down). Users wanting bugs imported generally don't give a
monkeys whether the ids of the imported bugs are contiguous, just that
all their data are in Launchpad.

>   For a dry run, we could take a similar approach, but rollback every
>   X minutes. It may also be fine to rollback between every bug yet
>   still get an accurate picture of how a non-dry run will behave.



-- 
Graham Binns | PGP Key: EC66FA7D