launchpad-dev team mailing list archive

[Gavin Panella <gavin.panella@xxxxxxxxxxxxx>]

I've had a few ideas swimming around in my head for a while to make
bug importing a little/lot less painful. They're probably similar to a
lot of the ideas that anyone who's had to do a bug import has had, but
here they are written down:

 * Allow imports via the web UI or API.

   For example, upload bug data to the librarian, job comes along,
   processes it, and sends email to the uploader and the project
   owner. This would be quite easy to do, now that we have experience
   with the job system (and that we have a job system).

   This is definitely not innovative, but with the two features below
   it becomes a complete solution, imo.

 * Commit only at the end.

   With this we can do dry runs, and not have to faff with cache
   files.

   (Currently the bug importer commits after each bug, and records
   those bugs that imported correctly in a cache file. This is then
   handed to subsequent bug import runs so that it doesn't attempt to
   import the same bug twice. Fiddly and fragile are two ways to
   describe it.)

   Additionally, a failure in an import can point to a broader data
   issue, also present in the bugs that did get imported but were not
   corrupted enough to fail. They cannot now be reverted, so we must
   wait for a refresh in staging, or figure out a manual fix in
   production.

   Only committing at the end would be an important part of allowing
   self-service imports to Launchpad users. As in, users could do
   trial imports for ever and a day, but can't commit. They would be
   responsible for refining the data, and a LOSA or LP dev would only
   be needed for support along the way, a final sanity check and a
   trigger-pull. In staging, this restriction could be relaxed.

 * More up-front validation.

   The bug importer currently works with anything that has the correct
   tags in there somewhere. We could add another layer of validation
   to the input. Not to be difficult, but to catch misunderstandings,
   to give good feedback when a field is formatted incorrectly or when
   there are extra tags that we don't recognise, all with guidance on
   how to make it better.

   Having a good reference validator also makes it easier for people
   to test any bug export code they come up with.

Even before addressing the first point and allowing anyone at all to
do/try LOSA-less imports, the commit-at-end and better-validation
ideas will help us a lot for a (at a guess) smallish amount of effort.

Any comments? It would be nice to get a story started for this even if
it's going to be a while before we can do any work on it.

Gavin.