launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

Hi, all.

I've been meaning to write a state of malone email for awhile now,
especially since others have already done this for other Launchpad
apps.  So here's my catchup notes on what's been happening in
development on malone (a.k.a. the bugs app).


Current work
============

Subscriptions story
-------------------

The main story we're doing development on lately is one that looks to
make our subscriptions handling much better.  The goal is to avoid
generating noise and make email notifications more configurable.  To
this end, we've already added a few new headers, fixed several bugs,
and made a few UI changes to both the email itself and some of the
pages on Launchpad.

The major parts of this work are not widely available yet, but we are
getting close.  When it's all done, you'll be able to set a
notification level on individual bugs and projects/packages.
Notification levels will include:

* send me everything (the default)
* send me changes to bugs but not comments
* send me only lifecycle notices (when bugs are opened or closed)

(These are worded differently on Launchpad, but you get the idea.)

Also, you'll have the ability to filter project/package subscriptions
based on advanced search criteria like status/importance, assignee,
tag, etc.  Yes, this gets us subscribe to a tag!  Hurrah! :-)

All of the backend work for this is done and Graham and Gavin, who are
doing most of the work here, only have to hook this up in the UI.
This was blocked for awhile on a tortuous lazr-js upgrade.  But that's
out of the way, and everyone is UI focused now.  We hope to finish
this before the Launchpad Thunderdome in January.  I think that's
doable.

Timeout/OOPS work
-----------------

Our timeout work goes slower than I would like, but Abel and I focus
on timeout issues as we can.  Abel has done some nice work on
refactoring a few queries to make them faster during the last couple
months.  Robert has really helped us a lot too, both in coding and in
sharing knowledge.  There's still a lot of work to do here.

For me, timeouts have taken precedence over OOPS work because we have
to prioritize somehow.  Obviously, if there's an OOPS causing
immediate issues, it jumps to the front of the queue.

Private attachment librarian work
---------------------------------

Abel did some work initially to create true private attachments.  This
had some negative impact on apport.  We got this working for apport
with work arounds in our DC, and Robert did some work to get a
token-based librarian going, which will fix any remaining issues.
Abel finished off getting Robert's branch ready to land early this
week, and I believe it's playing in ec2 now.

So the librarian will have true private attachments, with none of the
downsides for the API or performance that the first work had.  AIUI,
this will also fix some other related librarian bugs.

Testing
-------

One of the goals we all set for malone this year has been to get our
testing story fixed up.  We have a large number of bad doctests, and
it's often confusing to know where to test certain bits.  We've been
spending a lot of energy on moving bad doctests to unit tests and
converging on agreement about where everything should be tested.  I
think this lines up well with what other teams have done.  I'm still
not happy about testing web pages, which I've written at length about
previously, so I'm personally continuing to think about this a lot.

But overall, I'm happy about the direction we are headed in with
testing malone.  We're cleaning up tests frequently now.  It's common
for bugs hackers to land a branch refactoring tests first, then doing
the follow up dev work.  I'm proud we've moved in this direction.

Also, we know what we want to do now.  If you asked us six months ago
"Where do I put my test for $FOO?" we might have answered, "hmmm, I
don't know, let me look."  Now I think we can answer that type of
question clearly.


Other Components
================

Bug watches
-----------

Early this year we completed a story on making bug watch syncing
reliable and scalable.  There was also a lot of work to refactor bug
watch code (checkwatches and the external bug tracker classes) to make
the code more readable and understandable.  This work was a big
success and bug watch updating now hums along nicely.  If something
does go wrong, we have very good stats in place to know what has
happened (or to what class of watch it's happened to) and we have good
logging to more easily debug what has happened.

Bug watches are updated reliably now and we don't loose much time to
maintaining this system, where we once lost a lot of time to
constantly fiddling with checkwatches.  When something goes wrong
today, we can fix it quickly with little distraction to other work.

Bug expiry
----------

Bug expiry has laid dormant for awhile, but we have bug expiry
re-enabled now.  We're working through the backlog of Ubuntu bugs that
need expiring currently.  After another week or so, we'll turn this on
for every project.  We have been doing this in phases, mostly so that
we can catch any errors easily.  But so far this is going well.

This runs unattended with few issues.  Most of the problems in the
past were social, and I think we did a better job navigating those
issues this time.  There are still folk who dislike the idea of
expiring bugs, but I think we as a team were attentive to those
concerns.  We did a lot of QA on the expiry script before turning it
on to ensure we would only be expiring the bugs we said we would.

Bug Heat
--------

This is updated in real time, thanks to some smart code and a trigger.
 It performs well, and as a general indicator of bugs that are hot
with activity or interesting bits, this has been useful.  I heard from
several at this past UDS that this has been nice to have.  Many would
like to see a similar number for the state of a bug, i.e. a metric for
bugs that are ready to be worked on, since bugs can be hot and still
generally filled with useless information.

We've talked a bit on our team about what we call "Bug Q&A" as a way
to address this request.  Brian and Bryce seemed really in favor of
this idea while doing their rotations on the bugs team.  My hope is
that they evangelize it widely and it gets requested via the
stakeholders meeting. ;)

Bug imports
-----------

This is a constant source of pain for us.  We have a couple of months
where no one wants a bug import, and then suddenly 6 projects need bug
imports.  This is largely manually done now and requires help from a
losa.

We have done a bit of work recently to push back on those requesting
bug imports to get their data right first (via better help
documentation on the wiki, and pointers to a validation command that
can be run), rather than us cleaning it up for them.  But Gavin has
also added a script to the lp tree that will clean up some common
rough bits, like making attachments for long descriptions and
truncating the original.

We have bugs about the remaining work required to make this awesome
and painless.

Search && Managing bugs
-----------------------

I'm lumping these two together since they're related -- finding bugs
vs. lists/reports of bugs.  We haven't had the ability to focus on
this much yet, but we consistently get requests for improvements in
this area.

There's still a lot that could be done in this area.  I know Jono has
some ideas about this concerning future work, and we have a small
feature queued for custom columns in search results.  So we haven't
done much, but it's the next area of malone that should be attended
to.  This work will obviously shape up differently (in terms of who
will do it) with the coming Launchpad squads reorg, but this should be
the next thing to get cleaned up in malone, I believe.


Web UI
======

Bugs home page
--------------

The bugs home page -- bugs.lp.net/$FOO -- has received a lot of love
over the last year.  It's in pretty good shape now.  Information
displays correctly, depending on how the project uses malone.  We've
added a few new quick links in the portlet to jump to search results
people use commonly.  People seem to appreciate the new links.  I
think the hot bugs list is pretty useful now, in terms of a quick
glance at recently active, interesting bugs.

There are some small issues remaining, mostly to do with page
language, but generally this page is in pretty good shape.

Bug page
--------

The bug page hasn't received any serious UI attention in a year,
though we do fix issues as we work on stuff on the page.  I think it's
grown kind of ugly with all the icons, though it does fit well with
the holiday season with all the red and green colors and the icons
looking like little ornaments all over the page. :-)

Ok, I'm having some fun at the expense of the page.  It's not that
bad.  But it does need some UI work now.  It's grown organically over
time, and I think we should find a way to rework the UI so that your
brain doesn't have to process so much info at a glance, just to make
sense of a bug.  I don't have concrete ideas how to do this, but I
think it should involve less icons and not so many colors.  Related
bits of information should be grouped better on the page, too.

Other pages, JavaScript, etc.
-----------------------------

Of course, we have more pages than this, but the above are the big
two.  The rest don't get the attention of those and are generally
useful when you land on them.  There's also a ton of JavaScript, and
I'm personally very invested in getting our js story better (upgrades,
testing, easier to write code, etc.).  Gavin and Graham will likely be
feeling some of this pain more the next few weeks as they work on the
UI for the subscriptions feature.


Final thoughts
==============

When I started as team lead on malone, there was nothing that we
talked about doing that didn't seem like it would require months of
refactoring work.  The biggest of those were bugwatches,
subscriptions, and testing.  These are now in pretty good shape.  Of
course, there are always pieces of code that you look at and want to
turn away your eyes.  However, we've made serious progress over the
last year, and are left with a reasonable set of problems to chip away
at, which I tried to outline above along with the positives.

If you've made it this far, I appreciate the time spent with you. :-)

Cheers,
deryck

-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/