launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

On Tue, Oct 5, 2010 at 8:49 PM, Robert Collins
<robert.collins@xxxxxxxxxxxxx> wrote:
> I received 26 mails telling me about expiry changes; many people will
> have received more - I'm not particularly prolific in my involvement
> with upstream projects.
>
> I realise that expiry has been a hot potato and the goal was to make
> sure folk know... however I think that future things along this line
> would benefit by a few tweaks.
>
> Firstly, i think it would be nice if there were a blog post about it,
> because many *users* of the bug system will be affected even if they
> are not a 'bug supervisor'.
>
> With a blog post in place, if we really need to grab peoples
> attention, sending a single email with a summary of their projects,
> the settings on those projects and a link to the blog post for details
> would let folk know with only a single mail, and also give them a good
> reference list for the things they may want to change.
>
> I don't know how many folk will have received duplicate mails, but if
> we think its a large number, we might patch up things a bit by
> offering an apology..... on the blog. I do know at least one upstream
> who found the volume of emails distressing (and they only received 10
> or so).
>

Hi, Robert.

Thanks for the thoughtful inquiry about this.

I think it's worth being clear that this issue is completely due to
stupidity on my part.  I was the one who wrote the script to send the
emails, and I made a couple of mistakes that in hindsight are obvious
to me (and everyone else).  These mistakes had huge ramifications in
terms of the amount of mail that was sent.

I apologize for my bad handling of this email responsibility.  I'm
very sorry for any negative impact or embarrassment it brings on
Launchpad or Canonical.  I'm happy to write an apology on the
Launchpad blog if people feel it's warranted or will help.

I was indeed planning a blog post to broadcast the coming change to a
wider audience.  I was going to do it yesterday after sending the
emails, but with some of the things that went wrong and the great
amount of email I've received myself about this, I didn't get the
chance to write that blog post.  I'll do it very soon today.

This communications plan was highly discussed, so even though I
mishandled the email sending, the particular pieces of communication
were well laid out.  See the wiki and the original bug if you want
some of the history or previous discussion.

https://dev.launchpad.net/Bugs/CommuncationForBugExpiry
https://bugs.edge.launchpad.net/malone/+bug/333521

So what went wrong?

1.) Sending emails in the first place

I was hesitant to even send emails because I felt it was overkill, but
due to others' concern that the last bug expiration attempt was a huge
failure, I agreed.

My initial concern were that something could go horribly wrong because
we lack the tools to do mass mailings well.  This incident confirms
that.  In the future, I will be *highly* reluctant to send mass
emails, and I encourage anyone else who is asked to do this to think
long and hard about it before agreeing to do so.

I also think this kind of announcement is more appropriate on
Launchpad itself, perhaps using feature flags when a user visits any
page on his or her project for the first time after such a change is
made.

2.) Having to write ad hoc scripts to send email

Everywhere else that I've worked has had nicer tools for sending mass
emails, and also, the responsibility to send these emails never fell
on individual developers.  The fact that we as developers have to
write an ad hoc script to do this sort of thing indicates we're not
capable of doing it well.

At the every least, we should convert what I have into a generic class
that can take some precautions for us -- checking that we're not
sending multiple emails to the same address, etc.  Best case, we would
have a web-based tool that would allow us to select users based on
criteria and supply an email template.  I believe there are many
pre-existing tools for this sort of thing, but I'm not sure if any of
them are free software.  Whether or not they would work with Launchpad
is also another concern.

3.) Sending emails before making the change

I sent emails to users before we disabled the switch on Launchpad.  I
did one action right after the other, but clearly some people could
have clicked through and seen the switch still enabled, which could
have been confusing.  I was asked to re-send the emails because of
this, which is why project owners could have gotten two emails for the
same project.

I shouldn't have re-sent these emails.  I should have sent a
completely different follow up email or sent nothing at all.

4.) Worrying about projects and not people

Believe it or not, I was trying to be cautious about not spamming.
But I was focused on not sending duplicates to projects, not a
person's email address.  In hindsight, this is the fatal flaw, as many
of our users own multiple projects.  As Robert and others have noted,
I should have sent a single email listing all projects.

I have no excuse for this mistake.  I simply didn't see the forest for
the trees, if you'll allow the cliche.  Having an existing MassMailer
class that I could have sub-classed that warned or prevented multiple
mails could have saved me here.  Having some guidelines about how to
send mass mail would have helped, too.

I did do multiple test runs on staging and looked at log output, but
because I was focused on not sending multiple mails to projects and
there was a lot of output, I overlooked that I was sending multiple
mails to the same developer.

5.) Sending to bug supervisors

Part of our initial plan was to send to bug supervisors if one was
defined, rather than project owners, thinking this would be the person
responsible for this setting.  I should have confirmed this because
bug supervisors actually cannot change this setting.  You would think
I should know this since I manage the bugs team, but it wasn't obvious
to me.  Apparently, it's not obvious to anyone else because I was
clear that we would do this with everyone who asked about this and
reviewed the script.

I had to send follow up mails to these people saying "sorry, but you
should not have received that mail."

We should have clear descriptions of the roles on Launchpad and their
responsibilities and review this before sending any mass mails again.



That's about it for the issues.  Again, my deepest apologies for my
inept handling of this.  I hope others can learn from my mistakes and
that we can improve our guidelines and tools for doing this so that
it's easier and less stressful should someone else have to do this
again in the future.  I do think we should be extremely cautious about
sending mass email again and invest in our feature flag work to make
it trivial to announce these things to users on Launchpad itself.

FWIW, the Internet does have a sense of justice.  My Inbox is equally
flooded in auto responders and unsubscribe requests.


Cheers,
deryck



-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/