launchpad-dev team mailing list archive

["Francis J. Lacoste" <francis.lacoste@xxxxxxxxxxxxx>]

Hi Deryck,

Thanks for this write-up. A few thoughts:

* I think we should apologize on the blog.
* Can you create a PolicyandProcess/MassEmail wiki page containing the best 
practice here based on your experience and Robert's suggestions.
* A long time ago, Barry wrote massmail which was used to send our initial 
beta announcement. It doesn't support all the use cases you talk about, but 
does a few thing right. Unfortunately, unless you would have asked one of the 
few who used that (barry, mark, me, ...) you couldn't have guessed of its 
existence. It's not even on Launchpad (it's on devpad). The version I used is 
available on devpad:~flacoste/massmail But I guess, someone else might have a 
more up to date version.


-- 
Francis J. Lacoste
francis.lacoste@xxxxxxxxxxxxx

On October 6, 2010, Deryck Hodge wrote:
> On Tue, Oct 5, 2010 at 8:49 PM, Robert Collins
> 
> <robert.collins@xxxxxxxxxxxxx> wrote:
> > I received 26 mails telling me about expiry changes; many people will
> > have received more - I'm not particularly prolific in my involvement
> > with upstream projects.
> > 
> > I realise that expiry has been a hot potato and the goal was to make
> > sure folk know... however I think that future things along this line
> > would benefit by a few tweaks.
> > 
> > Firstly, i think it would be nice if there were a blog post about it,
> > because many *users* of the bug system will be affected even if they
> > are not a 'bug supervisor'.
> > 
> > With a blog post in place, if we really need to grab peoples
> > attention, sending a single email with a summary of their projects,
> > the settings on those projects and a link to the blog post for details
> > would let folk know with only a single mail, and also give them a good
> > reference list for the things they may want to change.
> > 
> > I don't know how many folk will have received duplicate mails, but if
> > we think its a large number, we might patch up things a bit by
> > offering an apology..... on the blog. I do know at least one upstream
> > who found the volume of emails distressing (and they only received 10
> > or so).
> 
> Hi, Robert.
> 
> Thanks for the thoughtful inquiry about this.
> 
> I think it's worth being clear that this issue is completely due to
> stupidity on my part.  I was the one who wrote the script to send the
> emails, and I made a couple of mistakes that in hindsight are obvious
> to me (and everyone else).  These mistakes had huge ramifications in
> terms of the amount of mail that was sent.
> 
> I apologize for my bad handling of this email responsibility.  I'm
> very sorry for any negative impact or embarrassment it brings on
> Launchpad or Canonical.  I'm happy to write an apology on the
> Launchpad blog if people feel it's warranted or will help.
> 
> I was indeed planning a blog post to broadcast the coming change to a
> wider audience.  I was going to do it yesterday after sending the
> emails, but with some of the things that went wrong and the great
> amount of email I've received myself about this, I didn't get the
> chance to write that blog post.  I'll do it very soon today.
> 
> This communications plan was highly discussed, so even though I
> mishandled the email sending, the particular pieces of communication
> were well laid out.  See the wiki and the original bug if you want
> some of the history or previous discussion.
> 
> https://dev.launchpad.net/Bugs/CommuncationForBugExpiry
> https://bugs.edge.launchpad.net/malone/+bug/333521
> 
> So what went wrong?
> 
> 1.) Sending emails in the first place
> 
> I was hesitant to even send emails because I felt it was overkill, but
> due to others' concern that the last bug expiration attempt was a huge
> failure, I agreed.
> 
> My initial concern were that something could go horribly wrong because
> we lack the tools to do mass mailings well.  This incident confirms
> that.  In the future, I will be *highly* reluctant to send mass
> emails, and I encourage anyone else who is asked to do this to think
> long and hard about it before agreeing to do so.
> 
> I also think this kind of announcement is more appropriate on
> Launchpad itself, perhaps using feature flags when a user visits any
> page on his or her project for the first time after such a change is
> made.
> 
> 2.) Having to write ad hoc scripts to send email
> 
> Everywhere else that I've worked has had nicer tools for sending mass
> emails, and also, the responsibility to send these emails never fell
> on individual developers.  The fact that we as developers have to
> write an ad hoc script to do this sort of thing indicates we're not
> capable of doing it well.
> 
> At the every least, we should convert what I have into a generic class
> that can take some precautions for us -- checking that we're not
> sending multiple emails to the same address, etc.  Best case, we would
> have a web-based tool that would allow us to select users based on
> criteria and supply an email template.  I believe there are many
> pre-existing tools for this sort of thing, but I'm not sure if any of
> them are free software.  Whether or not they would work with Launchpad
> is also another concern.
> 
> 3.) Sending emails before making the change
> 
> I sent emails to users before we disabled the switch on Launchpad.  I
> did one action right after the other, but clearly some people could
> have clicked through and seen the switch still enabled, which could
> have been confusing.  I was asked to re-send the emails because of
> this, which is why project owners could have gotten two emails for the
> same project.
> 
> I shouldn't have re-sent these emails.  I should have sent a
> completely different follow up email or sent nothing at all.
> 
> 4.) Worrying about projects and not people
> 
> Believe it or not, I was trying to be cautious about not spamming.
> But I was focused on not sending duplicates to projects, not a
> person's email address.  In hindsight, this is the fatal flaw, as many
> of our users own multiple projects.  As Robert and others have noted,
> I should have sent a single email listing all projects.
> 
> I have no excuse for this mistake.  I simply didn't see the forest for
> the trees, if you'll allow the cliche.  Having an existing MassMailer
> class that I could have sub-classed that warned or prevented multiple
> mails could have saved me here.  Having some guidelines about how to
> send mass mail would have helped, too.
> 
> I did do multiple test runs on staging and looked at log output, but
> because I was focused on not sending multiple mails to projects and
> there was a lot of output, I overlooked that I was sending multiple
> mails to the same developer.
> 
> 5.) Sending to bug supervisors
> 
> Part of our initial plan was to send to bug supervisors if one was
> defined, rather than project owners, thinking this would be the person
> responsible for this setting.  I should have confirmed this because
> bug supervisors actually cannot change this setting.  You would think
> I should know this since I manage the bugs team, but it wasn't obvious
> to me.  Apparently, it's not obvious to anyone else because I was
> clear that we would do this with everyone who asked about this and
> reviewed the script.
> 
> I had to send follow up mails to these people saying "sorry, but you
> should not have received that mail."
> 
> We should have clear descriptions of the roles on Launchpad and their
> responsibilities and review this before sending any mass mails again.
> 
> 
> 
> That's about it for the issues.  Again, my deepest apologies for my
> inept handling of this.  I hope others can learn from my mistakes and
> that we can improve our guidelines and tools for doing this so that
> it's easier and less stressful should someone else have to do this
> again in the future.  I do think we should be extremely cautious about
> sending mass email again and invest in our feature flag work to make
> it trivial to announce these things to users on Launchpad itself.
> 
> FWIW, the Internet does have a sense of justice.  My Inbox is equally
> flooded in auto responders and unsubscribe requests.
> 
> 
> Cheers,
> deryck

Attachment: signature.asc
Description: This is a digitally signed message part.