Re: [Ayatana] Usability testing as a show case of non-techies can contribute to Ubuntu

[Charline <charline.poirier@xxxxxxxxxxxxx>]

On 26/06/11 15:17, Søren Olesen wrote:

Hi there,

The community around Ubuntu in Aarhus, Denmark, has been invited to a
technology conference in late August. The theme of the conference is
user driven innovation and learning from your own mistakes. The audience
at the conference will be both techies and non-techies, though mostly
the non-techies. We have accepted the invitation and we will be at the
conference two days.

Our aim at the conference is twofold. First, we would like to present
Ubuntu to a wider audience. Second, we would like to show how users can
take part in making Ubuntu better in order to fit into the bigger theme
of user driven innovation. We have thought about ways to combine the two
and have come up with the idea of doing usability testing of Natty
Narwhal. Usability testing introduces people to Ubuntu and at the same
time gives information on how to make Ubuntu better. Our point of
departure is the usability test report from April on natty, the early
test of Unity and the usability test of Thunderbird. We are thinking
about doing short usability studies like  "install X program", "post an
update on Facebook" or "find Y file using a lens". The tests have to be
shorter than the ones in aforementioned test because we can't keep the
participants for too long.

I take it this would be at the conference?  This is a great idea!

In order to make the results we get from the usability results as valid
as possible we would like some input from on three issues. First, do any
of you have programs or features that you would like us to test? If you
do we would be happy to do so. Please keep in mind that the participants
are non-techies. Second, what kind of background information should we
get on the participants? I am thinking age, primary OS, experience using
their OS, how many hours they spend using a computer every day etc.

Third how should we conduct the test to ensure valid results? 

What do you intend to do with the results? How reliable do the findings need to be? If you intend this as a demonstration to non-tech users of how they can contribute, maybe your most important goals should be to 1) have them experienced a usability problem and make them realise that the problem is with the technology and not with them; and 2) give them a tool to make the experience better.  You could build your study in the following way:

• Select a task where you know there is definitely a usability problem
  
• Choose that usability problem one that you know normal users can formulate.  Remember that end users are not usability professionals or designers nor should they be.  But there are some usability problems users can figure out - a very good example is lack of feedback ("I don't know what it's doing.  I don't know what to do!  The system should be telling me something")
• Have them write a bug report about that problem - this would support the user-driven innovation part of the conference, would be empowering and drive the end user implication in Ubuntu.

At the end of the conference you could present a very nice summary of what you found, not only about usability problems (if you've read the reports, you know that we are already familiar  with them) but also you could speak about what end users  about said the issues they encountered and how they wrote bug reports.  You could even speak about how developers will use these bug reports to make Ubuntu better.  That should be interesting to your audience.

As for validity, keep in mind that valid results will come from:
1) how you select your participants, you might not have much choice there; 
2) how you set up and formulate the task.  Make sure you formulate your tasks in a context and scenario that make sense to your participants. Basically, you need to formulate your tasks from a user point of view:  For example, normal users don't install programmes, when they do, it's because there is something else they want to accomplish, the user goal is not to install a programme but it might be to edit their photos.  Make the task the user goal even better something they do often, like erase red eyes from a picture. and;
 3) how you will approach the analysis.  Above, you mentioned age, OS, and experience using OS.  You could compare all these different variables in your analysis. Generally, I find that Windows and Mac users have different expectations and consequently, different usability problems with Ubuntu.  Similarly with age.  As for 'expience using OS', you would probably be comparing, in the example of editing photos, people who have a lot of experience to people who don't (the latters would be 'learning' basically what to do,  which is different from using a programme).  But remember that the more variables you want to compare, the more participants you will need.

Would
desktop recorder or observation do or should we try to get a camera on a
tripod?

Desktop recording and observation are most useful.

Please let me know if you have ideas on what we could do or who we could
contact.

Feel free to get in touch with me with any questions.  This is a very interesting research and I would be happy to give you any help you need or discuss the usability reports you will be using. I am also very keen to hear about your results and how your contribution to the conference went.

Thanks,
Søren

--

CHARLINE POIRIER
User Research Programme Lead
Canonical
27th floor, 21-24 Millbank Tower
London SW1P 4QP UK 

Tel: +44 (0) 20 7630 2491
Mob: +44 (0) 78 8695 4514
www.Ubuntu.com <http://www.Ubuntu.com/>
www.Canonical.com <http://www.Canonical.com/>