unity-design team mailing list archive

[Søren Heldgaard Olesen <heldgaardolesen@xxxxxxxxx>]

2011/6/26 Charline <charline.poirier@xxxxxxxxxxxxx>

> **
> On 26/06/11 15:17, Søren Olesen wrote:
>
> Hi there,
>
> The community around Ubuntu in Aarhus, Denmark, has been invited to a
> technology conference in late August. The theme of the conference is
> user driven innovation and learning from your own mistakes. The audience
> at the conference will be both techies and non-techies, though mostly
> the non-techies. We have accepted the invitation and we will be at the
> conference two days.
>
> Our aim at the conference is twofold. First, we would like to present
> Ubuntu to a wider audience. Second, we would like to show how users can
> take part in making Ubuntu better in order to fit into the bigger theme
> of user driven innovation. We have thought about ways to combine the two
> and have come up with the idea of doing usability testing of Natty
> Narwhal. Usability testing introduces people to Ubuntu and at the same
> time gives information on how to make Ubuntu better. Our point of
> departure is the usability test report from April on natty, the early
> test of Unity and the usability test of Thunderbird. We are thinking
> about doing short usability studies like  "install X program", "post an
> update on Facebook" or "find Y file using a lens". The tests have to be
> shorter than the ones in aforementioned test because we can't keep the
> participants for too long.
>
>  I take it this would be at the conference?  This is a great idea!
>

Yes, it will take place at the conference. The basic idea is to show Ubuntu
to users who have no prior experience with Ubuntu while at the same time
showing how user driven (the theme of the conference) Ubuntu is. So if it
all works out well, we will be able to get valid results that can actually
help improve the usability of Ubuntu.


>
>  In order to make the results we get from the usability results as valid
> as possible we would like some input from on three issues. First, do any
> of you have programs or features that you would like us to test? If you
> do we would be happy to do so. Please keep in mind that the participants
> are non-techies. Second, what kind of background information should we
> get on the participants? I am thinking age, primary OS, experience using
> their OS, how many hours they spend using a computer every day etc.
>
>  Third how should we conduct the test to ensure valid results?
>
>  What do you intend to do with the results?
>

First of all we intend to do bug reporting in launchpad. Second, we are
planning on doing a blog post about 1) show casing Ubuntu, 2) our experience
doing the usability test live and 3) the results of the usability testing
and our huble suggestions on what could be done to fix the usability issues
we encountered.

How reliable do the findings need to be? If you intend this as a
> demonstration to non-tech users of how they can contribute, maybe your most
> important goals should be to 1) have them experienced a usability problem
> and make them realise that the problem is with the technology and not with
> them; and 2) give them a tool to make the experience better.  You could
> build your study in the following way:
>
>    - Select a task where you know there is definitely a usability problem
>     - Choose that usability problem one that you know normal users can
>    formulate.  Remember that end users are not usability professionals or
>    designers nor should they be.  But there are some usability problems users
>    can figure out - a very good example is lack of feedback ("I don't know what
>    it's doing.  I don't know what to do!  The system should be telling me
>    something")
>
> It is a bit of a balancing act. On the one hand we want to make sure that
the participants have a good experience using Ubuntu. We will be handing out
live CD's to the participants hoping that they might try it out on their own
computer if they like it. So picking a known usability problem might spoil
their experience.

On the other hand we want to get interesting and useful results from the
tests as well. I think that if we do a proper briefing and debriefing of the
participants it might work out even if we do the tests on a known issue. If
we tell the participants that this is a known usability problem; your
response is helping fix it, they might have a good overall experience
anyway. We might want to do tell them as a debriefing in order to ensure
that they do not go into the "experiment" knowing there is a usability
issue.

>
>    - Have them write a bug report about that problem - this would support
>    the user-driven innovation part of the conference, would be empowering and
>    drive the end user implication in Ubuntu.
>
> I think asking the users to do a bug report in Launchpad may be too much of
them. I think we will be giving them the choice between helping them do the
bug report in Launchpad and telling us about the problem. If they choose the
latter we will do the bug reporting from them.

>
>
> At the end of the conference you could present a very nice summary of what
> you found, not only about usability problems (if you've read the reports,
> you know that we are already familiar  with them) but also you could speak
> about what end users  about said the issues they encountered and how they
> wrote bug reports.  You could even speak about how developers will use these
> bug reports to make Ubuntu better.  That should be interesting to your
> audience.
>

This is a great idea! At the end of the conference we may have the
opportunity to do 6 minute lightning talk on an issue of our own choice.
Summing up our experience on issues our participants had would be thing to
talk about.

As for validity, keep in mind that valid results will come from:
> 1) how you select your participants, you might not have much choice there;
>
>

We talked a lot about this issue of self selection. But it might not
actually be as big a problem as it seems to be at first. Of course selecting
participants at random would have been better. However the audience at the
conference are ordinary, technology interested people who probably have no
prior experience using Linux.

Furthermore the conference is part of a local festival in Aarhus so it is
being held in central location. Therefore a good part of the audience is
probably just dropping by in order to see what this conference is about.

And we will of course be doing our best to get as many to try out Ubuntu and
take part in our "experiment".

2) how you set up and formulate the task.  Make sure you formulate your
> tasks in a context and scenario that make sense to your participants.
> Basically, you need to formulate your tasks from a user point of view:  For
> example, normal users don't install programmes, when they do, it's because
> there is something else they want to accomplish, the user goal is not to
> install a programme but it might be to edit their photos.  Make the task the
> user goal even better something they do often, like erase red eyes from a
> picture. and;
>

Very good point. We will be putting some work into this the coming months.
Suggestions or ideas on particular tasks are very welcome.


>   3) how you will approach the analysis.  Above, you mentioned age, OS, and
> experience using OS.  You could compare all these different variables in
> your analysis. Generally, I find that Windows and Mac users have different
> expectations and consequently, different usability problems with Ubuntu.
>

That is very interesting. We will definately be asking them what OS they
use.


> Similarly with age.
>

And age for that matter.


>   As for 'expience using OS', you would probably be comparing, in the
> example of editing photos, people who have a lot of experience to people who
> don't (the latters would be 'learning' basically what to do,  which is
> different from using a programme).
>

That is a good point. We will be keeping that in mind. We will probably be
asking them about 1) their general computer experience and 2) their
experience doing the particular task we ask them to do.


>
But remember that the more variables you want to compare, the more
> participants you will need.
>
>
We will be keeping that in mind. I hope that we will be able to a quite
large number of participants. We hope to be at the conference two full days
so we could get maybe fifty to a hundred participants if the task we ask
them to do is somewhat simple. It is of course anybody guess but I think we
would be able to a minimum of thirty participants.

 Would
> desktop recorder or observation do or should we try to get a camera on a
> tripod?
>
>  Desktop recording and observation are most useful.
>
>  Please let me know if you have ideas on what we could do or who we could
> contact.
>
>  Feel free to get in touch with me with any questions.  This is a very
> interesting research and I would be happy to give you any help you need or
> discuss the usability reports you will be using. I am also very keen to hear
> about your results and how your contribution to the conference went.
>

Thanks a lot! Your help is most appreciated. Your reply has really helped us
and lowered the "barrier" to contributing to making Ubuntu better. This is
so cool.

We will be working on the setup in the coming months. The conference is in
late August so we still have some time get all the practical stuff sorted
out.

>  Thanks,
> Søren
>
>
>
> --
>
> *CHARLINE POIRIER**User Research Programme Lead**Canonical**27th floor, 21-24 Millbank Tower**London SW1P 4QP UK *
> *Tel: +44 (0) 20 7630 2491**Mob: +44 (0) 78 8695 4514**www.Ubuntu.com <http://www.Ubuntu.com/> <http://www.Ubuntu.com/>**www.Canonical.com <http://www.Canonical.com/> <http://www.Canonical.com/>*
>
>
>