On 26/06/11 15:17, Søren Olesen wrote:
Hi there,
The community around Ubuntu in Aarhus, Denmark, has been invited to a
technology conference in late August. The theme of the conference is
user driven innovation and learning from your own mistakes. The audience
at the conference will be both techies and non-techies, though mostly
the non-techies. We have accepted the invitation and we will be at the
conference two days.
Our aim at the conference is twofold. First, we would like to present
Ubuntu to a wider audience. Second, we would like to show how users can
take part in making Ubuntu better in order to fit into the bigger theme
of user driven innovation. We have thought about ways to combine the two
and have come up with the idea of doing usability testing of Natty
Narwhal. Usability testing introduces people to Ubuntu and at the same
time gives information on how to make Ubuntu better. Our point of
departure is the usability test report from April on natty, the early
test of Unity and the usability test of Thunderbird. We are thinking
about doing short usability studies like "install X program", "post an
update on Facebook" or "find Y file using a lens". The tests have to be
shorter than the ones in aforementioned test because we can't keep the
participants for too long.
I take it this would be at the conference? This is a great idea!
Yes, it will take place at the conference. The basic idea is to show Ubuntu to users who have no prior experience with Ubuntu while at the same time showing how user driven (the theme of the conference) Ubuntu is. So if it all works out well, we will be able to get valid results that can actually help improve the usability of Ubuntu.
In order to make the results we get from the usability results as valid
as possible we would like some input from on three issues. First, do any
of you have programs or features that you would like us to test? If you
do we would be happy to do so. Please keep in mind that the participants
are non-techies. Second, what kind of background information should we
get on the participants? I am thinking age, primary OS, experience using
their OS, how many hours they spend using a computer every day etc.
Third how should we conduct the test to ensure valid results?
What do you intend to do with the results?
First of all we intend to do bug reporting in launchpad. Second, we are planning on doing a blog post about 1) show casing Ubuntu, 2) our experience doing the usability test live and 3) the results of the usability testing and our huble suggestions on what could be done to fix the usability issues we encountered.
How reliable do the
findings need to be? If you intend this as a demonstration to
non-tech users of how they can contribute, maybe your most important
goals should be to 1) have them experienced a usability problem and
make them realise that the problem is with the technology and not
with them; and 2) give them a tool to make the experience better.
You could build your study in the following way:
- Select a task where you know there is definitely a usability
problem
- Choose that usability problem one that you know normal users
can formulate. Remember that end users are not usability
professionals or designers nor should they be. But there are
some usability problems users can figure out - a very good
example is lack of feedback ("I don't know what it's doing. I
don't know what to do! The system should be telling me
something")
It is a bit of a balancing act. On the one hand we want to make
sure that the participants have a good experience using Ubuntu. We will
be handing out live CD's to the participants hoping that they might try
it out on their own computer if they like it. So picking a known
usability problem might spoil their experience.
On the other hand we want to get interesting and useful results from the
tests as well. I think that if we do a proper briefing and debriefing of the
participants it might work out even if we do the tests on a known issue. If we tell the participants that this is a known usability problem; your response is helping
fix it, they might have a good overall experience anyway. We might want
to do tell them as a debriefing in order to ensure that they do not go
into the "experiment" knowing there is a usability issue.
- Have them write a bug report about that problem - this would
support the user-driven innovation part of the conference, would
be empowering and drive the end user implication in Ubuntu.
I think asking the users to do a bug report in Launchpad may be too much of them. I think we will be giving them the choice between helping them do the bug report in Launchpad and telling us about the problem. If they choose the latter we will do the bug reporting from them.
At the end of the conference you could present a very nice summary
of what you found, not only about usability problems (if you've read
the reports, you know that we are already familiar with them) but
also you could speak about what end users about said the issues
they encountered and how they wrote bug reports. You could even
speak about how developers will use these bug reports to make Ubuntu
better. That should be interesting to your audience.
This is a great idea! At the end of the conference we may have the opportunity to do 6 minute lightning talk on an issue of our own choice. Summing up our experience on issues our participants had would be thing to talk about.
As for validity, keep in mind that valid results will come from:
1) how you select your participants, you might not have much choice
there;
We talked a lot about this issue of self selection. But it might not actually be as big a problem as it seems to be at first. Of course selecting participants at random would have been better. However the audience at the conference are ordinary, technology interested people who probably have no prior experience using Linux.
Furthermore the conference is part of a local festival in Aarhus so it is being held in central location. Therefore a good part of the audience is probably just dropping by in order to see what this conference is about.
And we will of course be doing our best to get as many to try out Ubuntu and take part in our "experiment".
2) how you set up and formulate the task. Make sure you formulate
your tasks in a context and scenario that make sense to your
participants. Basically, you need to formulate your tasks from a
user point of view: For example, normal users don't install
programmes, when they do, it's because there is something else they
want to accomplish, the user goal is not to install a programme but
it might be to edit their photos. Make the task the user goal even
better something they do often, like erase red eyes from a picture.
and;
Very good point. We will be putting some work into this the coming months. Suggestions or ideas on particular tasks are very welcome.
3) how you will approach the analysis. Above, you mentioned age,
OS, and experience using OS. You could compare all these different
variables in your analysis. Generally, I find that Windows and Mac
users have different expectations and consequently, different
usability problems with Ubuntu.
That is very interesting. We will definately be asking them what OS they use.
Similarly with age.
And age for that matter.
As for
'expience using OS', you would probably be comparing, in the example
of editing photos, people who have a lot of experience to people who
don't (the latters would be 'learning' basically what to do, which
is different from using a programme).
That is a good point. We will be keeping that in mind. We will probably be asking them about 1) their general computer experience and 2) their experience doing the particular task we ask them to do.
But remember that the more
variables you want to compare, the more participants you will need.
We will be keeping that in mind. I hope that we will be able to a quite large number of participants. We hope to be at the conference two full days so we could get maybe fifty to a hundred participants if the task we ask them to do is somewhat simple. It is of course anybody guess but I think we would be able to a minimum of thirty participants.
Would
desktop recorder or observation do or should we try to get a camera on a
tripod?
Desktop recording and observation are most useful.
Please let me know if you have ideas on what we could do or who we could
contact.
Feel free to get in touch with me with any questions. This is a
very interesting research and I would be happy to give you any help
you need or discuss the usability reports you will be using. I am
also very keen to hear about your results and how your contribution
to the conference went.
Thanks a lot! Your help is most appreciated. Your reply has really helped us and lowered the "barrier" to contributing to making Ubuntu better. This is so cool.
We will be working on the setup in the coming months. The conference is in late August so we still have some time get all the practical stuff sorted out.