elementary-dev-community team mailing list archive

[Alex Lourie <djay.il@xxxxxxxxx>]

Hello elementary friends,

As many of you know, devs are now in the process of developing the next
release of elementary, Isis. During various conversations there’s this one
subject that has risen quite often - how can testing be incorporated in the
process.

For the impatient ones, jump right away to the instructions at [1].

Now, before some anxious folks jump ahead, let me stop you all and say it's
NOT about TDD. We've been there, done the discussions, and for now it was
decided that no one is doing that [2]. What I am going to talk about now is
system/integration level testing, that is the curse called QE [3].

QE is somewhat unloved, seems dark and boring among and by many developers,
especially in open source. Mainly due to the fact, that in open source we
consider our users to be our testers. While it could be working well enough
for various libraries, and different software packages, I claim it cannot
work for users of the entire OS/DE, especially the one that aspires to
provide the best and most polished experience for its users. We cannot
expect high level of technical capabilities from our users to understand
what exactly is the problem they experience, let alone to report bugs they
find.

As such I had few private discussions with members of the elementary dev
teams, and came with to the following assumptions:

1. Nobody will be forced to do TDD. IF (and that's a big IF) someone wants
to do it, they can do it on their own expense.

2. Most elementary applications are not completely new inventions. They are
just new applications that are based on the mostly stable sets of
libraries, which have their own projects, boards and testing methods.

3. External dependencies are just expected to work, and we/users will post
bugs if they don't.

4. In system/integration 'testing' it is interesting to see that our
applications work; that is the other technologies are integrated correctly
and a good working interface is created.

5. Any type of automatic testing for the apps is fine, as long as it works.

6. System Testing can be done in any language.

Finally, as I am mostly familiar with Python, I thought I'd check the
autopilot framework, used by Ubuntu to test their entire system.

Hereby I wanted to tell that first preliminary testing is done. I've
successfully installed autopilot on my Isis dev build box, adopted one set
of tests [4] and they now pass. That means, that technically it is possible
to use autopilot for testing our applications.

So... to read how to run the tests on your own, read the How-To [1]

==== What next? ====

So here we are now, with one part of elementary that can be tested
automatically. I don't like to fool anyone nor myself with claims that now
we have implemented testing. So here few possible tasks that I think should
be done next to move with this stuff:

1. One possible task is to create more tests for pantheon-terminal. I'm
sure its developers will be swayed with joy.

2. Another possible task is to create some small set of tests for another
app.

I'm sure that the elementary dev/users community can come up with awesome
ideas as to what can be done with this. I'm all ears now and open to
suggestions.

So long, and remember, I talk too much, and I'm a crappy developer. Any
complaints are welcome in sane limits.

Cheers.


[1]
https://docs.google.com/document/d/1AHDT0hR8mjVcyhERsR7qC5Zs1GB1SNfUFHR6gFuJVTM/edit?usp=sharing

[2] Unless someone does it, maintains it and keeps doing it until someone
else does it too.

[3] Quality Engineering (once called Quality Assurance, but that's not
fare, as it somewhat equates quality engineers to robots. They are not.
Actually, they are very talented engineers, and if one considers that their
work is to do whatever there is in their power to break stuff, one could
probably like it too).

[4] Tests for pantheon-terminal were adopted from gnome-terminal and pass
completely (9 tests). The code for the tests is at
https://github.com/alourie/pantheon_terminal<http://unity.ubuntu.com/autopilot/faq/faq.html>

<http://unity.ubuntu.com/autopilot/faq/faq.html>
<http://unity.ubuntu.com/autopilot/faq/faq.html>

-- 

Alex Lourie