autopilot-users team mailing list archive

[Michael Zanetti <michael.zanetti@xxxxxxxxxxxxx>]

Hi,

lately I started working quite a bit with autopilot. After writing and 
collecting test cases that worked perfectly fine on my notebook, I hooked the 
according projects up to jenkins to execute all those autopilot tests in an 
automated manner. As usual, if something sounds straight forward, it is not. A 
high percentage of the tests were randomly failing in Jenkins so I had to fix 
them all (including some tunings in Jenkins itself). The problem was, that we 
use VM's in Jenkins which have only a fraction of the horsepower of the 
average desktop PC.

Most of the failures happened for quite obvious reasons, but if you're new to 
autopilot you most likely are not aware of the issues that can come along. To 
prevent you tapping into the same mistakes I'd like to share some hints that 
should help you writing more robust test cases:


* Application startup

An app might take longer to get visible while it already responds to autopilot 
requests in the background. This results in autopilot getting some coordinates 
for, lets say, a button and clicking in that area. Because the window is not 
visible yet, the click ends up in empty space and the test will fail. While I 
couldn't find a 100% rock solid solution for this (as the delay happens inside 
X11), there are ways to minimize the risk of this happening:

In your test's setUp() method, wait for at least the app believing its ready, 
for example by doing this:

    def setUp(self):
        super(TestMainWindow, self).setUp()
        self.assertThat(self.main_window.get_qml_view().visible, 
Eventually(Equals(True)))

If you have some sort of loader component, or a splash screen in your app, 
connect to it and wait for it to be completed before running the actual test 
case.



* Use Eventually - always

For example if you have a button with the text "click me" that changes to 
"clicked" when you actually click it, this works 100% reliable on your 
desktop:

self.assertThat(pushbutton.text, Equals("click me"))
self.mouse.move_to_object(pushbutton)
self.mouse.click()
self.assertThat(pushbutton.text, Equals("clicked"))

However, if executing this in a Jenkins farm with tons of other tests and 
builds going on a the same time, it starts behaving nasty. Your tests start 
failing because the pushbutton's text still says "click me" even though it has 
been already clicked and should have changed its text.

Always use Eventually() - also for immediate actions.

self.assertThat(pushbutton.text, Equals("click me"))
self.mouse.move_to_object(pushbutton)
self.mouse.click()
self.assertThat(pushbutton.text, Eventually(Equals("clicked")))

This in turn means, that if you want to use Autopilot to check if some action 
has an immediate reaction without any delay, you should not use some high-load 
Jenkins instance for this test but rather go for your real target hardware (I 
guess this is true for every timing related test anyways).



* Be aware of different sizes/resolutions that might introduce rounding errors:

Example:
We have a QML Rectangle that should be centered to the last mouse click's 
coordinates. This would be the first you might think of:

self.mouse.move(200, 100)
self.mouse.click()
self.assertThat(rectangle.globalRect[0] + rectangle.globalRect[2] / 2, 200)
self.assertThat(rectangle.globalRect[1] + rectangle.globalRect[3] / 2, 100)

While it might work 100% reliable on your desktop, it suddenly starts failing 
in a Jenkins VM because the rectangle's center is at 200,99 instead of 
200,100. This may happen due to rounding errors because your UI elements are 
scaled a little different. As pixels don't like to be devided and we are kinda 
stuck on integers here, a possible solution would be to do something like 
this:

self.mouse.move(200, 100)
self.mouse.click()
rect_center_x = rectangle.globalRect[0] + rectangle.globalRect[2] / 2
rect_center_y = rectangle.globalRect[1] + rectangle.globalRect[3] / 2
self.assertThat(abs(rect_center_x - 200), Eventually(LessThan(2)))
self.assertThat(abs(rect_center_y - 100), Eventually(LessThan(2)))


That's it for now, hope this helps you to avoid some jenkins build runs 
figuring what's wrong with your tests. If you have experienced similar 
situation, let me know.

Br,
Michael