ubuntu-touch-coreapps team mailing list archive

[Nicholas Skaggs <nicholas.skaggs@xxxxxxxxxxxxx>]

Our quest to test all the things 'Core Apps' has been tumultus to say 
the least over the past year. We lost our jenkins and migrated to a new 
one, all while experiencing growing pains. The platform as well has 
continued to shift and innovate and our poor acceptance tests have 
struggled to be run and keep up. This causes heartache for everyone, and 
the sea of red test failures isn't helpful.

So I decided we needed to take a long-term approach to fixing this. I 
drafted a plan with Alan and QA, along with some feedback from a couple 
of the core app developers. The idea is to right-size our testing 
strategy[1].

While we've advocated for using the testing pyramid to test 
applications, we've written and depended on mostly autopilot tests. This 
is partially because of our early adoption of autopilot, and because I 
find them so fun to write! I'm sure there is another person or two who 
might say the same, but by and large the autopilot tests have proven to 
be not easy to write or maintain. Hence there place on the top of the 
pyramid[2]. Acceptance tests such as autopilot should represent the 
minority of our test cases, not the majority. And as we've seen, despite 
our efforts, the platform has broken our unchanged autopilot tests and 
applications.

So, while I encourage you to take a quick review of the summary and 
action plans from the document[1], let me summarize the changes here as 
well. In short, we are going to:

1) Remove unmaintainable autopilot tests
2) Cleanup the core apps test suites for best practices, old code. And 
also add documentation and readme's.
3) Add new qml tests to enhance and expand our unit testing
4) Use manual testing for those tests that would be hard to automate

Writing qml tests should prove to be much easier to do, and easier to 
maintain. They test only our internal application code, and they are 
written in a language you already know! The takeaway for you is look for 
MP's landing that do all of the above, and start thinking about and 
including / expanding upon your unit test suites. I guarantee you'll 
find writing them to be more fun than autopilot!

Finally, please do provide feedback on the changes. The plan has been 
tweaked a few times already, and I'm sure it will see a few more tweaks. 
Thanks for a great set of core applications, and I hope these changes 
make your development experiences better.

Nicholas

P.S. You'll also note the workflow for release has changed, for one, AP 
tests don't run on merges now -- they will never block you. Check it 
out: https://wiki.ubuntu.com/Touch/CoreApps/Jenkins#Workflow

1. 
https://docs.google.com/document/d/1ApGwM9iI9qArbu4fMMdDs4Irb9fULfSVIjm79E-UBpQ/edit#heading=h.davy9ssj4q67
2. https://developer.ubuntu.com/en/start/quality/