kubuntu-council team mailing list archive

[Engineering and Systems Software Research Group - UFAL <mp+468863@xxxxxxxxxxxxxxxxxx>]

Engineering and Systems Software Research Group - UFAL has proposed merging lp:~easy-ufal/ubuntu-manual-tests/improve-1427 into lp:ubuntu-manual-tests.

Requested reviews:
  Ubuntu Testcase Admins (ubuntu-testcase)

For more details, see:
https://code.launchpad.net/~easy-ufal/ubuntu-manual-tests/improve-1427/+merge/468863

Greetings!

The goal of this merge request is to improve test cases by identifying and removing test smells. The term test smell is often used in automated tests but current research has demonstrated that they also exist in natural language test cases. Potential problems they can cause are non-deterministic behavior,  incomplete verification, poor maintainability, and loss of productivity.

In this merge request, we address the Ambiguous Test smell, which indicates an action in a test case not specific enough, leaving room for different interpretations by the testers. 

In the fourth verification, the tester is prompted about codec installation. We believe that this verification text can lead to confusion. Consider the case where the codecs are already installed. Then, the answer to “Did the easy coded installed recognize any codecs that aren’t installed and install them?” would be “No” as no codecs were installed. This could lead some testers to pass the test as no codec needs to be installed anyway, but others could follow the Smoke Tests logic and fail the test because one or more questions were answered “No”.

The solution we found was to specify in a precondition that the test should only be executed if the codecs were not installed.

References:

Manoel Aranda, et Al. "A Catalog of Transformations to Remove Test Smells in Natural Language Tests." 28th International Conference on Evaluation and Assessment in Software Engineering (EASE 2024).

Elvys Soares, et Al. "Manual Tests Do Smell! Cataloging and Identifying Natural Language Test Smells." 17th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2023).

-- 
Your team Ubuntu Testcase Admins is requested to review the proposed merge of lp:~easy-ufal/ubuntu-manual-tests/improve-1427 into lp:ubuntu-manual-tests.

=== modified file 'testcases/packages/1427_Rhythmbox'
--- testcases/packages/1427_Rhythmbox	2013-08-03 10:34:11 +0000
+++ testcases/packages/1427_Rhythmbox	2024-07-05 19:00:36 +0000
@@ -33,14 +33,20 @@
 
 Test-case name: rhythmbox/rhy-003
 This test will check that Rhythmbox can install codecs and play stream from MTP-compatible players
+<em>
+    To perform this test ensure that:
+    <ul>
+        <li>you have a MTP-compatible MP3 player</li>
+        <li>the required codecs to play the music are NOT installed</li>
+    </ul>
+</em>
 <dl>
-    <dd>Have any MTP-compatible MP3 player</dd>
 <dt> Open the dash and launch rhythmbox by pressing the super key, and then entering ‘rhythmbox’</dt>
     <dd>rhythmbox should launch</dd>
 <dt> Plug a music player device containing MP3 files into your system and check whether rhythmbox imports the music correctly.</dt>
     <dd>A new entry Devices appears</dd>
 <dt> Test playing music from the device.</dt>
-    <dd>Did the easy codec installer recognize any codecs that aren't installed and install them? Did playback start?</dd>
+    <dd>Did the easy codec installer recognize not installed codecs? Did it install them? Did playback start?</dd>
 </dl>
 
 Test-case name: rhythmbox/rhy-004