zorba-coders team mailing list archive

[Chris Hillery <874679@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

The second (optional) argument to ADD_TEST_DIRECTORY() contains a list
of test cases that should be skipped (ie, not created at all). This is
extremely dangerous. By bypassing them entirely, there is no way of
detecting when the bug is fixed. We have already had serious problems
caused by test removal hiding the introduction of new bugs in the past.

I understand that removing a test that segfaults is necessary on Windows
in order for the tests to run unattended. However, it should never be
done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the
bugs as EXPECTED_FAILURE()s on other platforms. (I realize that
*currently* all places that pass a list of segfaulting tests to
ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient
- ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped
only on Windows, or else sooner or later someone will accidentally do it
wrong. There's currently no documentation either, making this much more
likely.)

In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list
of pairs of the form (testname, bug ID), or else accept two parallel
lists (testnames and bug IDs), so it can pass the bug number to
EXPECTED_FAILURE().

A slightly more difficult approach would be to introduce a new
KNOWN_SEGFAULT() macro that handled this in conjunction with
ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though,
because it would (silently) not do anything for tests added by any means
other than ADD_TEST_DIRECTORY().

** Affects: zorba
     Importance: High
     Assignee: Gabriel Petrovay (gabipetrovay)
         Status: New

** Changed in: zorba
   Importance: Undecided => High

** Changed in: zorba
     Assignee: (unassigned) => Gabriel Petrovay (gabipetrovay)

** Changed in: zorba
    Milestone: None => 2.1

-- 
You received this bug notification because you are a member of Zorba
Coders, which is the registrant for Zorba.
https://bugs.launchpad.net/bugs/874679

Title:
  Clean up second arg to ADD_TEST_DIRECTORY()

Status in Zorba - The XQuery Processor:
  New

Bug description:
  The second (optional) argument to ADD_TEST_DIRECTORY() contains a list
  of test cases that should be skipped (ie, not created at all). This is
  extremely dangerous. By bypassing them entirely, there is no way of
  detecting when the bug is fixed. We have already had serious problems
  caused by test removal hiding the introduction of new bugs in the
  past.

  I understand that removing a test that segfaults is necessary on
  Windows in order for the tests to run unattended. However, it should
  never be done on other platforms. So ADD_TEST_DIRECTORY() should
  instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I
  realize that *currently* all places that pass a list of segfaulting
  tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not
  sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests
  are skipped only on Windows, or else sooner or later someone will
  accidentally do it wrong. There's currently no documentation either,
  making this much more likely.)

  In order to do this, ADD_TEST_DIRECTORY() needs to either accept a
  list of pairs of the form (testname, bug ID), or else accept two
  parallel lists (testnames and bug IDs), so it can pass the bug number
  to EXPECTED_FAILURE().

  A slightly more difficult approach would be to introduce a new
  KNOWN_SEGFAULT() macro that handled this in conjunction with
  ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea,
  though, because it would (silently) not do anything for tests added by
  any means other than ADD_TEST_DIRECTORY().

To manage notifications about this bug go to:
https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions