ecryptfs-devel team mailing list archive

[Serge Hallyn <serge.hallyn@xxxxxxxxxxxxx>]

Quoting Dustin Kirkland (kirkland@xxxxxxxxxx):
> On Thu, Aug 4, 2011 at 12:32 PM, Serge E. Hallyn
> <serge.hallyn@xxxxxxxxxxxxx> wrote:
> > It was suggested (URL) that we should look at the phoronix testsuite for
> > testing ecryptfs.  Phoronix very nicely automates the installation and running
> > of many benchmarks.  It doesn't however appear to have any actual correctness
> > tests.  It may be worth it to do periodic tests comparing some tests like fio
> > and dbench (both of which phoronix supports) with and without ecryptfs.
> > However, these sorts of tests would be harder to use since we can't
> > meaningfully run them on VMs.
> >
> > The posix test suite, AIUI, tests other kernel features like shm, but
> > does not test proper posix file behavior.

Ah, but I just saw http://lwn.net/Articles/276617/.  That's promising.

> > The LTP testsuite is one we sould look at.  The fs testsuite has a set of
> > both fs stress tests (like growfiles) and correctness tests (like inode01).
> >
> > We also could write some testcases of our own.  I've started a list of
> > potential tests at http://wiki.ubuntu.com/ecryptfs-tests.  Please feel
> > free to add entries.  Perhaps at the next UDS we can have a session
> > going over the accumulated list, prioritizing, and setting a definite
> > implementation plan.  ("i.e. serge, go code it, based on ltp" :)
> >
> > My own recommendation would be that we start by writing a wrapper which
> > fetches ltp and runs the fs testsuite on ext4 and ecryptfs-on-ext4.  I'll
> > happily script that and package the script as ecryptfs-testsuite.  Then
> > we can move on to writing some of our own testcases.
> >
> > How does that sound to people?  If you feel phoronix testsuite should
> > also be done, then I'd like to hear from someone who can volunteer some
> > bare metal, long term, for repeated tests.
> 
> +1 for automated testing, of course.  Phoronix is good for long term
> benchmarking, noticing major regressions, etc.  But
> functional/correctness testing is more important to me as a user and
> one of the maintainers.

Agreed.

> See the end of the ecryptfs-setup-private script for a very basic set
> of read/write correctness tests.  We should extend that concept,

Yeah, cool, that's a starting point.  Thanks.

> reading and writing tens of thousands of files, large and small,
> hundreds of megabytes, and also dig through dmesg and the logs looking
> for ecryptfs errors, as those are starting to creep up more and more.

Hm, that starts to worry me.

The reason I wanted to compile a list of useful test cases and discuss
them before digging in, is that I've seen it happen too often where
a bunch of crappy tests are written which are testing side effects of
current implementation rather than important indicators of correctness,
and eventually the test suite becomes more a nuisance than a help.
Looking for kernel msgs worries me in that same way.

thanks,
-serge