← Back to team overview

yellow team mailing list archive

Re: Updated the charm to serve Juju GUI releases. (issue 6977043)

 

On 2012/12/20 03:32:06, gary.poster wrote:

> The unit tests all passed just fine, for what it is worth.

Cool.

19:30:10,417 jitsu.test:INFO Running unit test: deploy.test
> test_api_agent (__main__.DeployTest) ... ok
> test_branch_source (__main__.DeployTest) ... ERROR
> test_customized_api_port (__main__.DeployTest) ... ERROR
> test_staging (__main__.DeployTest) ... ERROR
> test_api_agent (__main__.DeployToTest) ... 2012-12-19 20:10:10,421
> jitsu.test:WARNING Error running unit test deploy.test: 124

> I have a vague suspicion that your branch of the charm doesn't like
installing
> on top of itself, but there could be many other explanations.

It seems so looking at this output. This is really weird because the
tests always
pass when I run them. The last test in that output seems to be truncated
by the
timeout command used by jitsu test.

> I really want to know what you do to debug these functional tests.  I
did not
> get pdbs to work locally (stdout/stdin are not passed through for this
to work,
> AFAICT), and the logs collected by jitsu test left a lot to be
desired.


I usually run jitsu test in a shell:
JUJU_REPOSITORY=/home/frankban/devel/juju/store/ jitsu test juju-gui
--timeout 60m --logdir /home/frankban/devel/juju/guicharm/log/
In parallel, in another shell, I run "juju debug-log", so that I can see
the output of all the hooks while they are run.

> I wanted to investigate the charm log, but we don't get that from
jitsu test.

The missing charm log exposes a problem. To simplify, here is what jitsu
test does:

for each *.test file in tests/:
   bootstrap a juju env
   execute the file (using the shell command "timeout --kill-after 2m
TIMEOUT file"
   for each machine in the environment:
     gather logs using rsync: everything in /var/log/juju and
/var/lib/juju/units/*/charm.log
   destroy the environment

In our tests, unittest.TestCase.tearDown destroys the juju-gui service.
This allows us to reuse the same machine in subsequent tests, where we
usually re-deploy
the service, testing different configurations. And this is a good idea
IMHO.
But this also raises a problem: "destroy service" removes the service
directory in
/var/lib/juju/units/, and that directory includes charm.log.

Some ideas:

1) jitsu test seems to suggest you to run tests in separate files. This
would solve
the missing charm.log problem, but IMHO it's ugly:
- we can no longer use the --no-bootstrap option: we still need to
re-deploy the charm
   multiple times, and we need a machine in a clean state;
- even if we can decrease the timeout for each test, the overall time
elapsed will be
   a lot increased: for each test, we'd have to wait for the juju env to
be bootstrapped,
   and for one or two machines to be started on ec2;
- re-deploying the charm in the same machine adds value to our tests,
because we also
   check that our charm correctly cleans things up (e.g. services).
AFAIK, juju reuses
   machines, at least when the provider is ec2, so this represents a real
use case;
- IMHO the functional tests, the way they are written now, are compact
and easy to read:
   splitting them into different files doesn't seem something that should
be imposed
   by the testing framework.

2) we investigate the possibility of patching jitsu test so that logs
are retrieved
    when a service is destroyed. jitsu test is synchronous, but maybe we
could suggest
    something like "jitsu destroy-service", and use that in out tests.

3) workaround: we save charm.log in DeployTestMixin.tearDown, right
before destroying
   the service.
- we duplicate a functionality already present in the test framework,
but...
- the implementation should be straightforward.

> Maybe we ought to incorporate that into our own test infrastructure?
I have a
> weekly retrospective card to discuss this, and see if you or anyone
else have
> any good techniques, or ideas about what we can do to improve jitsu
test for
> debugging.

Thank you.



https://codereview.appspot.com/6977043/

-- 
https://code.launchpad.net/~frankban/charms/precise/juju-gui/bug-1088618-serve-releases/+merge/140745
Your team Juju GUI Hackers is requested to review the proposed merge of lp:~frankban/charms/precise/juju-gui/bug-1088618-serve-releases into lp:~juju-gui/charms/precise/juju-gui/trunk.


References