| Thread Previous • Date Previous • Date Next • Thread Next |
On 06/04/2012 04:00 PM, Kevin L. Mitchell wrote:
Today I've noticed some significant problems with nova's test suite leaving literally hundreds of python processes out. I'm guessing that this has to do with the unit tests for the multiprocess patch, which was just approved. This could be causing problems with jenkins, too… Anybody have any other insights?
Yes, several Jenkins slaves have been taken out by running nova unit tests. The one that I am able to log into seems to be continuously respawing python processes. Other slaves are inaccessible due to having exhausted their RAM.
I note that all of the tests run after that change merged carry this warning notice from Jenkins:
Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information
So I think it's fair to say that Jenkins corroborates your suspicion that change introduced a problem with leaking processes.
This is affecting any Jenkins slave that the nova unit tests job runs on, which in turn affects jobs for unrelated projects that happen to later run on that slave.
In addition to correcting this problem, I believe we should add a build step to Jenkins to ensure that all of the test processes have terminated correctly.
-Jim
| Thread Previous • Date Previous • Date Next • Thread Next |
