← Back to team overview

dolfin team mailing list archive

Re: Buildbot problems

 

On Fri, Oct 21, 2011 at 12:29 PM, Anders Logg <logg@xxxxxxxxx> wrote:
> On Fri, Oct 21, 2011 at 12:08:58PM +0200, Harish Narayanan wrote:
>> On 10/21/11 11:43 AM, Anders Logg wrote:
>> > On Fri, Oct 21, 2011 at 11:27:57AM +0200, Anders Logg wrote:
>> >> On Fri, Oct 21, 2011 at 10:41:57AM +0200, Martin Sandve Alnæs wrote:
>> >>> If this cannot be fixed quickly, can you please revert the commit that
>> >>> caused it?
>> >>>
>> >>> In general, quickly fixing or reverting when stuff is broken is the
>> >>> only way we can keep a consistently green buildbot. Having the
>> >>> buildbots in a red state over time hinders further development in
>> >>> maintrunk and/or hides problems caused by additional commits. This
>> >>> time the build problem did hide test breakage caused by both me and
>> >>> Anders, and the fixes were then much delayed. I think it is worth
>> >>> considering requiring quick revert of all developers (I know I've
>> >>> sinned there myself).
>> >>
>> >> Sorry for breaking the buildbot, but I've had trouble with my own
>> >> personal buildbot not running at all so I couldn't test there first.
>> >> (And the thing that currently breaks works fine here with me.)
>> >>
>> >> I'm not sure reverting now is a good idea since it is a big change
>> >> that is important for the book (cleanup of the Krylov methods and
>> >> preconditioner options).
>> >>
>> >> It looks to me like something needs to be updated or rebuilt on the
>> >> buildbot. The unit test that fails on the buildbot (quadrature on cut
>> >> cells using CGAL) works fine here, and also for Andre. Maybe the
>> >> buildbot just needs a new installation of CGAL and/or boost?
>> >
>> > The error message seems to indicate a problem with boost::assign in
>> > combination with CGAL. Maybe somehow the inclusion of
>> >
>> >   boost/assign/list_of.hpp
>> >
>> > in uBLASFactory.h and MTL4Factory.h propagated to the unit test and
>> > that breaks with CGAL.
>> >
>> > I've worked around the inclusion of this header file now (only
>> > included in .cpp files). Can those of you for whom this breaks try
>> > again?
>>
>> It now compiles, but I receive the following error when attempting to
>> run the test.
>>
>> megatr0n:cpp harish$ ./test_quadrature
>> [megatr0n:74469] *** Process received signal ***
>> [megatr0n:74469] Signal: Segmentation fault: 11 (11)
>> [megatr0n:74469] Signal code:  (0)
>> [megatr0n:74469] Failing at address: 0x0
>> [megatr0n:74469] [ 0] 2   libsystem_c.dylib
>> 0x00007fff92040cfa _sigtramp + 26
>> [megatr0n:74469] [ 1] 3   ???
>> 0xc000000000000000 0x0 + 13835058055282163712
>> [megatr0n:74469] [ 2] 4   test_quadrature
>> 0x0000000108ac1a22
>> _ZN4CGAL18Box_intersection_d21modified_two_way_scanIN9__gnu_cxx17__normal_iteratorIPNS_7Nef_boxINS_13SNC_decoratorINS_13SNC_structureINS_5EpeckENS_17SNC_indexed_itemsEbEEEEEESt6vectorISB_SaISB_EEEESG_NS_36binop_intersection_test_segment_treeISA_E24Bop_edge0_edge1_callbackINS_16Binary_operationIS9_E22Intersection_call_backISA_NS_16Nef_polyhedron_3IS7_S8_bE2ORENS_18ID_support_handlerIS8_SA_EEEEEENS0_18Predicate_traits_dINS0_12Box_traits_dISB_EELb1EEEEEvT_SY_T0_SZ_T1_T2_ib
>> + 162
>> [megatr0n:74469] [ 3] 5   test_quadrature
>> 0x0000000108b0332b
>> _ZN4CGAL18Box_intersection_d12segment_treeIN9__gnu_cxx17__normal_iteratorIPNS_7Nef_boxINS_13SNC_decoratorINS_13SNC_structureINS_5EpeckENS_17SNC_indexed_itemsEbEEEEEESt6vectorISB_SaISB_EEEESG_NS_36binop_intersection_test_segment_treeISA_E24Bop_edge0_edge1_callbackINS_16Binary_operationIS9_E22Intersection_call_backISA_NS_16Nef_polyhedron_3IS7_S8_bE2ORENS_18ID_support_handlerIS8_SA_EEEEEEdNS0_18Predicate_traits_dINS0_12Box_traits_dISB_EELb1EEEEEvT_SY_T0_SZ_T2_S10_T1_T3_lib
>> + 1147
>> [megatr0n:74469] [ 4] 6   test_quadrature
>> 0x0000000108b05ef5
>> _ZN4CGAL36binop_intersection_test_segment_treeINS_13SNC_decoratorINS_13SNC_structureINS_5EpeckENS_17SNC_indexed_itemsEbEEEEEclINS_16Binary_operationIS5_E22Intersection_call_backIS6_NS_16Nef_polyhedron_3IS3_S4_bE2ORENS_18ID_support_handlerIS4_S6_EEEEEEvRT_SJ_RS5_SK_
>> + 969
>> [megatr0n:74469] [ 5] 7   test_quadrature
>> 0x0000000108b073be
>> _ZN4CGAL16Binary_operationINS_13SNC_structureINS_5EpeckENS_17SNC_indexed_itemsEbEEEclINS_16Nef_polyhedron_3IS2_S3_bE2OREEEvPNS_17SNC_point_locatorINS_13SNC_decoratorIS4_EEEERKS4_PKSD_SG_SI_RKT_
>> + 3536
>> [megatr0n:74469] [ 6] 8   test_quadrature
>> 0x0000000108b077e9
>> _ZNK4CGAL16Nef_polyhedron_3INS_5EpeckENS_17SNC_indexed_itemsEbE4joinERKS3_
>> + 343
>> [megatr0n:74469] [ 7] 9   test_quadrature
>> 0x0000000108b07855
>> _ZN4CGAL16Nef_polyhedron_3INS_5EpeckENS_17SNC_indexed_itemsEbEpLERKS3_ + 29
>> [megatr0n:74469] [ 8] 10  test_quadrature
>> 0x0000000108b07947
>> _ZN10BaryCenter24add_disjoint_polyhedronsESt6vectorIiSaIiEERS0_IN4CGAL16Nef_polyhedron_3INS3_5EpeckENS3_17SNC_indexed_itemsEbEESaIS7_EERS0_IdSaIdEERS0_IN6dolfin5PointESaISF_EE
>> + 133
>> [megatr0n:74469] [ 9] 11  test_quadrature
>> 0x0000000108b09d2e _ZN10BaryCenter21testSimplePolyhedronsEv + 8758
>> [megatr0n:74469] [10] 12  libcppunit-1.12.1.dylib
>> 0x000000010a336990 _ZNK7CppUnit21TestCaseMethodFunctorclEv + 38
>> [megatr0n:74469] [11] 13  libcppunit-1.12.1.dylib
>> 0x000000010a32e5f4
>> _ZN7CppUnit16DefaultProtector7protectERKNS_7FunctorERKNS_16ProtectorContextE
>> + 36
>> [megatr0n:74469] [12] 14  libcppunit-1.12.1.dylib
>> 0x000000010a334721 _ZNK7CppUnit14ProtectorChain14ProtectFunctorclEv + 25
>> [megatr0n:74469] [13] 15  libcppunit-1.12.1.dylib
>> 0x000000010a3343b0
>> _ZN7CppUnit14ProtectorChain7protectERKNS_7FunctorERKNS_16ProtectorContextE
>> + 484
>> [megatr0n:74469] [14] 16  libcppunit-1.12.1.dylib
>> 0x000000010a33bef0
>> _ZN7CppUnit10TestResult7protectERKNS_7FunctorEPNS_4TestERKSs + 62
>> [megatr0n:74469] [15] 17  libcppunit-1.12.1.dylib
>> 0x000000010a336595 _ZN7CppUnit8TestCase3runEPNS_10TestResultE + 309
>> [megatr0n:74469] [16] 18  libcppunit-1.12.1.dylib
>> 0x000000010a336e15
>> _ZN7CppUnit13TestComposite15doRunChildTestsEPNS_10TestResultE + 81
>> [megatr0n:74469] [17] 19  libcppunit-1.12.1.dylib
>> 0x000000010a336d41 _ZN7CppUnit13TestComposite3runEPNS_10TestResultE + 37
>> [megatr0n:74469] [18] 20  libcppunit-1.12.1.dylib
>> 0x000000010a336e15
>> _ZN7CppUnit13TestComposite15doRunChildTestsEPNS_10TestResultE + 81
>> [megatr0n:74469] [19] 21  libcppunit-1.12.1.dylib
>> 0x000000010a336d41 _ZN7CppUnit13TestComposite3runEPNS_10TestResultE + 37
>> [megatr0n:74469] [20] 22  libcppunit-1.12.1.dylib
>> 0x000000010a33be80 _ZN7CppUnit10TestResult7runTestEPNS_4TestE + 40
>> [megatr0n:74469] [21] 23  libcppunit-1.12.1.dylib
>> 0x000000010a33e300 _ZN7CppUnit10TestRunner3runERNS_10TestResultERKSs + 54
>> [megatr0n:74469] [22] 24  test_quadrature
>> 0x0000000108a14682 main + 210
>> [megatr0n:74469] [23] 25  test_quadrature
>> 0x0000000108a142f4 start + 52
>> [megatr0n:74469] [24] 26  ???
>> 0x0000000000000001 0x0 + 1
>> [megatr0n:74469] *** End of error message ***
>> Segmentation fault: 11
>>
>> Harish
>
> Strange. It works for Johannes now so let's see what the buildbot
> says.

The buildbot liked the fix.

There are now a couple of issues with the OS X buildbot:

1. The DirichletBC Python test fails in parallel with the following error:

======================================================================
FAIL: test_get_values (__main__.DirichletBCTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./DirichletBC.py", line 66, in test_get_values
    self.assertTrue(dofs[i] in bc_values)
AssertionError: False is not true

2. The OS X buildbot times out on the eigenvalue C++ demo when run in
parallel. If I run this demo manually, I get the error below (before
it hangs).

Johannes

buildbot@buildbotmac:cpp$ mpirun -np 3 ./demo_eigenvalue
Small graph
Process 2: Partitioned mesh, edge cut is 69.
Process 0: Partitioned mesh, edge cut is 69.
Process 1: Partitioned mesh, edge cut is 69.
Largest eigenvalue: 10.0279
Process 1: Plotting u (a function).
Process 1: *** Warning: Plotting disabled when running in parallel;
see https://bugs.launchpad.net/dolfin/+bug/427534
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Invalid argument!
[1]PETSC ERROR: Vector is not ghosted!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Unknown Name on a darwin10. named
buildbotmac.simula.no by buildbot Fri Oct 21 13:55:44 2011
[1]PETSC ERROR: Libraries linked from /Users/buildbot/buildbots/main/local/lib
[1]PETSC ERROR: Configure run at Thu Oct 13 10:31:00 2011
[1]PETSC ERROR: Configure options COPTFLAGS=-O2
--prefix=/Users/buildbot/buildbots/main/local --with-debugging=0
--with-shared=1 --with-clanguage=cxx --with-fc=0 --with-parmetis=1
--with-parmetis-dir=/Users/buildbot/buildbots/main/local
--download-umfpack=1 --download-spooles=1
--prefix=/Users/buildbot/buildbots/main/local
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: VecGhostUpdateBegin() line 453 in src/vec/vec/impls/mpi/pbvec.c
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Invalid argument!
[1]PETSC ERROR: Vector is not ghosted!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Unknown Name on a darwin10. named
buildbotmac.simula.no by buildbot Fri Oct 21 13:55:44 2011
[1]PETSC ERROR: Libraries linked from /Users/buildbot/buildbots/main/local/lib
[1]PETSC ERROR: Configure run at Thu Oct 13 10:31:00 2011
[1]PETSC ERROR: Configure options COPTFLAGS=-O2
--prefix=/Users/buildbot/buildbots/main/local --with-debugging=0
--with-shared=1 --with-clanguage=cxx --with-fc=0 --with-parmetis=1
--with-parmetis-dir=/Users/buildbot/buildbots/main/local
--download-umfpack=1 --download-spooles=1
--prefix=/Users/buildbot/buildbots/main/local
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: VecGhostUpdateEnd() line 515 in src/vec/vec/impls/mpi/pbvec.c
Largest eigenvalue: 10.0279
Process 0: Plotting u (a function).
Process 0: *** Warning: Plotting disabled when running in parallel;
see https://bugs.launchpad.net/dolfin/+bug/427534
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Vector is not ghosted!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Unknown Name on a darwin10. named
buildbotmac.simula.no by buildbot Fri Oct 21 13:55:44 2011
[0]PETSC ERROR: Libraries linked from /Users/buildbot/buildbots/main/local/lib
[0]PETSC ERROR: Configure run at Thu Oct 13 10:31:00 2011
[0]PETSC ERROR: Configure options COPTFLAGS=-O2
--prefix=/Users/buildbot/buildbots/main/local --with-debugging=0
--with-shared=1 --with-clanguage=cxx --with-fc=0 --with-parmetis=1
--with-parmetis-dir=/Users/buildbot/buildbots/main/local
--download-umfpack=1 --download-spooles=1
--prefix=/Users/buildbot/buildbots/main/local
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecGhostUpdateBegin() line 453 in src/vec/vec/impls/mpi/pbvec.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Vector is not ghosted!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Unknown Name on a darwin10. named
buildbotmac.simula.no by buildbot Fri Oct 21 13:55:44 2011
[0]PETSC ERROR: Libraries linked from /Users/buildbot/buildbots/main/local/lib
[0]PETSC ERROR: Configure run at Thu Oct 13 10:31:00 2011
[0]PETSC ERROR: Configure options COPTFLAGS=-O2
--prefix=/Users/buildbot/buildbots/main/local --with-debugging=0
--with-shared=1 --with-clanguage=cxx --with-fc=0 --with-parmetis=1
--with-parmetis-dir=/Users/buildbot/buildbots/main/local
--download-umfpack=1 --download-spooles=1
--prefix=/Users/buildbot/buildbots/main/local
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecGhostUpdateEnd() line 515 in src/vec/vec/impls/mpi/pbvec.c
Largest eigenvalue: 10.0279
[2]PETSC ERROR: --------------------- Error Message
------------------------------------
[2]PETSC ERROR: Invalid argument!
[2]PETSC ERROR: Vector is not ghosted!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Unknown Name on a darwin10. named
buildbotmac.simula.no by buildbot Fri Oct 21 13:55:44 2011
[2]PETSC ERROR: Libraries linked from /Users/buildbot/buildbots/main/local/lib
[2]PETSC ERROR: Configure run at Thu Oct 13 10:31:00 2011
[2]PETSC ERROR: Configure options COPTFLAGS=-O2
--prefix=/Users/buildbot/buildbots/main/local --with-debugging=0
--with-shared=1 --with-clanguage=cxx --with-fc=0 --with-parmetis=1
--with-parmetis-dir=/Users/buildbot/buildbots/main/local
--download-umfpack=1 --download-spooles=1
--prefix=/Users/buildbot/buildbots/main/local
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: VecGhostUpdateBegin() line 453 in src/vec/vec/impls/mpi/pbvec.c
[2]PETSC ERROR: --------------------- Error Message
------------------------------------
[2]PETSC ERROR: Invalid argument!
[2]PETSC ERROR: Vector is not ghosted!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Unknown Name on a darwin10. named
buildbotmac.simula.no by buildbot Fri Oct 21 13:55:44 2011
[2]PETSC ERROR: Libraries linked from /Users/buildbot/buildbots/main/local/lib
[2]PETSC ERROR: Configure run at Thu Oct 13 10:31:00 2011
[2]PETSC ERROR: Configure options COPTFLAGS=-O2
--prefix=/Users/buildbot/buildbots/main/local --with-debugging=0
--with-shared=1 --with-clanguage=cxx --with-fc=0 --with-parmetis=1
--with-parmetis-dir=/Users/buildbot/buildbots/main/local
--download-umfpack=1 --download-spooles=1
--prefix=/Users/buildbot/buildbots/main/local
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: VecGhostUpdateEnd() line 515 in src/vec/vec/impls/mpi/pbvec.c
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[2]PETSC
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
find memory corruption errors
[2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
[2]PETSC ERROR: to get more information on the crash.
[2]PETSC ERROR: --------------------- Error Message
------------------------------------
[2]PETSC ERROR: Signal received!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Unknown Name on a darwin10. named
buildbotmac.simula.no by buildbot Fri Oct 21 13:55:44 2011
[2]PETSC ERROR: Libraries linked from /Users/buildbot/buildbots/main/local/lib
[2]PETSC ERROR: Configure run at Thu Oct 13 10:31:00 2011
[2]PETSC ERROR: Configure options COPTFLAGS=-O2
--prefix=/Users/buildbot/buildbots/main/local --with-debugging=0
--with-shared=1 --with-clanguage=cxx --with-fc=0 --with-parmetis=1
--with-parmetis-dir=/Users/buildbot/buildbots/main/local
--download-umfpack=1 --download-spooles=1
--prefix=/Users/buildbot/buildbots/main/local
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
[buildbotmac.simula.no:15269] MPI_ABORT invoked on rank 2 in
communicator MPI_COMM_WORLD with errorcode 59
^Cmpirun: killing job...

mpirun noticed that job rank 0 with PID 15267 on node
buildbotmac.simula.no exited on signal 15 (Terminated).
2 additional processes aborted (not shown)


References