ffc team mailing list archive

["Johannes Ring" <johannr@xxxxxxxxx>]

On Tue, March 31, 2009 10:35, Garth N. Wells wrote:
>
>
> Johannes Ring wrote:
>> On Tue, March 31, 2009 10:23, Garth N. Wells wrote:
>>>
>>> Johan Hake wrote:
>>>> On Tuesday 31 March 2009 09:45:33 Johannes Ring wrote:
>>>>> On Mon, March 30, 2009 20:19, Johan Hake wrote:
>>>>>> On Monday 30 March 2009 20:06:04 Anders Logg wrote:
>>>>>>> The buildbot is not responding at the moment so I can't check the
>>>>>>> status.
>>>>>> All is green except:
>>>>>>
>>>>>> The DOLFIN macbot how is complaining about the trilinos demo and the
>>>>>> DOLFIN
>>>>>> linux64-exp which complains about:
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>   File "./demo.py", line 8, in <module>
>>>>>>     from dolfin import *
>>>>>>
>>>>>> File
>>>>>> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/__init__.py",
>>>>>> line 16, in <module>
>>>>>>     from assemble import *
>>>>>>
>>>>>> File
>>>>>> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/assemble.py",
>>>>>> line 25, in <module>
>>>>>>     import cpp
>>>>>>   File
>>>>>> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/cpp.py",
>>>>>> line 25, in <module>
>>>>>>     import _cpp
>>>>>> ImportError:
>>>>>> /work/jhbuildbot/local/lib/openmpi/mca_paffinity_linux.so:
>>>>>> undefined symbol: mca_base_param_reg_int
>>>>>>
>>>>>> I know Johannes has tried to get into this but as far as I know with
>>>>>> no
>>>>>> success.
>>>>> This is a problem with PyDOLFIN and Open MPI 1.3. We (me and Johan)
>>>>> had
>>>>> success with a small hack this morning. By adding
>>>>>
>>>>>   import ctypes
>>>>>   ctypes.CDLL('libmpi.so', ctypes.RTLD_GLOBAL)
>>>>>
>>>>> before loading the cpp module in site-packages/dolfin/__init__.py the
>>>>> Python poisson demo ran just fine. Alternatively one should be able
>>>>> to
>>>>> use
>>>>> this instead:
>>>>>
>>>>>   import dl
>>>>>   import sys
>>>>>   flags = sys.getdlopenflags()
>>>>>   sys.setdlopenflags(flags | dl.RTLD_GLOBAL)
>>>>>
>>>>> but because of a bug in the python2.5 package in Hardy (missing dl
>>>>> module)
>>>>> we couldn't test this.
>>>>>
>>>>> It might also be possible to do this in C++ directly. Any suggestion
>>>>> on
>>>>> the best way to fix this issue? See also:
>>>>>
>>>>> http://fenics.org/pipermail/deb-dev/2009-March/000210.html
>>>>> http://www.open-mpi.org/faq/?category=running#loading-libmpi-dynamically
>>>>>
>>>>> Open MPI 1.3 is in Debian unstable so it would be great if we could
>>>>> fix
>>>>> this before the release.
>>>> The point is that the shared mpi library needs to be loaded using the
>>>> flag
>>>> RTLD_GLOBAL. Otherwise the loaded mpi libraries will be loaded into
>>>> private
>>>> namespaces causing the undefined symbol link error.
>>>>
>>>> I do not understand why this is not a problem when only using DOLFIN
>>>> and
>>>> not
>>>> PyDOLFIN. Probably the mpi library is only loaded one time to the
>>>> right
>>>> namespace? Well that's how deep my understanding is of this :P
>>>>
>>> Things went sour when DOLFIN was compiled with Trilinos and Trilinos
>>> had
>>> MPI enabled. Could this be related?
>>
>> Probably not. Trilinos is built with --disable-mpi on all the
>> buildslaves.
>>
>
> I mean that if this issue is resolved, using --enable-mpi for Trilinos
> will work.

OK, I see. Yes, that might be true.

Johannes

> Garth
>
>> Johannes
>>
>>> Garth
>>>
>>>> The hack consist of:
>>>> Before we load the compiled c++ module, i.e., the _cpp.do, we load the
>>>> shared
>>>> mpi library using the above mentioned calls. The library stays in
>>>> memory
>>>> and
>>>> is not loaded a second time when the _cpp.so module is loaded.
>>>>
>>>> This is my naive view of the problem and the why the hack works. It
>>>> could
>>>> probably be solved in a more gracious manner, e.g. before
>>>> PetscInitialize is
>>>> called in DOLFIN or maybe even better on the PETSc side?
>>>>
>>>> Cheers!
>>>>
>>>> Johan
>>>> _______________________________________________
>>>> FFC-dev mailing list
>>>> FFC-dev@xxxxxxxxxx
>>>> http://www.fenics.org/mailman/listinfo/ffc-dev
>>>
>>>
>>
>>
>
>
>