dolfin team mailing list archive

[Anders Logg <logg@xxxxxxxxx>]

On Sat, Jan 05, 2008 at 03:09:25PM +0100, Mads Hoel wrote:
> On Fri, 04 Jan 2008 21:22:16 +0100, Anders Logg <logg@xxxxxxxxx> wrote:
> 
> > On Fri, Jan 04, 2008 at 08:20:49PM +0000, Garth N. Wells wrote:
> >>
> >>
> >> Anders Logg wrote:
> >> > On Fri, Jan 04, 2008 at 08:12:48PM +0000, Garth N. Wells wrote:
> >> >>
> >> >> Anders Logg wrote:
> >> >>> On Fri, Jan 04, 2008 at 06:39:39PM +0000, Garth N. Wells wrote:
> >> >>>> Anders Logg wrote:
> >> >>>>> On Fri, Jan 04, 2008 at 06:25:11PM +0000, Garth N. Wells wrote:
> >> >>>>>> Anders Logg wrote:
> >> >>>>>>> On Fri, Jan 04, 2008 at 06:12:32PM +0000, Garth N. Wells wrote:
> >> >>>>>>>> Anders Logg wrote:
> >> >>>>>>>>> On Fri, Jan 04, 2008 at 05:50:53PM +0000, Garth N. Wells  
> >> wrote:
> >> >>>>>>>>>> Anders Logg wrote:
> >> >>>>>>>>>>> On Fri, Jan 04, 2008 at 10:25:35AM -0600, Matthew Knepley  
> >> wrote:
> >> >>>>>>>>>>>> On Jan 4, 2008 10:22 AM, Garth N. Wells <gnw20@xxxxxxxxx>  
> >> wrote:
> >> >>>>>>>>>>>>> We have a problem at the moment when using PETSc related  
> >> to conflicts
> >> >>>>>>>>>>>>> between dolfin::MPIManager and dolfin::PETScManger as to  
> >> who should
> >> >>>>>>>>>>>>> initialise and finalise MPI. The difficulty is that we  
> >> can't control the
> >> >>>>>>>>>>>>> order in which MPIManager and PETScManager are destroyed.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Given that we will probably have more 'manager' classes  
> >> in the future
> >> >>>>>>>>>>>>> (e.g. for Trilinos), what is the best approach? Some  
> >> possibilities are
> >> >>>>>>>>>>>>> one 'Manager' class that does the lot, or a  
> >> SingletonManager which can
> >> >>>>>>>>>>>>> control the order in which singleton manager classes are  
> >> destroyed. Ideas?
> >> >>>>>>>>>>>> When using multiple packages with  MPI, you should always  
> >> have a single MPI
> >> >>>>>>>>>>>> manage class. If MPI is already initialized when PETSc is  
> >> initialized,
> >> >>>>>>>>>>>> it won't mess
> >> >>>>>>>>>>>> with MPI (and won't finalize it either).
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>   Matt
> >> >>>>>>>>>>> Good, if it's always the case that PETSc does not finalize  
> >> when it has
> >> >>>>>>>>>>> not initialized, then maybe we just need to do the same?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> As we have implemented it, MPIManager checks if someone  
> >> else (maybe
> >> >>>>>>>>>>> PETSc, maybe someone else) has already initialized MPI and  
> >> in that
> >> >>>>>>>>>>> case does not initialize it. Then it should also assume  
> >> that someone
> >> >>>>>>>>>>> else will finalize it.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> We can just add a bool member initialized_here and then do
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>   if (initialized_here)
> >> >>>>>>>>>>>     MPIManager::finalize();
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> in the destructor of MPIManager.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Would that help?
> >> >>>>>>>>>>>
> >> >>>>>>>>>> Unfortunately not.
> >> >>>>>>>>>>
> >> >>>>>>>>>> The problem is that MPIManager might finalize MPI before  
> >> PETSc has been
> >> >>>>>>>>>> finalised. If MPI is initialised before PETSc, then PETSc  
> >> should be
> >> >>>>>>>>>> finalised before MPI is finalised. The problem is that we  
> >> have no
> >> >>>>>>>>>> control over the order in which MPIManager and PETScManager  
> >> are destroyed.
> >> >>>>>>>>> ok.
> >> >>>>>>>>>
> >> >>>>>>>>>> I'm thinking of a singleton class DolfinManager which is  
> >> responsible for
> >> >>>>>>>>>> the creation and destruction of various managers in the  
> >> appropriate order.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Garth
> >> >>>>>>>>> Perhaps it would be better to have a single class that takes  
> >> care of
> >> >>>>>>>>> all initializations (rather than having a class that takes  
> >> care of
> >> >>>>>>>>> manager classes) to keep things simple?
> >> >>>>>>>>>
> >> >>>>>>>> ok.
> >> >>>>>>>>
> >> >>>>>>>>> We could put a class named Init (for example) in  
> >> src/kernel/main/
> >> >>>>>>>>> with some static functions:
> >> >>>>>>>>>
> >> >>>>>>>>>   static void init(int argc, char* argv[]);
> >> >>>>>>>>>
> >> >>>>>>>>>   static void initPETSc();
> >> >>>>>>>>>   static void initPETSc(int argc, char* argv[]);
> >> >>>>>>>>>
> >> >>>>>>>>>   static void initMPI();
> >> >>>>>>>>>
> >> >>>>>>>>> We can then remove init.cpp, init.h and also PETScManager.
> >> >>>>>>>>>
> >> >>>>>>>> ok.
> >> >>>>>>>>
> >> >>>>>>>>> MPIManager can be renamed to MPI and just contain MPI utility
> >> >>>>>>>>> functions (like everything in MPIManager now except  
> >> init/finalize).
> >> >>>>>>>>>
> >> >>>>>>>> What about calling it DolfinManager as it won't be strictly  
> >> for MPI?
> >> >>>>>>>> Without MPI, we still need to initialise PETSc.
> >> >>>>>>>>
> >> >>>>>>>> Garth
> >> >>>>>>> The thing I suggest calling MPI is strictly for MPI (the things
> >> >>>>>>> currently in MPIManager except init/finalize).
> >> >>>>>>>
> >> >>>>>> Do you still propose having a class MPI?
> >> >>>>> Yes, and it contains the following things:
> >> >>>>>
> >> >>>>>     /// Return proccess number
> >> >>>>>     static uint processNumber();
> >> >>>>>
> >> >>>>>     /// Return number of processes
> >> >>>>>     static uint numProcesses();
> >> >>>>>
> >> >>>>>     /// Determine whether we should broadcast (based on current  
> >> parallel policy)
> >> >>>>>     static bool broadcast();
> >> >>>>>
> >> >>>>>     /// Determine whether we should receive (based on current  
> >> parallel policy)
> >> >>>>>     static bool receive();
> >> >>>>>
> >> >>>>>>> I agree the class that manages initialization should be called
> >> >>>>>>> something neutral. I'm not very fond of "DolfinManager" since  
> >> (1)
> >> >>>>>>> maybe it should then be DOLFINManager (which is maybe not very  
> >> nice)
> >> >>>>>>> and (2) it is not something that manages DOLFIN.
> >> >>>>>>>
> >> >>>>>> Suggestions? SubSystemManager?
> >> >>>>> Sounds good.
> >> >>>>>
> >> >>>>> For simplicity, we could also have a class named PETSc with a  
> >> static
> >> >>>>> init() function that would just call SubSystemManager. Since we  
> >> need
> >> >>>>> to call this in all PETSc data structures, it's convenient if we  
> >> can
> >> >>>>> write
> >> >>>>>
> >> >>>>>   PETSc::init()
> >> >>>>>
> >> >>>>> rather than
> >> >>>>>
> >> >>>>>   SubSystemManager::initPETSc();
> >> >>>>>
> >> >>>>> Similarly, we can have an init() function in the class MPI that  
> >> calls
> >> >>>>> SubSystemManager::initMPI().
> >> >>>>>
> >> >>>>> So three classes:
> >> >>>>>
> >> >>>>>    SubSystemManager: singleton manager of all subsystem with some
> >> >>>>>    logic for the order of initialization/finalization
> >> >>>>>
> >> >>>>>    MPI: MPI convenience class
> >> >>>>>
> >> >>>>>    PETSc: PETSc convenience class
> >> >>>>>
> >> >>>>>
> >> >>>> OK. I'll add something. We could just let
> >> >>>>
> >> >>>>    SubSystemManager::init();
> >> >>>>
> >> >>>> take care of all the initialisation.
> >> >>>>
> >> >>>> Garth
> >> >>> ok, but shouldn't we be able to initialize one but not the other  
> >> (of MPI
> >> >>> and PETSc)?
> >> >>>
> >> >> Yes. We have compiler flags that tell what is needed.
> >> >>
> >> >> Garth
> >> >
> >> > I mean even if one has both MPI and PETSc installed, then one might
> >> > want to use MPI (for the mesh) without initializing PETSc, and one
> >> > might want to use PETSc without initializing MPI (if possible).
> >> >
> >>
> >> Yes, it is desirable to be able to do this. In practice at the moment
> >> this is pretty much determined by the compiler flags. Once we have a
> >> class SubSystemManager, we will be pretty flexible in what we can do.
> >>
> >> Garth
> >
> > ok.
> >
> 
> What about designing the classes that use PETSc, MPI or some other  
> library, needing initialization call the corresponding initialization  
> function of the SubSystemManager. For instance:
> 
> class PETScObject {
> 	public:
> 	PETScObject()
> 	{
> 		SubSystemManager.initPETSc();
> 	}
> 
> 	~PETScObject()
> 	{
> 		SubSystemManager.finalizePETSc();
> 	}
> };
> 
> class PETScMat : public PETScObject
> {
> 	...
> };
> 
> So that once I construct any PETSc object, and not before that, PETSc gets  
> initialized.

Yes, that's how we do it already. PETScMatrix, PETScVector etc call

  PETScManager::init();

in their constructors. But we don't have a common base class
PETScObject that takes care of the initialization, but it seems like a
good idea to add. Then we don't need the PETSc class I suggested
yesterday (only SubSystemManager, PETScObject and MPI).

Garth?

-- 
Anders