dolfin team mailing list archive

[Anders Logg <logg@xxxxxxxxx>]

On Fri, Aug 15, 2008 at 09:56:38AM +0200, Niclas Jansson wrote:
> Anders Logg wrote:
> > On Thu, Aug 14, 2008 at 03:26:07PM +0200, Niclas Jansson wrote:
> >   
> >> The code could be fetched from
> >>
> >> http://www.csc.kth.se/~njansson/dolfin-2008-08-14.tar.bz2
> >>
> >> Requires PETSc and parmetis, (customLinkFlags = '-lmetis -lparmetis')
> >>
> >> DofMap::build don't construct a global dofmap. It reorders the map in
> >> order to minimize communication during assembly.
> >> The global dofmap is obtained from the ordinary tabulate_dofs call.
> >>
> >> Niclas
> >>     
> >
> > I think it looks good.
> >
> > As far as I understand, you build a global numbering of all mesh
> > entities (which may be different from the local numbering on each
> > processor), and then the (global parallel) local-to-global mapping
> > follows from tabulate_dofs just as usual.
> >
> > So, the difference is that you build a global numbering of the mesh
> > entities, and we wanted to build a global numbering of the dofs. The
> > only advantage I can see with our approach is that it may use less
> > memory, since we don't need to store an extra numbering scheme for all
> > mesh entities but this not a big deal.
> >
> > A few questions:
> >
> > 1. Is the above interpretation correct?
> >   
> Yes.
> 
> Another disadvantage with the global numbering scheme is the mesh 
> connectivity calculations (mesh.init in MeshRenumber).

Why is this a problem? As far as I understand, there are always two
different numberings of mesh entities, one local (same as we have
now) and one global. The local can be computed as usual and then the
global can be reconstructed from the local + the overlap.

(overlap = how the local pieces fit together)

> > 2. Is there a simple example that I can run to test. The code built
> > fine but I didn't find any demo.
> >   
> 
> The poisson, elasticity and nonlinearpoisson demos should work with the 
> following modifications
> 1) Meshes must be loaded from file.
> 2) plot(u) must be disabled.

ok.

> or fetch the following http://www.csc.kth.se/~njansson/poisson.tar.bz2
> Poisson with mesh refinement and load balancing.
> > 3. Does the partitioning require that one processor reads the entire mesh
> > and then sends it to ParMetis?
> >   
> No, the mesh is always distributed.

How does this work? Is the IO also parallel? I can't see any parallel
stuff in XMLMesh.cpp.

> > 4. Is the extra storage dynamic? If only vertices are needed (for P1
> > elements), then we only need to store extra vertex numbers.
> >   
> No, it always stores a global number for each mesh entity.

ok, this needs to be fixed but it shouldn't be hard. The DofMap knows
which entities are needed. This might be an argument in favor of
computing a parallel dof map from local dof maps rather than computing
a parallel mesh numbering from local mesh numberings.

> > 5. MeshRenumber seems specific to triangles and tets. Can it be done
> > without reference to specific entities with special cases put in
> > CellType?
> >   
> Maybe, the problem is to construct a key that could be used to uniquely 
> identify the entity.

Isn't (dim, index) enough? Or is this because you only have vertex
information in your overlap?

> But, since each entity could be seen as a set of vertices it shouldn't 
> be any problem.
> > 6. Does it work for assembly over interior facets (like in DG
> > methods)?
> >   
> I'm not sure, haven't tried any DG type problems.

I expect it to be problematic. For interior facet integrals, we need
to iterate over all interior facets in the mesh and for each facet
access its two adjacent cells. One of these may be on another
processor.

> > 7. Is it possible to make it work with SCOTCH (in addition to
> > ParMetis)?
> >   
> Yes, with some modification to MPIMeshCommunicator.

ok.

> > Then some suggestions:
> >
> > 1. I'd like to move the implementation of DofMap::build() to
> > DofMapBuilder (to simplify DofMap.cpp).
> >
> > 2. Function names should be fooBar(), not foo_bar().
> >   
> I recently found doc/misc/policy :)
> > 3. There needs to be some #ifdef HAS_PARMETIS so it may be built
> > without Parmetis.
> >
> > 4. I'd like to add a new class GlobalNumbering (to replace
> > MeshDistributedData) that holds the global numbering scheme.
> >
> > The Mesh class can have a pointer to a GlobalNumbering object which is
> > 0 by default so no extra data (or at least not more than 4 bytes) is
> > stored when not running in parallel.
> >
> > Then we can add a function MeshEntity::number() which returns the same
> > as index() if GlobalNumbering is 0. Otherwise, it returns what is
> > stored in GlobalNumbering.
> >
> > GlobalNumbering can have an array of MeshFunctions, one for each
> > topological dimension, that maps the local entity indices to their
> > global numbers.
> >
> > Thus, a MeshEntity will have two functions index() and number().
> > These will return the same value in sequential and possibly different
> > values in parallel.
> >
> > Let's await some more comments and then get started. It would be nice
> > to get it in small patches to give us an opportunity to comment/edit.
> > Adding GlobalNumbering and MeshEntity::number() would be a good start.
> >
> Sounds good, However I think the {T,S,F,V,O} design mentioned by Johan 
> is more efficient.

These are different issues. One issue is how to represent the overlap,
however we choose to do that, either {T,S,F,V,O} or just {T, S, F}.
The other issue is how we store either the global mesh numbering or
the global dof map. We need to discuss both.

I'd like to continue this discussion for some time before we jump on
to either option.

1. Do we want to compute a global mesh numbering and then let the
global dof map follow as usual, or do we want to have just a local
mesh numbering and then compute a global dof map from the local dof
maps + the overlap?

In either case, the resulting global dof map can be renumbered to
reduce bandwidth/communication.

2. How do we want to represent the overlap? Is it enough to store it
as {T, S, F} or do we need more? Do we need a class Overlap that
stores the overlap?

-- 
Anders

Attachment: signature.asc
Description: Digital signature