dolfin team mailing list archive

[Anders Logg <logg@xxxxxxxxx>]

On Mon, Aug 18, 2008 at 11:05:30AM +0200, Niclas Jansson wrote:
> Anders Logg wrote:
> >>> I think it looks good.
> >>>
> >>> As far as I understand, you build a global numbering of all mesh
> >>> entities (which may be different from the local numbering on each
> >>> processor), and then the (global parallel) local-to-global mapping
> >>> follows from tabulate_dofs just as usual.
> >>>
> >>> So, the difference is that you build a global numbering of the mesh
> >>> entities, and we wanted to build a global numbering of the dofs. The
> >>> only advantage I can see with our approach is that it may use less
> >>> memory, since we don't need to store an extra numbering scheme for all
> >>> mesh entities but this not a big deal.
> >>>
> >>> A few questions:
> >>>
> >>> 1. Is the above interpretation correct?
> >>>
> >>>       
> >> Yes.
> >>
> >> Another disadvantage with the global numbering scheme is the mesh
> >> connectivity calculations (mesh.init in MeshRenumber).
> >>     
> >
> > Why is this a problem? As far as I understand, there are always two
> > different numberings of mesh entities, one local (same as we have
> > now) and one global. The local can be computed as usual and then the
> > global can be reconstructed from the local + the overlap.
> >
> > (overlap = how the local pieces fit together)
> >
> >
> >   
> 
> Iterating over the local + overlap requires some mesh connectivity, 
> which are costly to generate.

What's your point? Are you arguing against a global numbering scheme
for the mesh entities? I thought this is what you have implemented.

> >> No, the mesh is always distributed.
> >>     
> >
> > How does this work? Is the IO also parallel? I can't see any parallel
> > stuff in XMLMesh.cpp.
> >
> >   
> 
> 
> The mesh file is parsed in parallel, PXMLMesh.cpp. It depends on a 
> geometric partitioner for an initial distribution of the coordinates, 
> thus not SCOTCH friendly.

ok.

> >>> 4. Is the extra storage dynamic? If only vertices are needed (for P1
> >>> elements), then we only need to store extra vertex numbers.
> >>>
> >>>       
> >> No, it always stores a global number for each mesh entity.
> >>     
> >
> > ok, this needs to be fixed but it shouldn't be hard. The DofMap knows
> > which entities are needed. This might be an argument in favor of
> > computing a parallel dof map from local dof maps rather than computing
> > a parallel mesh numbering from local mesh numberings.
> >
> >   
> >>> 5. MeshRenumber seems specific to triangles and tets. Can it be done
> >>> without reference to specific entities with special cases put in
> >>> CellType?
> >>>
> >>>       
> >> Maybe, the problem is to construct a key that could be used to uniquely
> >> identify the entity.
> >>     
> >
> > Isn't (dim, index) enough? Or is this because you only have vertex
> > information in your overlap?
> >
> >   
> 
> Yes. With more information in the overlap, (dim, index) should be enough.

ok.

> >> But, since each entity could be seen as a set of vertices it shouldn't
> >> be any problem.
> >>     
> >>> 6. Does it work for assembly over interior facets (like in DG
> >>> methods)?
> >>>
> >>>       
> >> I'm not sure, haven't tried any DG type problems.
> >>     
> >
> > I expect it to be problematic. For interior facet integrals, we need
> > to iterate over all interior facets in the mesh and for each facet
> > access its two adjacent cells. One of these may be on another
> > processor.
> >
> >   
> 
> 
> But shouldn't a set of ghosted cells for the interior boundary solve 
> this problem?

Do you mean an overlap consisting of cells, not facets?

> >>> 7. Is it possible to make it work with SCOTCH (in addition to
> >>> ParMetis)?
> >>>
> >>>       
> >> Yes, with some modification to MPIMeshCommunicator.
> >>     
> >
> > ok.
> >
> >   
> >>> Then some suggestions:
> >>>
> >>> 1. I'd like to move the implementation of DofMap::build() to
> >>> DofMapBuilder (to simplify DofMap.cpp).
> >>>
> >>> 2. Function names should be fooBar(), not foo_bar().
> >>>
> >>>       
> >> I recently found doc/misc/policy :)
> >>     
> >>> 3. There needs to be some #ifdef HAS_PARMETIS so it may be built
> >>> without Parmetis.
> >>>
> >>> 4. I'd like to add a new class GlobalNumbering (to replace
> >>> MeshDistributedData) that holds the global numbering scheme.
> >>>
> >>> The Mesh class can have a pointer to a GlobalNumbering object which is
> >>> 0 by default so no extra data (or at least not more than 4 bytes) is
> >>> stored when not running in parallel.
> >>>
> >>> Then we can add a function MeshEntity::number() which returns the same
> >>> as index() if GlobalNumbering is 0. Otherwise, it returns what is
> >>> stored in GlobalNumbering.
> >>>
> >>> GlobalNumbering can have an array of MeshFunctions, one for each
> >>> topological dimension, that maps the local entity indices to their
> >>> global numbers.
> >>>
> >>> Thus, a MeshEntity will have two functions index() and number().
> >>> These will return the same value in sequential and possibly different
> >>> values in parallel.
> >>>
> >>> Let's await some more comments and then get started. It would be nice
> >>> to get it in small patches to give us an opportunity to comment/edit.
> >>> Adding GlobalNumbering and MeshEntity::number() would be a good start.
> >>>
> >>>       
> >> Sounds good, However I think the {T,S,F,V,O} design mentioned by Johan
> >> is more efficient.
> >>     
> >
> > These are different issues. One issue is how to represent the overlap,
> > however we choose to do that, either {T,S,F,V,O} or just {T, S, F}.
> > The other issue is how we store either the global mesh numbering or
> > the global dof map. We need to discuss both.
> >
> > I'd like to continue this discussion for some time before we jump on
> > to either option.
> >
> > 1. Do we want to compute a global mesh numbering and then let the
> > global dof map follow as usual, or do we want to have just a local
> > mesh numbering and then compute a global dof map from the local dof
> > maps + the overlap?
> >
> > In either case, the resulting global dof map can be renumbered to
> > reduce bandwidth/communication.
> >
> > 2. How do we want to represent the overlap? Is it enough to store it
> > as {T, S, F} or do we need more? Do we need a class Overlap that
> > stores the overlap?
> >
> 
> I think the {V,O} information is needed in order to redistribute 
> (reconstruct with the MeshEditor) a mesh during computation (due to load 
> balancing). A Global numbering is (probably) also needed for 
> refinement/coarsening.

Again, aren't these separate issues? One issue is how we compute the
global dof map (which is needed for assembly). The other is how we
store information about the overlap which we need both for computing a
global numbering or global dof map, and for load balancing,
refinement, coarsening etc.

Other opinions?

Do we want to (i) build a global numbering of mesh entities or (ii)
have local numbering of mesh entities and build a global dof map?

-- 
Anders

Attachment: signature.asc
Description: Digital signature