I think it looks good.
As far as I understand, you build a global numbering of all mesh
entities (which may be different from the local numbering on each
processor), and then the (global parallel) local-to-global mapping
follows from tabulate_dofs just as usual.
So, the difference is that you build a global numbering of the mesh
entities, and we wanted to build a global numbering of the dofs. The
only advantage I can see with our approach is that it may use less
memory, since we don't need to store an extra numbering scheme for all
mesh entities but this not a big deal.
A few questions:
1. Is the above interpretation correct?
Yes.
Another disadvantage with the global numbering scheme is the mesh
connectivity calculations (mesh.init in MeshRenumber).
Why is this a problem? As far as I understand, there are always two
different numberings of mesh entities, one local (same as we have
now) and one global. The local can be computed as usual and then the
global can be reconstructed from the local + the overlap.
(overlap = how the local pieces fit together)
4. Is the extra storage dynamic? If only vertices are needed (for P1
elements), then we only need to store extra vertex numbers.
No, it always stores a global number for each mesh entity.
ok, this needs to be fixed but it shouldn't be hard. The DofMap knows
which entities are needed. This might be an argument in favor of
computing a parallel dof map from local dof maps rather than computing
a parallel mesh numbering from local mesh numberings.
5. MeshRenumber seems specific to triangles and tets. Can it be done
without reference to specific entities with special cases put in
CellType?
Maybe, the problem is to construct a key that could be used to uniquely
identify the entity.
Isn't (dim, index) enough? Or is this because you only have vertex
information in your overlap?
But, since each entity could be seen as a set of vertices it shouldn't
be any problem.
6. Does it work for assembly over interior facets (like in DG
methods)?
I'm not sure, haven't tried any DG type problems.
I expect it to be problematic. For interior facet integrals, we need
to iterate over all interior facets in the mesh and for each facet
access its two adjacent cells. One of these may be on another
processor.
7. Is it possible to make it work with SCOTCH (in addition to
ParMetis)?
Yes, with some modification to MPIMeshCommunicator.
ok.
Then some suggestions:
1. I'd like to move the implementation of DofMap::build() to
DofMapBuilder (to simplify DofMap.cpp).
2. Function names should be fooBar(), not foo_bar().
I recently found doc/misc/policy :)
3. There needs to be some #ifdef HAS_PARMETIS so it may be built
without Parmetis.
4. I'd like to add a new class GlobalNumbering (to replace
MeshDistributedData) that holds the global numbering scheme.
The Mesh class can have a pointer to a GlobalNumbering object which is
0 by default so no extra data (or at least not more than 4 bytes) is
stored when not running in parallel.
Then we can add a function MeshEntity::number() which returns the same
as index() if GlobalNumbering is 0. Otherwise, it returns what is
stored in GlobalNumbering.
GlobalNumbering can have an array of MeshFunctions, one for each
topological dimension, that maps the local entity indices to their
global numbers.
Thus, a MeshEntity will have two functions index() and number().
These will return the same value in sequential and possibly different
values in parallel.
Let's await some more comments and then get started. It would be nice
to get it in small patches to give us an opportunity to comment/edit.
Adding GlobalNumbering and MeshEntity::number() would be a good start.
Sounds good, However I think the {T,S,F,V,O} design mentioned by Johan
is more efficient.
These are different issues. One issue is how to represent the overlap,
however we choose to do that, either {T,S,F,V,O} or just {T, S, F}.
The other issue is how we store either the global mesh numbering or
the global dof map. We need to discuss both.
I'd like to continue this discussion for some time before we jump on
to either option.
1. Do we want to compute a global mesh numbering and then let the
global dof map follow as usual, or do we want to have just a local
mesh numbering and then compute a global dof map from the local dof
maps + the overlap?
In either case, the resulting global dof map can be renumbered to
reduce bandwidth/communication.
2. How do we want to represent the overlap? Is it enough to store it
as {T, S, F} or do we need more? Do we need a class Overlap that
stores the overlap?
--
Anders
