dolfin team mailing list archive

["Martin Sandve Alnæs" <martinal@xxxxxxxxx>]

On 12/7/06, Garth N. Wells <g.n.wells@xxxxxxxxxx> wrote:
> Martin Sandve Alnæs wrote:
> > On 12/6/06, Garth N. Wells <g.n.wells@xxxxxxxxxx> wrote:
> >> Martin Sandve Alnæs wrote:
> >> > On 12/6/06, Garth N. Wells <g.n.wells@xxxxxxxxxx> wrote:
> >> >> I used METIS. Take a look at the first function in the file
> >> >> src/test/passembly.main.cpp.
> >> >
> >> > I think your performance results may be a result of using the wrong
> >> > partitioning function. I've had a look at the METIS manual, and it
> >> > says this about the function you've used:
> >> >
> >>
> >> What do you mean? I was very happy with the performance. Given some of
> >> the rough edges in the test program, I was surprised.
> >
> > I'm thinking about having to renumber the dofs to reduce the
> > communication. Communication size should preferably be independent of
> > the dof numbering, only on the partition.
> >
>
> This is possible for assembly if entries assembled on a process reside
> on that process, but then for arbitrary dof numbering linear algebra
> operations will be deadly slow. The key goal of course is to minimise
> the couplings between blocks on different processes. To do this, you
> need to minimise the number of "connected" degrees of freedom which
> reside on different processes.
>
> If no renumbering is required for performance, then you could partition
> as mesh arbitrarily (just divide the number of cells across processes
> equally). No need to bother with a partitioning library like METIS (a
> lot of people would be out if business).

But minimizing the number of connected dofs between processes (~)
should be a direct result of minimizing the boundary between the
partitioned meshes. So when the partitioning library creates a good
mesh partition with low edge-cut, small communication amount should be
a direct result independent of the global dof numbering. Of coarse,
each process must store its "owned" dofs in a contiguous array, and
thus there must be a renumbering or index mapping. Renumbering this
array for local efficiency still shouldn't affect communication
amount.

> Effective renumbering (based on a good partition) is the key to
> performance. Even for a single process, renumbering can improve performance.

Yes, but that's a separate issue.

We're probably just talking past each other here.

After looking at your code again, I think the misunderstanding is in this:
(I don't know PetSC, so I might be wrong here)

You call VecCreateMPI(..., n_local, n_global, &vec), which probably
distributes the global vector entries in contiguous chunks to
processes. If you didn't renumber, I now see your "problem".

I've used an Epetra_Map(..., array_of_local_entries, n_local) which
distributes the global vector entries with a completely general
local-to-global mapping. But the local Epetra_Vector will store its
local entries in a contiguous array, so there is an implicit
renumbering here. But it is not a renumbering of the global dofs, it
is a separate renumbering or mapping between local and global vector
indices. With this approach, there is no connection between the
numbering of the global dofs and the communication amount.

martin