dolfin team mailing list archive

["Garth N. Wells" <g.n.wells@xxxxxxxxxx>]

Martin Sandve Alnæs wrote:
> > The partition is related to this. Entries in a matrix associated with a
> > particular partition are formed on the process to which the mesh
> > partition belongs - no problems there. But you need to make sure
> > (through the dof numbering) that (nearly) all of the terms computed by a
> > process are also stored by that process.
>
> In the index_set built in the beforementioned algorithm, all nodes
> related to a cell partition are included. Nodes shared between two
> processes (partitions) are stored on both processes. This is
> independent of the global dof numbering.
>
> During the assembly, I'm not sure matrix->SumIntoGlobalValues(...)
> does any communication at all. At the end of assembly there is a call
> I didn't mention, matrix->GlobalAssemble(), which does the
> communication of shared values which are then added together.
>
> > The FFC mapping for vector-valued equations is particularly unsuited to
> > this as two unknowns corresponding to a single node (say x and y
> > components) are located far from each other (the distance is 1/2 of the
> > matrix size).
>
> I don't see this as being the same issue. 


> Which nodes reside on a
> particular process is defined by the cell partition, independent of
> the global dof numbering. 

The cell partition determines by which process the contribution of a 
cell is computed. The degree of freedom numbering determines to which 
process the matrix entry belongs. Consider a 2n x 2n matrix stored on 
two processes,

  Prcocess 0  | A  B |
  Prcocess 1  | C  D |

were the sub-matrices A and B reside on process 0, and C and D reside on 
process 1. All sub-matrices have size n x n. Consider a cell in the 
middle of partition 0 and it's local degrees of freedom (0, 1, 2) map to 
(n+1, n+2, n+3) in the global matrix. The contribution of this cell is 
computed on process 0, but for matrix assembly it is communicated to 
process 1.  I don't know Epetra, but I suspect that GlobalAssemble() 
performs this communication.

This is what the algorithm does; given a
> global dof numbering and a cell partition, select all the global dofs
> touched by cells in this partition. (In the local renumbered dof
> vector, the distance between these nodes will be 1/2 of the local
> vector size.)
>
> Note that I haven't done much parallell stuff at all, so my arguments
> here are pretty much colored by the design of Epetra as I've
> understood it so far. But this seems very flexible.

We're in the same boat here :).

Garth

> martin