dolfin team mailing list archive

[Anders Logg <logg@xxxxxxxxx>]

Ola and I have now finished up the first round of getting DOLFIN to
run in parallel. In short, we can now parse meshes from file in
parallel and partition meshes in parallel (using ParMETIS).

We reused some good ideas that Niclas Jansson had implemented in
PXMLMesh before, but have also made some significant changes as
follows:

1. The XML reader does not handle any partitioning.

2. The XML reader just reads in a chunk of the mesh data on each
processor (in parallel) and stores that into a LocalMeshData object
(one for each processor). The data is just partitioned in blocks so
the vertices and cells may be completely unrelated.

3. The partitioning takes place in MeshPartitioning::partition,
which gets a LocalMeshData object on each processor. It then calls
ParMETIS to compute a partition (in parallel) and then redistributes
the data accordingly. Finally, a mesh is built on each processor using
the local data.

4. All direct MPI calls (except one which should be removed) have been
removed from the code. Instead, we mostly rely on dolfin::MPI::distribute
which handles most cases of parallel communication and works with STL
data structures.

5. There is just one ParMETIS call (no initial geometric
partitioning). It seemed like an unnecessary step, or are there good
reasons to perform the partitioning in two steps?

For testing, go to sandbox/passembly, build and then run

  mpirun -n 4 ./demo
  ./plot_partitions 4

Everyone interested in the parallel design and implementation is
encouraged to have a look at the following new classes:

  LocalMeshData:    storage for local mesh data (not yet a mesh)
  XMLLocalMeshData: parallel parser for local mesh data
  MeshPartitioning: partitioning algorithm, calling ParMETIS

-- 
Anders


On Fri, Nov 28, 2008 at 12:36:42PM +0100, Anders Logg wrote:
> Ola and I have started to look into the parallelization again and have
> come to some conclusions and suggestions:
> 
> The code in PXMLMesh.cpp is a complex (and impressive) beast which
> handles quite a few things at the same time: parsing XML, partitioning
> with ParMetis, redistribution of mesh data with MPI, and building a
> distributed mesh with DynamicMeshEditor.
> 
> It would be good to break all this code up into pieces. In particular,
> it seems non-optimal to let an XML parser handle partitioning and
> parallel communication.
> 
> We therefore suggest that the XML parser should only read in data from
> file. No calls to MPI, no calls to ParMetis, just reading data.
> 
> Instead, the mesh must be distributed by first parsing portions of a
> mesh XML file on each processor and then calling Mesh::distribute() to
> do the actual work (partitioning and distribution). Something like
> this:
> 
>   Mesh mesh;
>   ParallelMeshData pdata("mesh.xml");
>   mesh.distribute(pdata); // calls MeshDistribution::distribute()
> 
> I'm not entirely happy with the above syntax so suggestions are
> welcome.
> 
> Based on the above algorithms, we can also create a simple script
> which allows offline distribution of meshes:
> 
>   dolfin-partition mesh.xml 16
> 
> This will create 16 mesh files named mesh-0.xml, mesh-1.xml etc.
> 
> One may then read in the partitioned data on each processor by
> 
>   Mesh mesh("mesh-*.xml");
> 
> So in summary, we suggest that we let all XML parsers just parse mesh
> data, and move the parallel partitioning and distribution to separate
> classes MeshPartitioning and/or MeshDistribution. We have made some
> progress on reworking PXMLMesh.cpp already and can hopefully have a
> first prototype ready pretty soon (unless I get tied up with finishing
> the Python interface for the next release).
> 



> _______________________________________________
> DOLFIN-dev mailing list
> DOLFIN-dev@xxxxxxxxxx
> http://www.fenics.org/mailman/listinfo/dolfin-dev

Attachment: signature.asc
Description: Digital signature