dolfin team mailing list archive

["Garth N. Wells" <gnw20@xxxxxxxxx>]

On Fri, 2010-08-06 at 21:19 +0200, Anders Logg wrote:
> On Fri, Aug 06, 2010 at 08:13:13PM +0100, Garth N. Wells wrote:
> > On Fri, 2010-08-06 at 21:06 +0200, Anders Logg wrote:
> > > On Fri, Aug 06, 2010 at 07:55:54PM +0100, Garth N. Wells wrote:
> > > > On Fri, 2010-08-06 at 20:53 +0200, Anders Logg wrote:
> > > > > On Fri, Aug 06, 2010 at 07:51:18PM +0100, Garth N. Wells wrote:
> > > > > > On Fri, 2010-08-06 at 20:36 +0200, Anders Logg wrote:
> > > > > > > On Fri, Aug 06, 2010 at 04:55:44PM +0100, Garth N. Wells wrote:
> > > > > > > > On Fri, 2010-08-06 at 08:42 -0700, Johan Hake wrote:
> > > > > > > > > On Friday August 6 2010 08:16:26 you wrote:
> > > > > > > > > > ------------------------------------------------------------
> > > > > > > > > > revno: 4896
> > > > > > > > > > committer: Garth N. Wells <gnw20@xxxxxxxxx>
> > > > > > > > > > branch nick: dolfin-all
> > > > > > > > > > timestamp: Fri 2010-08-06 16:13:29 +0100
> > > > > > > > > > message:
> > > > > > > > > >   Add simple Stokes solver for parallel testing.
> > > > > > > > > >
> > > > > > > > > >   Other Stokes demos don't run in parallel because MeshFunction io is not
> > > > > > > > > >   supported in parallel.
> > > > > > > > >
> > > > > > > > > Does anyone have an overview of what is needed for this to be fixed. I
> > > > > > > > > couldn't find a blueprint on it.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Here it is:
> > > > > > > >
> > > > > > > >     https://blueprints.launchpad.net/dolfin/+spec/parallel-io
> > > > > > > >
> > > > > > > > > I am interested in getting this fixed :)
> > > > > > > > >
> > > > > > > >
> > > > > > > > Me too! We need to look at all the io since much of it is broken in
> > > > > > > > parallel.
> > > > > > > >
> > > > > > > > We need to settle on how to handle XML data. I favour (and I know Niclas
> > > > > > > > Janson does too) the VTK approach in which we have a 'master file' that
> > > > > > > > points to other XML files which contain portions of the vector/mesh,
> > > > > > > > etc. Process zero can read the 'master file' and then instruct the other
> > > > > > > > processes on which file(s) they should read in.
> > > > > > >
> > > > > > > This only works if the data is already partitioned. Most of our demos
> > > > > > > assume that we have the mesh in one single file which is then
> > > > > > > partitioned on the fly.
> > > > > > >
> > > > > >
> > > > > > The approach does work for data which is not partitioned. Just like with
> > > > > > VTK, one can read the 'master file' or the individual files.
> > > > > >
> > > > > > > The initial plan was to support two different ways of reading data in parallel:
> > > > > > >
> > > > > > > 1. One file and automatic partitioning
> > > > > > >
> > > > > > > DOLFIN gets one file "mesh.xml", each process reads one part of it (just
> > > > > > > skipping other parts of the file), then the mesh is partitioned and
> > > > > > > redistributed.
> > > > > > >
> > > > > > > 2. Several files and no partitioning
> > > > > > >
> > > > > > > DOLFIN get multiple files and each process reads one part. In this
> > > > > > > case, the mesh and all associated data is already partitioned. This
> > > > > > > should be very easy to fix since everything that is needed is already
> > > > > > > in place; we just need to fix the logic. In particular, the data
> > > > > > > section of each local mesh contains all auxilliary parallel data.
> > > > > > >
> > > > > > > This can be handled in two different ways. Either a user specifies the
> > > > > > > name of the file as "mesh*.xml", in which case DOLFIN appends say
> > > > > > >
> > > > > > >   "_%d" % MPI::process_number()
> > > > > > >
> > > > > > > on each local process.
> > > > > > >
> > > > > > > The other way is to have a master file which lists all the other
> > > > > > > files. In this case, I don't see a need for process 0 to take any kind
> > > > > > > of responsibility for communicating file names. It would work fine for
> > > > > > > each process to read the master file and then check which file it
> > > > > > > should use. Each process could also check that the total number of
> > > > > > > processes matches the number of partitions in the file. We could let
> > > > > > > process 0 handle the parsing of the master file and then communicate
> > > > > > > the file names but maybe that is an extra complication.
> > > > > > >
> > > > > >
> > > > > > This fails when the number of files differs from the number of
> > > > > > processes. It's very important to support m files on n processes. We've
> > > > > > discussed this at length before.
> > > > >
> > > > > I don't remember. Can you remind me of what the reasons are?
> > > > >
> > > >
> > > > I perform a simulation using m processes, and write the result to m
> > > > files. Later I want to use the result later in another computation using
> > > > n processors.
> > >
> > > I assume you did your first simulation (with m processors) starting
> > > from one big file?
> > >
> >
> > What do you mean? The first simulation might not read in any file.
> 
> I assumed you had an input for your mesh with some nontrivial
> geometry. The only other possibilities I see are either if you use one
> of the builtin meshes or if you have some custom program that
> generates your mesh. Or is the thing that you have done mesh
> refinement?
> 
> > > Can't you just restart from that file when you later want to run with
> > > n processors? It would not be much extra work, and maybe it would even
> > > be faster considering all the extra communication.
> > >
> >
> > The communication cost is negligible when reading a vector just once and
> > distributing.
> >
> > It will be very easy to read m files on n processes, so I don't get why
> > would we wish to prevent it?
> 
> Yes, it would be easy but we would need to redistribute it before the
> call to ParMETIS.
> 
> Another "problem" is if m > n. Then we would have to decide which
> processes read multiple files and how many.
> 
> > Also, we can't rely in the data that we read in being suitably
> > partitioned.
> 
> In my thinking, that is one of the main points of being able to read
> in multiple files, that you have already done the work of the
> partitioning and have a ready-made partition and just want to run the
> simulation again, or perhaps another simulation on the same mesh. In
> other words to allow computing in parallel without needing to go
> through the partitioning step.
> 

Partitioning is not the most important issue. The important issue is
scalability - if I use 10^3 processes, I can't be gathering all data on
process 0 for io, at possibly many steps in a simulation. We should be
looking at this as a questions of io.

On a shared cluster one usually has time limits and will need restarts,
but may not always have access to the same number of processors.

Garth

> --
> Anders