dolfin team mailing list archive

["Murtazo Nazarov" <murtazo@xxxxxxxxxxx>]

> On Sat 2008-05-17 19:48, Murtazo Nazarov wrote:
>> I think the MatSetFromOptions() and MatSeqAIJSetPreallocation() are
>> called
>> after creating Petsc matrix. I don't know where is the right place to
>> put
>> them.
>
> Creating the matrix just creates an empty container.  The big malloc
> happens in
> MatSeqAIJSetPreallocation (actually MatSeqAIJSetPreallocation_IMPL).  Just
> put
> MatSetFromOptions before MatSeqAIJSetPreallocation.  If you really want to
> override user options (effectively make certain options unavailable from
> the
> command line), you can put them after MatSetFromOptions.
>
>> Attached I send you log files using -info and -log_summary. I use
>> 50x50x50
>> mesh.
>
>   [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 397953 X 397953; storage
> space: 0 unneeded,17351559 used
>   [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
>   [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 45
>   [0] Mat_CheckInode(): Found 397953 nodes out of 397953 rows. Not using
> Inode routines
>   time 21.770000
>
> I assume this is a scalar problem with linear elements on a tetrahedral
> mesh.
> The preallocation is done correctly here so you're unlikely to be able to
> improve this by huge amounts.  The reason the first assembly is more
> expensive
> is that the structure on each row is not known so it needs to be shuffled
> around.  Since you know where the entries are (from the mesh) you could do
> a
> dummy assemble (call MatSetValues once per row with the correct column
> numbers
> and zeros for the entries, then MatAssembleBegin/End with
> MAT_FLUSH_ASSEMBLY (I
> think this is enough)).  This should take around 2 seconds for your
> matrix,
> similar to a finite difference assembly.  After that, all assemblies
> should take
> the same time.  I don't think 7 seconds is terribly bad for this matrix.
> It is
> about 3 times longer than my quoted time for a finite difference assembly
> of a
> matrix of similar size on a Core 2 Duo CPU T9300 @ 2.50GHz.  If you want
> to
> speed up the subsequent assemblies, I suspect you will have take advantage
> of
> the special structure of mesh traversal to avoid updating the same values
> many
> times, but I don't know how much this would gain.  Is the 7 second
> assembly a
> bottleneck for solving this system?  If so, this matrix must be pretty
> well
> conditioned.  In any case, assembly should scale well in parallel and well
> with
> problem size.
>

This is from 3D Navier-Stokes equation with linear elements. The first
time assembly takes longer because it does initialization of the matrix,
it calculates the sparsity pattern. But, then the sparsity pattern and
A.init() is not done. Maybe I cannot gain so much, but as I posted in
previous posts, I did exactly the same test with another package, with the
same mesh, the same elements and the same equations, and that was at least
3 times faster than assembly we have in dolfin. So, instead of 7 seconds
it is spent 7/3 seconds which I can gain 10 days from my simulation which
takes now 14 days.

>> > Before the preallocation fix:
>> >
>> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 25140 X 25140; storage
>> space:
>> > 261180 unneeded,730140 used
>> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
>> is
>> > 57708
>> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59
>> >
>> > After the fix:
>> >
>> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 25140 X 25140; storage
>> space: 0
>> > unneeded,730140 used
>> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
>> is 0
>> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59
>> >
>> > The time difference is around 3 orders of magnitude.
>>
>> This excellent. I wounder do I get the same or not. How did you do this
>> profiling?
>
> 3 orders of magnitude doesn't require profiling :-)
>
> Note that this example matrix was very small compared to yours.
>
> Jed
> _______________________________________________
> DOLFIN-dev mailing list
> DOLFIN-dev@xxxxxxxxxx
> http://www.fenics.org/mailman/listinfo/dolfin-dev
>