dolfin team mailing list archive

["Garth N. Wells" <gnw20@xxxxxxxxx>]

On 03/03/11 19:48, Johan Hake wrote:
> On Thursday March 3 2011 11:20:03 Marie E. Rognes wrote:
>> On 03/03/2011 08:03 PM, Johan Hake wrote:
>>> Hello!
>>>
>>> I am using Mixed spaces with Reals quite alot. It turnes out that
>>> assemble forms with functions from MixedFunctionSpaces containing Real
>>> spaces are dead slow. The time spent also increase with the number of
>>> included Real spaces, even if none of them are included in form which is
>>> assembled.
>>>
>>> The attached test script illustrates this.
>>
>> By replacing "CG", 1 by "R", 0 or?
> 
> OMG!! Yes, *flush*
> 
> That explains the memory usage :P
> 
>>> The test script also reviels that an unproportial time is spent in FFC
>>> generating the code. This time also increase with the number of Real
>>> spaces included. Turning of FErari helped a bit with this point.
>>
>> I can take a look on the FFC side, but not today.
> 
> Nice!
> 
> With the update correction from Marie the numbers now looks like:
> 
> With PETSc backend
> 
> Tensor without Mixed space  |       0.11211     0.11211     1
> With 1 global dofs          |        1.9482      1.9482     1
> With 2 global dofs          |        2.8725      2.8725     1
> With 4 global dofs          |        5.1959      5.1959     1
> With 8 global dofs          |        10.524      10.524     1
> With 16 global dofs         |        25.574      25.574     1
> 
> With Epetra backend
> 
> Tensor without Mixed space  |       0.87544     0.87544     1
> With 1 global dofs          |        1.7089      1.7089     1
> With 2 global dofs          |        2.6868      2.6868     1
> With 4 global dofs          |          4.28        4.28     1
> With 8 global dofs          |         8.123       8.123     1
> With 16 global dofs         |        17.394      17.394     1
> 
> Still a pretty big increase in time for just adding 16 scalar dofs to a system 
> of 274625 dofs in teh first place.
> 

I have seen this big slow down for large problems. The first issue,
which was the computation of the sparsity pattern, has been 'resolved'
by using boost::unordered_set. This comes at the expense of the a small
slow down for regular problems.

I also noticed that Epetra performs much better for these problems than
PETSc does. We need to check the matrix initialisation, but it could
ultimately be a limitation of the backends. Each matrix row
corresponding to a global dof is full, and it may be that backends
designed for large sparse matrices do not handle this well.

The best approach is probably to add the entire row at once for global
dofs. This would require a modified assembler.

There is a UFC Blueprint to identify global dofs:

    https://blueprints.launchpad.net/ufc/+spec/global-dofs

If we can identify global dofs, we have a better chance of dealing with
the problem properly. This includes running in parallel with global dofs.

Garth

> Johan
> 
>> --
>> Marie
>>
>>> I have not profiled any of this, but I just throw it out there. I do not
>>> recognize any difference between for example Epetra or PETSc backend as
>>> suggested in the fixed bug for building of sparsity pattern with global
>>> dofs.
>>>
>>> My test has been done on a DOLFIN 0.9.9+. I haven't profiled it yet.
>>>
>>> Output from summary:
>>>    Tensor without Mixed space  |       0.11401     0.11401     1
>>>    With 1 global dofs          |       0.40725     0.40725     1
>>>    With 2 global dofs          |       0.94694     0.94694     1
>>>    With 4 global dofs          |         2.763       2.763     1
>>>    With 8 global dofs          |        9.6149      9.6149     1
>>>
>>> Also the amount of memory used to build the sparsity patter seams to
>>> double for each step. The memory print for a 32x32x32 unit cube with 16
>>> global dofs was 1.6 GB memory(!?).
>>>
>>> Johan
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dolfin
>>> Post to     : dolfin@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dolfin
>>> More help   : https://help.launchpad.net/ListHelp
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dolfin
>> Post to     : dolfin@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dolfin
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dolfin
>> Post to     : dolfin@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dolfin
>> More help   : https://help.launchpad.net/ListHelp