yade-mpi team mailing list archive

[François <francois.kneib@xxxxxxxxx>]

Yes, the merge in mpy.py only copies pos, vel, angVel, ori (13 floats). We
lack everything else, including all variables that carry contact history
information. The most common is the shear force that starts at 0 then
increase during tangential load. As a result, using MPIBodyContainer (and
then set replace the interactions) should fix parallel code to exactly fit
a serial simulation.

Le mar. 30 avr. 2019 à 15:38, Deepak Kn <deepak.kn1990@xxxxxxxxx> a écrit :

> Yes that was the case with the hackthon too. There is a small difference*
> in the forces between the mpi version (with merge) and the serial version,
> From what I understood (from previous discussions), that this is due to the
> lack of information on the interactions during the merge operation.
> *20000 bodies serial : force value = -35759751.3195
> *20000 bodies parallel : mpf : force value = -37733943.6400
>
> On Tue, Apr 30, 2019 at 3:01 PM François <francois.kneib@xxxxxxxxx> wrote:
>
>> In the case without merge, the results are exactly the same in comparison
>>> to mpy.py (test2D_MPI.py, no merge.) . We need to add a a list of things to
>>> be done (and which are already done ) in the gitlab issue page. I will
>>> start his.
>>>
>> Yes, this is also what I have seen but I can't remember: was it already
>> like that at the end of the hackathon ?
>>
>> Nice list !
>> I will work on the second point: exchanging bodies with MPIBodyContainer.
>> Even though at this time we don't care about the "when" question.
>>
>> Le mar. 30 avr. 2019 à 14:57, Deepak Kn <deepak.kn1990@xxxxxxxxx> a
>> écrit :
>>
>>> Hello,
>>>
>>> So I have added a 'to do list', at gitlab based on my understanding of
>>> yade-mpi and the previous discussions : (
>>> https://gitlab.com/yade-dev/trunk/issues/68). What do you think?
>>>
>>> On Tue, Apr 30, 2019 at 2:07 PM Deepak Kn <deepak.kn1990@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> *Did you try a simulation with merge/split operations ?*
>>>>
>>>> I am working on the merge operation right now : The issue is once we
>>>> send all the bodies to the master proc, the interactions of these bodies
>>>> have to be set in to the master's interaction container, I'm looking at
>>>> this now. So I have not tried any simulations with the merge/split
>>>> operations.
>>>>
>>>> *are you able to get the right force on the bottom wall ?*
>>>>
>>>> In the case without merge, the results are exactly the same in
>>>> comparison to mpy.py (test2D_MPI.py, no merge.) . We need to add a a list
>>>> of things to be done (and which are already done ) in the gitlab issue
>>>> page. I will start his.
>>>>
>>>>
>>>> On Tue, Apr 30, 2019 at 1:42 PM François <francois.kneib@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Thanks deepak,
>>>>>
>>>>> I see that, in addition to merge(), you have moved most of the python
>>>>> splitScene() into c++ Subdomain::findInterSections(). That's great, not the
>>>>> easiest part ! So is that part of the code working ? Did you try a
>>>>> simulation with merge/split operations ?
>>>>> If I'm right we know communicate bodies/ids only, so the last (but not
>>>>> least) question: are you able to get the right force on the bottom wall ?
>>>>>
>>>>> Le lun. 29 avr. 2019 à 15:21, Deepak Kn <deepak.kn1990@xxxxxxxxx> a
>>>>> écrit :
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Yes, checkout from the branch : mpi_dpk. : there's an example in
>>>>>> examples/mpi/testnomerge.py
>>>>>> also, there's another py module in py : py/yadempi.py (this is more
>>>>>> or less similar to mpy.py but we send bodies instead of states).
>>>>>> If you want to see the sizes of messages  : (size as in number of
>>>>>> elems in the buffer)
>>>>>> add this line  in Subdomain.cpp , function void Subdomain::
>>>>>> recvContainerString after line 429 : std::cout << "sizes " <<
>>>>>> recvdStringSizes[i] << std::endl;
>>>>>> let me know if you have any troubles, also if you find some part of
>>>>>> the code inefficient (or could be made better), go ahead and fix it!
>>>>>>
>>>>>> Thanks and regards,
>>>>>> deepak
>>>>>>
>>>>>> On Mon, Apr 29, 2019 at 2:52 PM François <francois.kneib@xxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Deepak, could you please provide an example script to test this
>>>>>>> feature ?
>>>>>>>
>>>>>>> Le lun. 29 avr. 2019 à 13:39, François <francois.kneib@xxxxxxxxx> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi, and nice work Deepak.
>>>>>>>> I would add that we can't compare directly the number of elements
>>>>>>>> that are transmitted, but rather the total volume, as size of char is 8bits
>>>>>>>> and size of double is 64bits by default on modern cpus.
>>>>>>>> As a result, we should rather compare 4180*8=33440 with 215456,
>>>>>>>> which still represents a factor of 10.
>>>>>>>>
>>>>>>>> Le lun. 29 avr. 2019 à 13:16, Bruno Chareyre <
>>>>>>>> bruno.chareyre@xxxxxxxxxxxxxxx> a écrit :
>>>>>>>>
>>>>>>>>> Hi Deepak,
>>>>>>>>> Clearly there is no reason to compare speed of sending pos+vel vs.
>>>>>>>>> speed of sending the entire body, since these communications are not used
>>>>>>>>> at the same stages.
>>>>>>>>> Sending serialized bodies is an alternative to sending the whole
>>>>>>>>> scene, and I guess (?) sending 5% of the bodies in a scene will take less
>>>>>>>>> time than sending the scene.
>>>>>>>>> The communication which occures at each iteration will always be
>>>>>>>>> only pos+vel.
>>>>>>>>>
>>>>>>>>> @Janek, when we break a scene in multiple pieces we want to send
>>>>>>>>> everything because we don't want to worry about what needs to be sent in
>>>>>>>>> each other particular case, and actually if an attribute is there then we
>>>>>>>>> can safely assume that there is a reason for it. When it's already broken
>>>>>>>>> in pieces we update (communicate) only positions and velocity because
>>>>>>>>> otherwise it's too expensive, as Deepak found, but it's not foolproof. For
>>>>>>>>> instance differential growing of particles in each subdomain would break
>>>>>>>>> this concept since radius is not communicated at every iteration.
>>>>>>>>>
>>>>>>>>> Bruno
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, 28 Apr 2019 at 19:58, Deepak Kn <deepak.kn1990@xxxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello All,
>>>>>>>>>>
>>>>>>>>>> I have finished implementing the functions to serialize and
>>>>>>>>>> deserialize the bodies and the necessary communications to exchange the
>>>>>>>>>> serialized bodies. The serialized messages (binary) are several times
>>>>>>>>>> larger than the usual (state) messages in mpy/mpf.py (vel,pos, ori, id,
>>>>>>>>>> bounds) etc. And as a result the simulations are far slower ! Here's how it
>>>>>>>>>> scales : for exchanging 220 bodies, the serialized messages were around
>>>>>>>>>> 215456 elements (chars), and in the previous versions, the message sizes
>>>>>>>>>> were 4180 (doubles). (testMPI2D no merge, on 3 procs, 20000 bodies.)
>>>>>>>>>>
>>>>>>>>>> Most of the functions are implemented in Subdomain.cpp
>>>>>>>>>> Subdomain.hpp and MPIBodyContainer.hpp, (branch : mpi_dpk on gitlab) I
>>>>>>>>>> followed the same logic of the previous version (mpy/mpf), non blocking
>>>>>>>>>> sends and blocking recvs (with probes) are used for the communications
>>>>>>>>>> between the workers, and blocking communication for sending the forces to
>>>>>>>>>> the master (the same as in mpy/mpf).
>>>>>>>>>>
>>>>>>>>>> I think this can be improved, we can serialize the states,bounds
>>>>>>>>>> and send these instead of the bodies? what do you think? (all the functions
>>>>>>>>>> do serialize, deserialize and communicate are ready. We just need a
>>>>>>>>>> container to hold only the required info.)
>>>>>>>>>> --
>>>>>>>>>> Mailing list: https://launchpad.net/~yade-mpi
>>>>>>>>>> Post to     : yade-mpi@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>> Unsubscribe : https://launchpad.net/~yade-mpi
>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> --
>>>>>>>>> _______________
>>>>>>>>> Bruno Chareyre
>>>>>>>>> Associate Professor
>>>>>>>>> ENSE³ - Grenoble INP
>>>>>>>>> Lab. 3SR
>>>>>>>>> BP 53
>>>>>>>>> 38041 Grenoble cedex 9
>>>>>>>>> Tél : +33 4 56 52 86 21
>>>>>>>>> ________________
>>>>>>>>>
>>>>>>>>> Email too brief?
>>>>>>>>> Here's why: email charter
>>>>>>>>> <https://marcuselliott.co.uk/wp-content/uploads/2017/04/emailCharter.jpg>
>>>>>>>>> --
>>>>>>>>> Mailing list: https://launchpad.net/~yade-mpi
>>>>>>>>> Post to     : yade-mpi@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~yade-mpi
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>> --
>>>>>>> Mailing list: https://launchpad.net/~yade-mpi
>>>>>>> Post to     : yade-mpi@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~yade-mpi
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>> --
>>>>> Mailing list: https://launchpad.net/~yade-mpi
>>>>> Post to     : yade-mpi@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~yade-mpi
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>> --
>> Mailing list: https://launchpad.net/~yade-mpi
>> Post to     : yade-mpi@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~yade-mpi
>> More help   : https://help.launchpad.net/ListHelp
>>
>