p2psp team mailing list archive

["L.G.Casado" <leo@xxxxxx>]

Dear Ilshat,

Very nice piece of work.
Now we can move to study the details of the experimentation. In order
to do so and to facilitate the evaluation you should include a brief
description of the results and the details about how you get them. Excu
se me, but I am involved in many other things and I need this level of
description in order to help to the improvement.

I have many questions. I know I can get them from the code, but I would
like you to answer them in order to be aware of what is being measured.

In your example:

> > ./test_strpe.py -n 48 -t 2 -m 50 -w 200
why n are different? 
What is the waiting time. It was the same in all the experiments?
As far I understood from the excel, n is always 100.
This is just a curiosity, for instance in the first case (m=10), 10
rounds are needed to have 10 malicious and 100 well intended peers. 
Are they included randomly? The concern is if the initialization will
affect the results.
When malicious start to attack?
What type of attack is measured?
How is buffer_correctness calculated? (an average?, best and worst case
will be also good to know).
Why well intended peers leave the team?

The interesting information will be the summary from the excel.
Once the type of attack is fixed. For instance: 
How buffer_correctness is affected by the number of good and malicious
peers.
Can we guest it mathematically?
What is the number of rounds to expel a given number of malicious?
Does it depends on the number of good/trusted peers? 
I think there is a direct correlation between/among:
i)buffer_correctness and number of malicious
ii)number of rounds to expel malicious- number of good peers -number of
trusted peers.
Taking into account the number of trusted peers:
Comparing -n 100 -m 10 -t 1 with  -n 100 -m 10 -t 2 malicious are
expelled at 75 and 30 rounds resp.
This seems logic. More trusted earlier expelled.
Curiosity, Why in the first case one malicious remains and not in the
second?
Comparing -n 100 -m 25 -t 1 with  -n 100 -m 25 -t 2, malicious are
expelled at 91 and 84, resp, which is not too much difference You save
7 rounds adding one trusted peer. This is not logic.
For the last case,  comparing -n 100 -m 25 -t 1 with  -n 100 -m 25 -t 2
malicious are expelled at 139 (remains 2 malicious, why?)  and 102.
They are not comparable.
 
In order to compare similar things, please, can you run the experiments
until no malicious peer are in the team. Can we guess the final number
of good peers? 
I think the attack should start when all peers are in the team, to hide
the initialization phase. And those rounds do not count in determining
how many rounds were needed to expel them all.
Additionally, due to the non deterministic behavior 5 runs should be
done, removing the best and worst in terms on the number of rounds to
expel all malicious peers and provide the average of the other three.
Regarding 
> To be honest, I dont see how splitter can determine malicious peers if
> there more than 50% of malicious peers in the team. Could you please
> advice some direction?
We have some ideas, but it is better to know how and why system works first.
I do not know if this can be seen as included in GSoC but the answer to
previous questions is what we want to know, because from them we can
determine which code will be finally included.   
Best regards,
Leo



El lun, 03-08-2015 a las 12:46 +0500, Ilshat Shakirov escribió:
> Hello!,> 
> I've written couple of scripts for testing only STrPe mechanism (for now). Here it is:
> https://github.com/ishakirov/p2psp/blob/master/tools/test_strpe.py
> https://github.com/ishakirov/p2psp/blob/master/tools/parse_test_strpe.py> To run it on not Mac systems, you have to change "runStream" function (since it runs mac specific path for vlc).> Also this commit contains changes for splitter and peer for logging some data (buffer correctnes, currenct round, teamsize).> The fist script run the experiment, it has 4 params:> n - teamsize without malicious and trusted peers> t - number of trusted peers> m - number of malicious peers> w - wait time in seconds> > And the second script parse the result files and produce an output, which can be 'copy-pasted' to excel or google sheet.> It runs with params:> n - teamsize with malicious and trusted peers> m - number of malicious peers.> 
> So these scripts can be used like:
> > ./test_strpe.py -n 48 -t 2 -m 50 -w 200
> > ./parse_test_strpe.py -n 100 -m 50
> > And here is some results:
> https://docs.google.com/spreadsheets/d/1BEHWLtKTVnZoqjEXNs-uLeH0sRU7SCXmZHSMuWc_-lo/edit#gid=985076992> 
> Now I am doing the same for the STrPe-DS mechanism. Could you please give me some feedback for this graphs? =)> I will write a blogpost later, when STrPe-DS will be ready.
> > Your reputation system, as many others, may produce false positive and false negative.
> > 
> > False positive: a bunch of malicious peers complaint about a well intended one.
> > False negative: a bunch of malicious peers does not complain about a malicious peer.
> > 
> > About time on the team: a malicious peer can perform correctly for a long period and suddenly make an attack.
> > 
> > It is not an easy task.
> To be honest, I dont see how splitter can determine malicious peers if there more than 50% of malicious peers in the team. Could you please advice some direction?> > Thanks!

> 2015-07-28 12:36 GMT+05:00 L.G.Casado > <leo@xxxxxx>> :
> > Dear ILshat,
> > > > Your reputation system, as many others, may produce false positive and false negative.
> > > > False positive: a bunch of malicious peers complaint about a well intended one.
> > False negative: a bunch of malicious peers does not complain about a malicious peer.
> > > > About time on the team: a malicious peer can perform correctly for a long period and suddenly make an attack.
> > > > It is not an easy task.
> > > > Best,
> > > > Leo
> > > > > > > > El mar, 28-07-2015 a las 12:00 +0500, Ilshat Shakirov escribió:
> > > As I said we can assign some reputation parameter to each peer. Let's say that we exclude peer P from the team if Q > X, where X some > > > threshold value and

> > > where I_i = 1 if i-th peer marked peer P as malicious and 0 in the other case.
> > > w_i can be assigned by the time of existing peer in the team. So if peer exists in team from the start of streaming, and it hasn't > > > complaints from the other peers, it has the maximum reputation value (w_i parameter).
> > > So the question is how to choose w_i more optimal (and the threshold value X respectively).

> > > 2015-07-28 1:08 GMT+05:00 Juan Álvaro Muñoz Naranjo > > > <juanalvaro83@xxxxxxxxx>> > > :
> > > > Hi again,> > > > 

> > > > > > > Also, I wanted to develop heuristic for the excluding malicious peers from the team based on the all the team (not only trusted peers). Do you have any ideas? I think about smth like: 'exclude peer if more than x% of the team marked it as malicious'. Also, we can assign 'reputation' to each peer, so some peers will have more influence on the decision of excluding peer. What do you think?


> > > > > > > > Yeah, we've been considering it. The problem with the x% solution is that i can easily turn against us. Imagine the attacker controls a high percentage of the nodes in the network (that would be easy, just run a huge number of peers > x% in a small number of machines and you've got it) and starts complaining against valid peers. The valid peers would be expulsed by the splitter. That would be an easy DoS. > > > > 

> > > > So, to reduce the impact of attackers, let's say that we set x to 50%: the attacker would need to control more than half of the team in order to expulse someone. But let's say the attacker controls 45% of the peers. Not enough to expulse anyone, but now he can act inversely: he uses that 45% of malicious peers to send corrupted chunks to a set of peers smaller than 50%. Those affected legal peers will not be able to play 45% of the packets, so they will probably abandon the team due to playback quality problems. Again a DoS. And the attackers will not be expulsed since the splitter did not receive at least 50% of complaints! > > > > 
> > > > Any idea on this?> > > > 

> > > > Juan
> > > >  
> > > > > > > Thanks! =)> > > > > > >  

> > > > > > > 2015-07-23 2:01 GMT+05:00 Juan Álvaro Muñoz Naranjo > > > > > > > <juanalvaro83@xxxxxxxxx>> > > > > > > :
> > > > > > > > Hi Ilshat,> > > > > > > > 

> > > > > > > > first of all thanks for your update, it was very interesting. Just one thing: when the DS technique is completed we'll send the public key under a X.509 certificate format. Ideally this certificate should be signed by a trusted certificate authority and contain information about the organization managing the splitter to offer some degree of trust. The certificate might even be distributed with the software, or be given by the web page if we were in a web player with WebRTC. Otherwise the attacker might send its own public key to the peers impersonating the splitter. But for now it is ok like that.> > > > > > > > 

> > > > > > > > Now, let's get to the point. How to run the experiments. Vicente already suggested the use of tools/create_a_team.sh in a previous message (thank you Vicente!). Also, Cristóbal suggests this: https://github.com/cristobalmedinalopez/p2psp-chunk-scheduling/blob/master/tools/run_experiment.sh 
> > > > > > > > These solutions are for experiments in one machine of course, which is enough for us. If you need more peers you should be able to combine several machines by running one script per machine. Of course, we're interested in seeing how peers' buffers are filled with chunks and not in video playback: as you can see, both scripts send the video signal to /dev/null.> > > > > > > > 

> > > > > > > > Which experiment to run? We propose the following: we're interested in average expulsion times for an attacker, and if all of them are expulsed after a given time. Also, the average percentage of gaps in the peers' buffers (so we can see if playback is possible in presence of attackers and after how long). I think you should measure time in terms of sending rounds (you know, one round would be the splitter sending one chunk to every member of the team).> > > > > > > > 

> > > > > > > > So, let's say that you have a team of 100 peers. From that team, a percentage of peers will be malicious: 1%, 10%, 25%, 50%. I imagine a plot in which the X axis is time (number of rounds) and in which we depict: number of remaining malicious peers in the team (because some of them will be expulsed) and average filling of peers' buffers. Ideally, as the number of remaining malicious peers decreases the filling of buffers should increase.> > > > > > > > 

> > > > > > > > Showing the number of complains from peers in the first technique would be also interesting.> > > > > > > > 

> > > > > > > > Another thing to measure would be the percentage of bandwidth used for real multimedia data (this is, how many bytes from the total are really used for transmitting the video). You can compare the baseline (no security measures, just plain video without malicious attackers) against both techniques. > > > > > > > > 

> > > > > > > > So, for running these experiments you'll need to decide which information you want to store from each peer (buffer filling percentage at each iteration, how many malicious peers at each iteration, how many bytes were sent and how many of them were used for video, how many complains arrived to the splitter in every iteration). Am I forgetting anything?> > > > > > > > 

> > > > > > > > My suggestion is run the experiment for the first technique and see how it goes. Make sure to run the experiment more than once, say 5 times, and then get the average of them all.> > > > > > > > 

> > > > > > > > Good work,> > > > > > > > 

> > > > > > > > Juan

> > > > > > > > 2015-07-21 20:06 GMT+02:00 Vicente Gonzalez > > > > > > > > <vicente.gonzalez.ruiz@xxxxxxxxx>> > > > > > > > :
> > > > > > > > > Hi Ilshat,> > > > > > > > > > > > > > > > > > did you try tools/create_a_team.sh?
> > > > > > > > > > > > > > > > > > (I tested to run up to 100 peers in my 8HG Mac machine)
> > > > > > > > > > > > > > > > > > Regards,
> > > > > > > > > Vi.> > > > > > > > > 

> > > > > > > > > On Sun, Jul 19, 2015 at 8:36 PM Ilshat Shakirov <im.shakirov@xxxxxxxxx> wrote:
> > > > > > > > > > Hello!,> > > > > > > > > > 
> > > > > > > > > > Sorry for the long delay.> > > > > > > > > > 
> > > > > > > > > > Here is status update about CIS of rules project: http://shakirov-dev.blogspot.ru/2015/07/5-6-7-week.html> > > > > > > > > > 
> > > > > > > > > > Also, I need some help with testing a big (ie, 20 peers) p2psp-teams. I want solution that allows to reproduce testing experiments easily. So the commenting lines (to remove need in running vlc) is not suitable for this.> > > > > > > > > > I've wrote simple script which runs several peers (in one machine) and here is result. I think it's quite hard to understand smth in this (and reproduce). So, what is the best solution for testing p2psp-teams and gather some stats?> > > > > > > > > > 
> > > > > > > > > > Thanks!

> > > > > > > > > > 2015-06-25 16:13 GMT+05:00 Vicente Gonzalez > > > > > > > > > > <vicente.gonzalez.ruiz@xxxxxxxxx>> > > > > > > > > > :
> > > > > > > > > > > 

> > > > > > > > > > > On Wed, Jun 24, 2015 at 5:48 PM L.G.Casado <leo@xxxxxx> wrote:> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > >   
> > > > > > > > > > > >   
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > Hi all,

> > > > > > > > > > > > 
> > > > > > > > > > > > El mié, 24-06-2015 a las 16:44 +0500, Ilshat Shakirov escribió:
> > > > > > > > > > > > 
> > > > > > > > > > > > Ok; Is there any option run peer without running a player? I'm going to run all peers in one local machine, is it right?
> > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > > > > > At this moment, the easiest way to test a lot of peers in one machine is to connect to each peer a NetCat client [http://netcat.sourceforge.net/]. It is not the most efficient solution, but you should be able to run hundreds of peers in a 8GB machine. However, is quite simple to avoid sending the stream in each peer. Just comment (temporally) the code that feeds the player.
> > > > > > > > > > >  
> > > > > > > > > > > Regards,
> > > > > > > > > > > Vi.
> > > > > > > > > > > -- 
> > > > > > > > > > > -- > > > > > > > > > > > Vicente González Ruiz
> > > > > > > > > > > Depto de Informática> > > > > > > > > > > Escuela Técnica Superior de Ingeniería
> > > > > > > > > > > Universidad de Almería
> > > > > > > > > > > 
> > > > > > > > > > > Carretera Sacramento S/N
> > > > > > > > > > > 04120, La Cañada de San Urbano
> > > > > > > > > > > Almería, España> > > > > > > > > > > 

> > > > > > > > > > > e-mail: vruiz@xxxxxx
> > > > > > > > > > > http://www.ual.es/~vruiz
> > > > > > > > > > > tel: +34 950 015711> > > > > > > > > > > fax: +34 950 015486
> > > > > > > > > > > 


> > > > > > > > > > 

> > > > > > > > > -- > > > > > > > > > -- > > > > > > > > > Vicente González Ruiz
> > > > > > > > > Depto de Informática> > > > > > > > > Escuela Técnica Superior de Ingeniería
> > > > > > > > > Universidad de Almería
> > > > > > > > > > > > > > > > > > Carretera Sacramento S/N
> > > > > > > > > 04120, La Cañada de San Urbano
> > > > > > > > > Almería, España> > > > > > > > > 

> > > > > > > > > e-mail: vruiz@xxxxxx
> > > > > > > > > http://www.ual.es/~vruiz> > > > > > > > > tel: +34 950 015711
> > > > > > > > > fax: +34 950 015486
> > > > > > > > > 



> > > > > > > > 

> > > > > > > 


> > > > > > 


> > > > > 



> > > > 

> > > 

> > > -- 
> > > Mailing list: https://launchpad.net/~p2psp
> > > 
> > > Post to     : p2psp@xxxxxxxxxxxxxxxxxxx
> > > 
> > > Unsubscribe : https://launchpad.net/~p2psp
> > > 
> > > More help   : https://help.launchpad.net/ListHelp> > > 



>