launchpad-dev team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

On Jun 21, 2010, at 12:19 PM, Julian Edwards wrote:

> Hi folks
> 
> We had an incident on Friday where the PPA publisher fell over in a heap.  We 
> tracked this down the fact that it was trying to write an OOPS report to 
> somewhere it didn't have permission, which was easily fixed.
> 
> However this has revealed a separate problem which was the cause of the OOPS 
> in the first place.  Apparently there were two rows in the PackageUpload table 
> for the same thing which should be impossible!
> 
> I've done some sleuthing and it turns out that it was a package that was 
> copied from a private PPA to a public one.  The way we do this is to create a 
> PackageUpload record in the "accepted" state and then the process-accepted 
> script will work out that it's a package copy rather than an upload and copy 
> the files from the restricted librarian into the public one.  It also has a 
> check to see if the packages files are already accepted and if they are - it 
> blows up as we saw.
> 
> I managed to re-create this on dogfood, eventually, by clicking the "copy 
> packages" button twice in quick succession.  If I clicked it on separate page 
> loads, then the code that checks for the duplication works as it should.
> 
> Today I've tried to write a page test to re-create this by posting the form 
> twice from 2 separate browsers, and I can't make it fail, the checker code 
> works fine.
> 
> So, I've got 2 questions for you Foundations experts:
> 
> 1. How is this happening?

I'm guessing it is because of optimistic conflict resolution.  Here's the scenario.

- One transaction starts and modifies a record.
- Another transaction starts and modifies the same record the same way.
- The first transaction commits.
- The second transaction commits.  There would be a conflict, but optimistic conflict resolution says "oh, look, they both changed the same value at the same time!  No problem, I can resolve that!"

This is multi-version concurrency control, or MVCC, and it is at the DB level.  The ZODB has this behavior by default, and I know from past research that Postgres has similar behavior (configurable, but also configured by default, IIRC).

Stuart could verify both generally and for your specific case.  In the ZODB, the solution is to write your own conflict resolution for a given object that says "No, it's not OK to resolve this particular conflict!"  I don't know what the answer is for Postgres, and a quick Google search doesn't give me any immediate ideas ("conflict resolution rules"?).

> 2. How can I write a test for it?

If I'm right, and if you can do it just with storm and transaction calls, you should be able to dupe with non-threaded transaction managers.  Here's an example.  It's for the ZODB, but it's the identical use of transaction managers that you would use, given our set up.

http://svn.zope.org/zc.queue/trunk/src/zc/queue/queue.txt?rev=77086&view=auto

To do it through the browser would be much trickier.  I've been reduced to threads, threadlocks, and monkeypatches in the past when trying to test these things.  Hopefully you don't need to go there, either because you can show things happening in a low-level test like the example above, or because someone has a better idea.

Gary

> 
> Once these are answered I can work out how to fix it!
> 
> Cheers
> J
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~launchpad-dev
> Post to     : launchpad-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~launchpad-dev
> More help   : https://help.launchpad.net/ListHelp