← Back to team overview

pyexiv2-developers team mailing list archive

Re: Pickling and multiprocessing

 

On Fri, Jan 21, 2011 at 4:43 AM, Olivier Tilloy <olivier@xxxxxxxxxx> wrote:

> I suppose that the daemon in charge of renaming photos only needs a
> rather small subset of the EXIF/IPTC/XMP metadata in order to proceed to
> renaming a photo. How about pickling and passing only this subset, e.g.
> as a dictionary (should be easy since tags can be pickled)?
> You’ll potentially save a lot of bandwidth in your inter-process
> communication.
>

Yes, of course you're right..... it's a splendidly good point :)


>
> > My preliminary testing using multiprocessing, queues and pipes indicates
> > that in the case of copying and renaming photos in parallel, the
> > scanning phase (determining what photos are at a location and loading
> > them into a TreeView to show the user) takes only 6% of the time it
> > takes to do the same thing with threads and locks. Clearly the
> > performance gains can be enormous.
>
> Impressive gain indeed, way to go!
>

I think there is much to be said for an approach that eschews shared state
where possible and instead relies on sending messages between processes.
It's a pity I was utterly ignorant of this fact when I started writing Rapid
Photo Downloader 4 years ago ;-) It  was my first attempt at program with a
GUI and my first time to learn about threads. Of course, the project would
not have been possible without all your impressive work on pyexiv2....

Best,
Damon
-- 
http://www.damonlynch.net

Follow ups

References