pyexiv2-developers team mailing list archive
-
pyexiv2-developers team
-
Mailing list archive
-
Message #00079
Re: Pickling and multiprocessing
On Fri, Jan 21, 2011 at 4:43 AM, Olivier Tilloy <olivier@xxxxxxxxxx> wrote:
> I suppose that the daemon in charge of renaming photos only needs a
> rather small subset of the EXIF/IPTC/XMP metadata in order to proceed to
> renaming a photo. How about pickling and passing only this subset, e.g.
> as a dictionary (should be easy since tags can be pickled)?
> You’ll potentially save a lot of bandwidth in your inter-process
> communication.
>
Yes, of course you're right..... it's a splendidly good point :)
>
> > My preliminary testing using multiprocessing, queues and pipes indicates
> > that in the case of copying and renaming photos in parallel, the
> > scanning phase (determining what photos are at a location and loading
> > them into a TreeView to show the user) takes only 6% of the time it
> > takes to do the same thing with threads and locks. Clearly the
> > performance gains can be enormous.
>
> Impressive gain indeed, way to go!
>
I think there is much to be said for an approach that eschews shared state
where possible and instead relies on sending messages between processes.
It's a pity I was utterly ignorant of this fact when I started writing Rapid
Photo Downloader 4 years ago ;-) It was my first attempt at program with a
GUI and my first time to learn about threads. Of course, the project would
not have been possible without all your impressive work on pyexiv2....
Best,
Damon
--
http://www.damonlynch.net
Follow ups
References