duplicity-team team mailing list archive

[Howard Kaye <question631423@xxxxxxxxxxxxxxxxxxxxx>]

Question #631423 on Duplicity changed:
https://answers.launchpad.net/duplicity/+question/631423

Howard Kaye posted a new comment:
I think it just moved the error somewhere else.

On May 17, 2017 4:09 AM, "edso" <question631423@xxxxxxxxxxxxxxxxxxxxx>
wrote:

> Your question #631423 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/631423
>
> edso posted a new comment:
> nice catch Howie!
>
> wonder why this worked with os.tmpfile() but not with
> tempfile.TemporaryFile() before though.. ede/duply.net
>
> On 16.05.2017 20:33, Howard Kaye wrote:
> > Question #631423 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/631423
> >
> >     Status: Answered => Solved
> >
> > Howard Kaye confirmed that the question is solved:
> > OK, I found and fixed the root cause of this error:
> >
> > Duplicity was failing in the glue code (in C), between Python and
> librsync.
> > It did an fdopen() on the file descriptor from a Python File object, and
> then never
> > closed the File * which was returned.  Even though the Python File
> object (and the
> > file descriptor shared by the File * and the Python File) got closed
> when it was
> > deallocated, the File *'s were being leaked, and eventually the stdio
> library ran
> > out of File *'s (even though the number of open files was not that
> large).
> >
> > The fix is to dup the file descriptor, and then close the file in the
> deallocator
> > routine in the glue code.  Duping the file lets the C code and the
> Python code
> > each close the file, when they are done with it.
> >
> > *** _librsyncmodule.c.orig      2017-01-06 09:23:21.000000000 -0500
> > --- _librsyncmodule.c   2017-05-16 11:12:58.000000000 -0400
> > ***************
> > *** 23,28 ****
> > --- 23,29 ----
> >    * -----------------------------------------------------------------------
> */
> >
> >   #include <Python.h>
> > + #include <errno.h>
> >   #include <librsync.h>
> >   #define RS_JOB_BLOCKSIZE 65536
> >
> > ***************
> > *** 287,292 ****
> > --- 288,294 ----
> >     PyObject_HEAD
> >     rs_job_t *patch_job;
> >     PyObject *basis_file;
> > +   FILE *cfile;
> >   } _librsync_PatchMakerObject;
> >
> >   /* Call with the basis file */
> > ***************
> > *** 296,302 ****
> >     _librsync_PatchMakerObject* pm;
> >     PyObject *python_file;
> >     int fd;
> > -   FILE *cfile;
> >
> >     if (!PyArg_ParseTuple(args, "O:new_patchmaker", &python_file))
> >       return NULL;
> > --- 298,303 ----
> > ***************
> > *** 305,318 ****
> >       PyErr_SetString(PyExc_TypeError, "Need true file object");
> >       return NULL;
> >     }
> >     Py_INCREF(python_file);
> >
> >     pm = PyObject_New(_librsync_PatchMakerObject,
> &_librsync_PatchMakerType);
> >     if (pm == NULL) return NULL;
> >
> >     pm->basis_file = python_file;
> > !   cfile = fdopen(fd, "rb");
> > !   pm->patch_job = rs_patch_begin(rs_file_copy_cb, cfile);
> >
> >     return (PyObject*)pm;
> >   }
> > --- 306,327 ----
> >       PyErr_SetString(PyExc_TypeError, "Need true file object");
> >       return NULL;
> >     }
> > +   /* get our own private copy of the file, so we can close it later. */
> > +   fd = dup(fd);
> > +   if (fd == -1) {
> > +     char buf[256];
> > +     strerror_r(errno, buf, sizeof(buf));
> > +     PyErr_SetString(PyExc_TypeError, buf);
> > +     return NULL;
> > +   }
> >     Py_INCREF(python_file);
> >
> >     pm = PyObject_New(_librsync_PatchMakerObject,
> &_librsync_PatchMakerType);
> >     if (pm == NULL) return NULL;
> >
> >     pm->basis_file = python_file;
> > !   pm->cfile = fdopen(fd, "rb");
> > !   pm->patch_job = rs_patch_begin(rs_file_copy_cb, pm->cfile);
> >
> >     return (PyObject*)pm;
> >   }
> > ***************
> > *** 323,328 ****
> > --- 332,340 ----
> >     _librsync_PatchMakerObject *pm = (_librsync_PatchMakerObject *)self;
> >     Py_DECREF(pm->basis_file);
> >     rs_job_free(pm->patch_job);
> > +   if (pm->cfile) {
> > +     fclose(pm->cfile);
> > +   }
> >     PyObject_Del(self);
> >   }
> >
>
> --
> You received this question notification because you asked the question.
>

-- 
You received this question notification because your team duplicity-team
is an answer contact for Duplicity.