duplicity-team team mailing list archive

[edso <question631423@xxxxxxxxxxxxxxxxxxxxx>]

Question #631423 on Duplicity changed:
https://answers.launchpad.net/duplicity/+question/631423

edso posted a new comment:
Howie, Ken,

would you agree then, that the whole os.tmpfile() exception

http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/patchdir.py#L498
"  
if sys.platform.startswith(('cygwin', 'windows', 'darwin')):
  tempfp = os.tmpfile()
else:
"

can be removed after the librsync adapter is patched?.. ede

On 17.05.2017 12:27, Howard Kaye wrote:
> Question #631423 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/631423
> 
> Howard Kaye posted a new comment:
> I think it just moved the error somewhere else.
> 
> On May 17, 2017 4:09 AM, "edso" <question631423@xxxxxxxxxxxxxxxxxxxxx>
> wrote:
> 
>> Your question #631423 on Duplicity changed:
>> https://answers.launchpad.net/duplicity/+question/631423
>>
>> edso posted a new comment:
>> nice catch Howie!
>>
>> wonder why this worked with os.tmpfile() but not with
>> tempfile.TemporaryFile() before though.. ede/duply.net
>>
>> On 16.05.2017 20:33, Howard Kaye wrote:
>>> Question #631423 on Duplicity changed:
>>> https://answers.launchpad.net/duplicity/+question/631423
>>>
>>>     Status: Answered => Solved
>>>
>>> Howard Kaye confirmed that the question is solved:
>>> OK, I found and fixed the root cause of this error:
>>>
>>> Duplicity was failing in the glue code (in C), between Python and
>> librsync.
>>> It did an fdopen() on the file descriptor from a Python File object, and
>> then never
>>> closed the File * which was returned.  Even though the Python File
>> object (and the
>>> file descriptor shared by the File * and the Python File) got closed
>> when it was
>>> deallocated, the File *'s were being leaked, and eventually the stdio
>> library ran
>>> out of File *'s (even though the number of open files was not that
>> large).
>>>
>>> The fix is to dup the file descriptor, and then close the file in the
>> deallocator
>>> routine in the glue code.  Duping the file lets the C code and the
>> Python code
>>> each close the file, when they are done with it.
>>>
>>> *** _librsyncmodule.c.orig      2017-01-06 09:23:21.000000000 -0500
>>> --- _librsyncmodule.c   2017-05-16 11:12:58.000000000 -0400
>>> ***************
>>> *** 23,28 ****
>>> --- 23,29 ----
>>>    * -----------------------------------------------------------------------
>> */
>>>
>>>   #include <Python.h>
>>> + #include <errno.h>
>>>   #include <librsync.h>
>>>   #define RS_JOB_BLOCKSIZE 65536
>>>
>>> ***************
>>> *** 287,292 ****
>>> --- 288,294 ----
>>>     PyObject_HEAD
>>>     rs_job_t *patch_job;
>>>     PyObject *basis_file;
>>> +   FILE *cfile;
>>>   } _librsync_PatchMakerObject;
>>>
>>>   /* Call with the basis file */
>>> ***************
>>> *** 296,302 ****
>>>     _librsync_PatchMakerObject* pm;
>>>     PyObject *python_file;
>>>     int fd;
>>> -   FILE *cfile;
>>>
>>>     if (!PyArg_ParseTuple(args, "O:new_patchmaker", &python_file))
>>>       return NULL;
>>> --- 298,303 ----
>>> ***************
>>> *** 305,318 ****
>>>       PyErr_SetString(PyExc_TypeError, "Need true file object");
>>>       return NULL;
>>>     }
>>>     Py_INCREF(python_file);
>>>
>>>     pm = PyObject_New(_librsync_PatchMakerObject,
>> &_librsync_PatchMakerType);
>>>     if (pm == NULL) return NULL;
>>>
>>>     pm->basis_file = python_file;
>>> !   cfile = fdopen(fd, "rb");
>>> !   pm->patch_job = rs_patch_begin(rs_file_copy_cb, cfile);
>>>
>>>     return (PyObject*)pm;
>>>   }
>>> --- 306,327 ----
>>>       PyErr_SetString(PyExc_TypeError, "Need true file object");
>>>       return NULL;
>>>     }
>>> +   /* get our own private copy of the file, so we can close it later. */
>>> +   fd = dup(fd);
>>> +   if (fd == -1) {
>>> +     char buf[256];
>>> +     strerror_r(errno, buf, sizeof(buf));
>>> +     PyErr_SetString(PyExc_TypeError, buf);
>>> +     return NULL;
>>> +   }
>>>     Py_INCREF(python_file);
>>>
>>>     pm = PyObject_New(_librsync_PatchMakerObject,
>> &_librsync_PatchMakerType);
>>>     if (pm == NULL) return NULL;
>>>
>>>     pm->basis_file = python_file;
>>> !   pm->cfile = fdopen(fd, "rb");
>>> !   pm->patch_job = rs_patch_begin(rs_file_copy_cb, pm->cfile);
>>>
>>>     return (PyObject*)pm;
>>>   }
>>> ***************
>>> *** 323,328 ****
>>> --- 332,340 ----
>>>     _librsync_PatchMakerObject *pm = (_librsync_PatchMakerObject *)self;
>>>     Py_DECREF(pm->basis_file);
>>>     rs_job_free(pm->patch_job);
>>> +   if (pm->cfile) {
>>> +     fclose(pm->cfile);
>>> +   }
>>>     PyObject_Del(self);
>>>   }
>>>
>>
>> --
>> You received this question notification because you asked the question.
>>
>

-- 
You received this question notification because your team duplicity-team
is an answer contact for Duplicity.