duplicity-team team mailing list archive

[Howard Kaye <question631423@xxxxxxxxxxxxxxxxxxxxx>]

Question #631423 on Duplicity changed:
https://answers.launchpad.net/duplicity/+question/631423

    Status: Answered => Open

Howard Kaye is still having a problem:
Getting it work is great, but making it not break in the future is key.

I caught the exception thrown by open in path.py:567.

I had it run lsof, sysctl sample, checked the getrlimit settings, even
bumped them up and retried.  The highest open file descriptor was 99.
There are 21 gpg processes.

command line is:

duplicity --archive-dir=/var/gbackup/cache/howie/restore
--encrypt-key=XXXXXXXX --log-fd=9 --verbosity=info --restore-time=0D
restore archivist://BBBBBBB/
/Users/howie/howie-macpro2.restore.2017-05-08-11-18-19

The way it breaks, I am starting to think there is a thread safety issue
somewhere in the file handling.

On Mon, May 8, 2017 at 3:57 PM, Kenneth Loafman <
question631423@xxxxxxxxxxxxxxxxxxxxx> wrote:

> Your question #631423 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/631423
>
>     Status: Open => Answered
>
> Kenneth Loafman proposed the following answer:
> Getting it to work is the major issue.  Making it work the way it should is
> kinda nice.
>
> When you do an *'lsof -p NNNN | wc -l*', where NNNN is duplicity's pid,
> you're getting a low count?
>
> Unless something is broken, you should only have 2-3 gpg tasks, no more.
>
> Where exactly did you break.  Did you check lsof then?
>
> What is your command line?
>
>
> On Mon, May 8, 2017 at 2:38 PM, Howard Kaye <
> question631423@xxxxxxxxxxxxxxxxxxxxx> wrote:
>
> > Question #631423 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/631423
> >
> >     Status: Answered => Open
> >
> > Howard Kaye is still having a problem:
> > Increasing the number of open files by 2 orders of magnitude allowed this
> > particular restore to proceed.
> >
> > In debugging the issue though, I tried catching the exception, doing an
> > os.open() in the handler, and this succeeds, though a subsequent call to
> > open() does not.
> > The process does not have an excessive number of open files.
> > So, the root cause of this does not appear to be actual open files, but
> > perhaps a bug somewhere else.
> >
> > Rather than mask it by increasing the ulimit for open files, I'd prefer
> to
> > find the root cause.
> >
> > On Mon, May 8, 2017 at 3:23 PM, Kenneth Loafman <
> > question631423@xxxxxxxxxxxxxxxxxxxxx> wrote:
> >
> > > Your question #631423 on Duplicity changed:
> > > https://answers.launchpad.net/duplicity/+question/631423
> > >
> > >     Status: Open => Answered
> > >
> > > Kenneth Loafman proposed the following answer:
> > > That bug applies only to runs made with --no-encryption and
> > > --no-compression.  He's got gpg processes, so he's not running that
> way.
> > >
> > > Also, that bug should only affect the memory use, some.
> > >
> > > Really should try my settings.
> > >
> > >
> > > On Mon, May 8, 2017 at 9:58 AM, Howard Kaye <
> > > question631423@xxxxxxxxxxxxxxxxxxxxx> wrote:
> > >
> > > > Question #631423 on Duplicity changed:
> > > > https://answers.launchpad.net/duplicity/+question/631423
> > > >
> > > >     Status: Answered => Open
> > > >
> > > > Howard Kaye is still having a problem:
> > > > That does not seem to be related to the open files.
> > > >
> > > > On Mon, May 8, 2017 at 6:43 AM, edso <question631423@answers.
> > > launchpad.net
> > > > >
> > > > wrote:
> > > >
> > > > > Your question #631423 on Duplicity changed:
> > > > > https://answers.launchpad.net/duplicity/+question/631423
> > > > >
> > > > > edso proposed the following answer:
> > > > > On 05.05.2017 20:13, Howard Kaye wrote:
> > > > > > New question #631423 on Duplicity:
> > > > > > https://answers.launchpad.net/duplicity/+question/631423
> > > > > >
> > > > > > In testing a full restore, I am seeing an exception get raised
> for
> > > "Too
> > > > > many open files" from within Path.open:
> > > > > >
> > > > > > I am running on macOS 10.12.4, duplicity 0.7.10
> > > > > >
> > > > > > I have raised the ulimit for the number of files up to 12000.
> > > > > > lsof shows there are only 89 open files
> > > > > > sample shows a number of threads blocked on subprocesses (gpg
> > > > processes)
> > > > > > sysctl -a shows:
> > > > > >   kern.maxfiles: 24576
> > > > > >   kern.maxfilesperproc: 10240
> > > > > >   kern.num_files: 5356
> > > > > >
> > > > > > There are currently only 17 incrementals following my latest
> full.
> > > We
> > > > > usually do monthly fulls.
> > > > > >
> > > > > > Is anyone else seeing this recently?
> > > > > >
> > > > >
> > > > > there's going to be a fix in 0.7.13, replacing the use of
> > os.system('cp
> > > > > ...') with the native python routine, which should reduce the open
> > > files
> > > > > count dramatically
> > > > >   http://duplicity.nongnu.org/CHANGELOG
> > > > >
> > > > > it's explained in more detail here
> > > > >   https://bugs.launchpad.net/duplicity/+bug/1684312
> > > > >
> > > > > ..ede/duply.net
> > > > >
> > > > > --
> > > > > If this answers your question, please go to the following page to
> let
> > > us
> > > > > know that it is solved:
> > > > > https://answers.launchpad.net/duplicity/+question/631423/+
> > > > > confirm?answer_id=4
> > > > >
> > > > > If you still need help, you can reply to this email or go to the
> > > > > following page to enter your feedback:
> > > > > https://answers.launchpad.net/duplicity/+question/631423
> > > > >
> > > > > You received this question notification because you asked the
> > question.
> > > > >
> > > >
> > > > --
> > > > You received this question notification because your team
> > duplicity-team
> > > > is an answer contact for Duplicity.
> > > >
> > > > _______________________________________________
> > > > Mailing list: https://launchpad.net/~duplicity-team
> > > > Post to     : duplicity-team@xxxxxxxxxxxxxxxxxxx
> > > > Unsubscribe : https://launchpad.net/~duplicity-team
> > > > More help   : https://help.launchpad.net/ListHelp
> > > >
> > >
> > > --
> > > If this answers your question, please go to the following page to let
> us
> > > know that it is solved:
> > > https://answers.launchpad.net/duplicity/+question/631423/+
> > > confirm?answer_id=6
> > >
> > > If you still need help, you can reply to this email or go to the
> > > following page to enter your feedback:
> > > https://answers.launchpad.net/duplicity/+question/631423
> > >
> > > You received this question notification because you asked the question.
> > >
> >
> > --
> > You received this question notification because your team duplicity-team
> > is an answer contact for Duplicity.
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~duplicity-team
> > Post to     : duplicity-team@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~duplicity-team
> > More help   : https://help.launchpad.net/ListHelp
> >
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/duplicity/+question/631423/+
> confirm?answer_id=8
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/duplicity/+question/631423
>
> You received this question notification because you asked the question.
>

-- 
You received this question notification because your team duplicity-team
is an answer contact for Duplicity.