duplicity-team team mailing list archive

Thread
Date
Re: [Duplicity-talk] verify fails on 0.6.18 and 0.7.01

To: Edgar Soldin <edgar.soldin@xxxxxx>
From: Kenneth Loafman <kenneth@xxxxxxxxxxx>
Date: Sun, 8 Feb 2015 10:32:40 -0600
Cc: Discussion of the backup program duplicity <duplicity-talk@xxxxxxxxxx>, duplicity-team <duplicity-team@xxxxxxxxxxxxxxxxxxx>
In-reply-to: <54D77DB2.3000505@web.de>
Sender: kenneth.loafman@xxxxxxxxx
Were there any errors during the initial backup?  This looks like a 32-bit
Python limit.

...Ken




On Sun, Feb 8, 2015 at 9:16 AM, <edgar.soldin@xxxxxx> wrote:

> On 08.02.2015 15:46, Rupert Levene wrote:
> > On 8 February 2015 at 14:32,  <edgar.soldin@xxxxxx> wrote:
> >> On 08.02.2015 15:05, Rupert Levene wrote:
> >>> Hi,
> >>>
> >>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB
> >>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was
> >>> created, and duplicity chokes on this. It pauses for a while (I guess
> >>> while it downloads the file) and then says
> >>>
> >>> Attempt 1 failed: BackendException: Failed to download file
> >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
> >>> folder 'i7-bigstore': join() result is too long for a Python string
> >>> Backtrace of previous error: Traceback (innermost last):
> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> >>> 311, in iterate
> >>>     return fn(*args, **kwargs)
> >>>   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> >>> line 144, in get
> >>>     % (remote_filename, self.folder.title.text, str(e)), raise_errors)
> >>>   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> >>> line 182, in __handle_error
> >>>     raise BackendException(message)
> >>>  BackendException: Failed to download file
> >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
> >>> folder 'i7-bigstore': join() result is too long for a Python string
> >>>
> >>> It'll then try again, and give the same error, at which point I hit
> control-C.
> >>>
> >>> Version 0.7.01 fails more quickly:
> >>>
> >>> $ apt-cache policy duplicity
> >>> duplicity:
> >>>   Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
> >>>   Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
> >>>   Version table:
> >>>  *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0
> >>>         500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/
> >>> precise/main i386 Packages
> >>>         100 /var/lib/dpkg/status
> >>>      0.6.18-0ubuntu3.5 0
> >>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main
> >>> i386 Packages
> >>>      0.6.18-0ubuntu3 0
> >>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386
> Packages
> >>> $ duplicity -V
> >>> duplicity 0.7.01
> >>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist
> >>> /home/rupert/.duply/i7-bigstore/exclude
> >>> gdocs://rupert.levene@xxxxxx/backups/duplicity/i7-bigstore
> >>> /mnt/bigstore
> >>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx
> >>> Using backup name: xxxxxxxxxxxxxxxxxxxxx
> >>> Import of duplicity.backends.azurebackend Succeeded
> >>> Import of duplicity.backends.botobackend Succeeded
> >>> Import of duplicity.backends.cfbackend Succeeded
> >>> Import of duplicity.backends.copycombackend Succeeded
> >>> Import of duplicity.backends.dpbxbackend Succeeded
> >>> Import of duplicity.backends.gdocsbackend Succeeded
> >>> Import of duplicity.backends.giobackend Succeeded
> >>> Import of duplicity.backends.hsibackend Succeeded
> >>> Import of duplicity.backends.hubicbackend Succeeded
> >>> Import of duplicity.backends.imapbackend Succeeded
> >>> Import of duplicity.backends.lftpbackend Succeeded
> >>> Import of duplicity.backends.localbackend Succeeded
> >>> Import of duplicity.backends.megabackend Succeeded
> >>> Import of duplicity.backends.ncftpbackend Succeeded
> >>> Import of duplicity.backends.onedrivebackend Failed: No module named
> requests
> >>> Import of duplicity.backends.par2backend Succeeded
> >>> Import of duplicity.backends.pydrivebackend Succeeded
> >>> Import of duplicity.backends.rsyncbackend Succeeded
> >>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
> >>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
> >>> Import of duplicity.backends.swiftbackend Succeeded
> >>> Import of duplicity.backends.sxbackend Succeeded
> >>> Import of duplicity.backends.tahoebackend Succeeded
> >>> Import of duplicity.backends.webdavbackend Succeeded
> >>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir
> >>> Traceback (most recent call last):
> >>>   File "/usr/bin/duplicity", line 1497, in <module>
> >>>     with_tempdir(main)
> >>>   File "/usr/bin/duplicity", line 1491, in with_tempdir
> >>>     fn()
> >>>   File "/usr/bin/duplicity", line 1324, in main
> >>>     action = commandline.ProcessCommandLine(sys.argv[1:])
> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
> >>> line 1056, in ProcessCommandLine
> >>>     backup, local_pathname = set_backend(args[0], args[1])
> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
> >>> line 949, in set_backend
> >>>     globals.backend = backend.get_backend(bend)
> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> >>> 221, in get_backend
> >>>     obj = get_backend_object(url_string)
> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> >>> 207, in get_backend_object
> >>>     return factory(pu)
> >>>   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> >>> line 61, in __init__
> >>>     entries = self._fetch_entries(parent_folder_id, 'folder',
> folder_name)
> >>>   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> >>> line 161, in _fetch_entries
> >>>     entries = self.client.get_all_resources(uri=uri)
> >>> AttributeError: 'DocsClient' object has no attribute
> 'get_all_resources'
> >>>
> >>> Are these two independent bugs? Is there a workaround or fix?
> >>>
> >>
> >> yes, it looks like two different bugs with two different error messages.
> >>
> >> first try latest stable duplicity 0.6.25.. there were changes in the
> gdocs backend since your version.
> >>
> >> if that fails - can you try downloading your whole backups to a local
> folder and try to restore from there?
> >>
> >> ..ede/duply.net
> >>
> >> _______________________________________________
> >> Duplicity-talk mailing list
> >> Duplicity-talk@xxxxxxxxxx
> >> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> > I had tried 0.6.25 too, and it was giving the same get_all_resources
> > error as 0.7.01. However since then I used pip to upgrade gdata, and
> > 0.7.01 gives me very similar output to my original error from 0.6.18:
> >
> > $ duplicity verify --verbosity '9' --exclude-globbing-filelist
> > /home/rupert/.duply/i7-bigstore/exclude
> > gdocs://rupert.levene@xxxxxx/backups/duplicity/i7-bigstore
> > /mnt/bigstore
> > Using archive dir:
> > /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > Import of duplicity.backends.azurebackend Succeeded
> > Import of duplicity.backends.botobackend Succeeded
> > Import of duplicity.backends.cfbackend Succeeded
> > Import of duplicity.backends.copycombackend Succeeded
> > Import of duplicity.backends.dpbxbackend Succeeded
> > Import of duplicity.backends.gdocsbackend Succeeded
> > Import of duplicity.backends.giobackend Succeeded
> > Import of duplicity.backends.hsibackend Succeeded
> > Import of duplicity.backends.hubicbackend Succeeded
> > Import of duplicity.backends.imapbackend Succeeded
> > Import of duplicity.backends.lftpbackend Succeeded
> > Import of duplicity.backends.localbackend Succeeded
> > Import of duplicity.backends.megabackend Succeeded
> > Import of duplicity.backends.ncftpbackend Succeeded
> > Import of duplicity.backends.onedrivebackend Failed: No module named
> requests
> > Import of duplicity.backends.par2backend Succeeded
> > Import of duplicity.backends.pydrivebackend Succeeded
> > Import of duplicity.backends.rsyncbackend Succeeded
> > Import of duplicity.backends.ssh_paramiko_backend Succeeded
> > Import of duplicity.backends.ssh_pexpect_backend Succeeded
> > Import of duplicity.backends.swiftbackend Succeeded
> > Import of duplicity.backends.sxbackend Succeeded
> > Import of duplicity.backends.tahoebackend Succeeded
> > Import of duplicity.backends.webdavbackend Succeeded
> > Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude
> > Main action: verify
> >
> ================================================================================
> > duplicity 0.7.01 ($reldate)
> > Args: /usr/bin/duplicity verify --verbosity 9
> > --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude
> > gdocs://rupert.levene@xxxxxx/backups/duplicity/i7-bigstore
> > /mnt/bigstore
> > Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed
> > Dec 17 00:38:38 UTC 2014 i686 i686
> > /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52)
> > [GCC 4.6.3]
> >
> ================================================================================
> > Synchronizing remote metadata to local cache...
> > Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local
> cache.
> > Using temporary directory /tmp/duplicity-L9vwHt-tempdir
> > Registering (mktemp) temporary file
> > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
> > Backtrace of previous error: Traceback (innermost last):
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> > 361, in inner_retry
> >     return fn(self, *args)
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 534, in get
> >     self.backend._get(remote_filename, local_path)
> >   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> > line 110, in _get
> >     self.client.DownloadResource(entry, local_path.name)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 369, in download_resource
> >     self._download_file(uri, file_path, **kwargs)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 468, in _download_file
> >     f.write(self._get_content(uri, **kwargs))
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 451, in _get_content
> >     return server_response.read()
> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
> >     return self._read_chunked(amt)
> >   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
> >     return ''.join(value)
> >  OverflowError: join() result is too long for a Python string
> >
> > Attempt 1 failed. OverflowError: join() result is too long for a Python
> string
> > Backtrace of previous error: Traceback (innermost last):
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> > 361, in inner_retry
> >     return fn(self, *args)
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 534, in get
> >     self.backend._get(remote_filename, local_path)
> >   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> > line 110, in _get
> >     self.client.DownloadResource(entry, local_path.name)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 369, in download_resource
> >     self._download_file(uri, file_path, **kwargs)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 468, in _download_file
> >     f.write(self._get_content(uri, **kwargs))
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 451, in _get_content
> >     return server_response.read()
> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
> >     return self._read_chunked(amt)
> >   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
> >     value.append(self._safe_read(chunk_left))
> >   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
> >     chunk = self.fp.read(min(amt, MAXAMOUNT))
> >   File "/usr/lib/python2.7/socket.py", line 396, in read
> >     buf.write(data)
> >  MemoryError: out of memory
> >
> > Attempt 2 failed. MemoryError: out of memory
> > Backtrace of previous error: Traceback (innermost last):
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> > 361, in inner_retry
> >     return fn(self, *args)
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 534, in get
> >     self.backend._get(remote_filename, local_path)
> >   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> > line 110, in _get
> >     self.client.DownloadResource(entry, local_path.name)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 369, in download_resource
> >     self._download_file(uri, file_path, **kwargs)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 468, in _download_file
> >     f.write(self._get_content(uri, **kwargs))
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 451, in _get_content
> >     return server_response.read()
> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
> >     return self._read_chunked(amt)
> >   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
> >     return ''.join(value)
> >  OverflowError: join() result is too long for a Python string
> >
> > Attempt 3 failed. OverflowError: join() result is too long for a Python
> string
> > Backtrace of previous error: Traceback (innermost last):
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> > 361, in inner_retry
> >     return fn(self, *args)
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 534, in get
> >     self.backend._get(remote_filename, local_path)
> >   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> > line 110, in _get
> >     self.client.DownloadResource(entry, local_path.name)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 369, in download_resource
> >     self._download_file(uri, file_path, **kwargs)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 468, in _download_file
> >     f.write(self._get_content(uri, **kwargs))
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 451, in _get_content
> >     return server_response.read()
> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
> >     return self._read_chunked(amt)
> >   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
> >     value.append(self._safe_read(chunk_left))
> >   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
> >     chunk = self.fp.read(min(amt, MAXAMOUNT))
> >   File "/usr/lib/python2.7/socket.py", line 400, in read
> >     buf.write(data)
> >  MemoryError: out of memory
> >
> > Attempt 4 failed. MemoryError: out of memory
> > Backtrace of previous error: Traceback (innermost last):
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> > 361, in inner_retry
> >     return fn(self, *args)
> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 534, in get
> >     self.backend._get(remote_filename, local_path)
> >   File
> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> > line 110, in _get
> >     self.client.DownloadResource(entry, local_path.name)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 369, in download_resource
> >     self._download_file(uri, file_path, **kwargs)
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 468, in _download_file
> >     f.write(self._get_content(uri, **kwargs))
> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> > line 451, in _get_content
> >     return server_response.read()
> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
> >     return self._read_chunked(amt)
> >   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
> >     return ''.join(value)
> >  OverflowError: join() result is too long for a Python string
> >
> > Giving up after 5 attempts. OverflowError: join() result is too long
> > for a Python string
> > Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c>
> > Removing still remembered temporary file
> > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
> > $
> >
> > I am running 32-bit linux with 4gb RAM and 4gb swap.
> >
> > I'll try downloading the backup to local storage as you suggest.
> >
>
> please do.. looks like httplib fails on 2GB files and there isn't much we
> can do about that.
>
> Mike, Ken: any idea?
>
> ..ede/duply.net
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to     : duplicity-team@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help   : https://help.launchpad.net/ListHelp
>
References

Re: [Duplicity-talk] verify fails on 0.6.18 and 0.7.01
From: edgar . soldin, 2015-02-08