duplicity-team team mailing list archive

[Byron <question687576@xxxxxxxxxxxxxxxxxxxxx>]

Question #687576 on Duplicity changed:
https://answers.launchpad.net/duplicity/+question/687576

Byron posted a new comment:
Ok, sorry. I saw the python2 shebangs and missed the python version
check in setup.py. I see now.

I've made a set of preliminary changes that should cover all the cases
for compression at least. I have also run some preliminary tests just to
show my suspicions...

Here is the original duplicity run (version 0.8.04)
byron@Desktop:~/src/OpenSource$ time duplicity full --no-encryption /home/byron/VirtualBox\ VMs/Ubuntu\ x64\ -\ ReachCMS/ file:///tmp/VMBackup-gzip
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
--------------[ Backup Statistics ]--------------
StartTime 1577528067.50 (Sat Dec 28 18:14:27 2019)
EndTime 1577528225.29 (Sat Dec 28 18:17:05 2019)
ElapsedTime 157.79 (2 minutes 37.79 seconds)
SourceFiles 10
SourceFileSize 5597132154 (5.21 GB)
NewFiles 10
NewFileSize 5597132154 (5.21 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 10
RawDeltaSize 5597123962 (5.21 GB)
TotalDestinationSizeChange 1602344515 (1.49 GB)
Errors 0
-------------------------------------------------


real	2m40.195s
user	2m35.594s
sys	0m3.209s


Here is the same backup but with my mgzip modifications

(venv) byron@Desktop:~/src/OpenSource$ time python3 venv/bin/duplicity full --no-encryption /home/byron/VirtualBox\ VMs/Ubuntu\ x64\ -\ ReachCMS/ file:///tmp/VMBackup
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
--------------[ Backup Statistics ]--------------
StartTime 1577527973.04 (Sat Dec 28 18:12:53 2019)
EndTime 1577528010.94 (Sat Dec 28 18:13:30 2019)
ElapsedTime 37.90 (37.90 seconds)
SourceFiles 10
SourceFileSize 5597132154 (5.21 GB)
NewFiles 10
NewFileSize 5597132154 (5.21 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 10
RawDeltaSize 5597123962 (5.21 GB)
TotalDestinationSizeChange 1609150119 (1.50 GB)
Errors 0
-------------------------------------------------


real	0m43.352s
user	2m49.017s
sys	0m5.035s


This is a substantial speed increase and is expected on my 6x core machine.


I will need some assistance now though as restoring the mgzip backup has a problem. The backup itself looks ok. I can look inside and test the integrity of the difftar.gz files and they seem ok. The difftar files all seem ok inside as well (with a cursory look). Same with the sigtar.

(venv) byron@Desktop:/tmp$ time python3 ~/src/OpenSource/venv/bin/duplicity restore --no-encryption file:///tmp/VMBackup/ /tmp/VMRestore
Synchronising remote metadata to local cache...
Copying duplicity-full-signatures.20191228T101253Z.sigtar.gz to local cache.
Copying duplicity-full.20191228T101253Z.manifest to local cache.
Last full backup date: Sat Dec 28 18:12:53 2019
Traceback (innermost last):
  File "/home/byron/src/OpenSource/venv/bin/duplicity", line 101, in <module>
    with_tempdir(main)
  File "/home/byron/src/OpenSource/venv/bin/duplicity", line 87, in with_tempdir
    fn()
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/dup_main.py", line 1539, in main
    do_backup(action)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/dup_main.py", line 1619, in do_backup
    restore(col_stats)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/dup_main.py", line 725, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/patchdir.py", line 580, in Write_ROPaths
    ITR(ropath.index, ropath)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/lazy.py", line 362, in __call__
    last_branch.fast_process, args)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/robust.py", line 41, in check_common_error
    return function(*args)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/patchdir.py", line 634, in fast_process
    ropath.copy(self.base_path.new_index(index))
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/path.py", line 456, in copy
    other.writefileobj(self.open(u"rb"))
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/path.py", line 652, in writefileobj
    buf = fin.read(_copy_blocksize)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/patchdir.py", line 227, in read
    if not self.addtobuffer():
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/patchdir.py", line 252, in addtobuffer
    self.tarinfo_list[0] = next(self.tar_iter)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/site-packages/duplicity/patchdir.py", line 359, in __next__
    return next(self.tar_iter)
  File "/home/byron/src/OpenSource/venv/lib/python3.7/tarfile.py", line 2405, in __iter__
    tarinfo = self.next()
  File "/home/byron/src/OpenSource/venv/lib/python3.7/tarfile.py", line 2283, in next
    raise ReadError("unexpected end of data")
 tarfile.ReadError: unexpected end of data


real	0m6.475s
user	0m7.736s
sys	0m2.092s

-- 
You received this question notification because your team duplicity-team
is an answer contact for Duplicity.