duplicity-team team mailing list archive

[Michael Terry <michael.terry@xxxxxxxxxxxxx>]

Michael Terry has proposed merging lp:~mterry/duplicity/look-at-partials-during-sync into lp:duplicity.

Requested reviews:
  duplicity-team (duplicity-team)
Related bugs:
  Bug #703142 in Duplicity: "AssertionError: assert len(chain_list) == 2"
  https://bugs.launchpad.net/duplicity/+bug/703142

For more details, see:
https://code.launchpad.net/~mterry/duplicity/look-at-partials-during-sync/+merge/67375

To fix bug 703142, we need to pay attention to local partial metadata files (see comment 8 in that bug).

This patch makes sure (A) that when we have a local-partial, we don't copy remote->local-final and (B) that we delete a local-final if we have a local-partial already.

It's fine to not have a local-final in these situations because clearly we got interrupted during the signature writing.  Which means we still have a partial manifest hanging around, we will resume the backup, and the last volume will be re-uploaded.  Thus, the signature and manifest will get re-uploaded anyway.  So any remote or local-final signature at the start of the run is forgettable.

This patch also ensures that we run sync_archives() during a collection-status.  But it makes sure to avoid a regression of bug 777377 by adding an argument that allows decryption or not.  Without running sync_archives during a collection-status, we'd still hit bug 703142 during that operation.
-- 
https://code.launchpad.net/~mterry/duplicity/look-at-partials-during-sync/+merge/67375
Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/look-at-partials-during-sync into lp:duplicity.

=== modified file 'duplicity-bin'
--- duplicity-bin	2011-06-27 11:18:01 +0000
+++ duplicity-bin	2011-07-08 19:27:10 +0000
@@ -815,7 +815,7 @@
                    _("Rerun command with --force option to actually delete."))
 
 
-def sync_archive():
+def sync_archive(decrypt):
     """
     Synchronize local archive manifest file and sig chains to remote archives.
     Copy missing files from remote to local as needed to make sure the local
@@ -824,7 +824,7 @@
     @rtype: void
     @return: void
     """
-    suffixes = [".g", ".gpg", ".z", ".gz"]
+    suffixes = [".g", ".gpg", ".z", ".gz", ".part"]
 
     def get_metafiles(filelist):
         """
@@ -832,30 +832,31 @@
         Files of interest are:
           sigtar - signature files
           manifest - signature files
+          duplicity partial versions of the above
         Files excluded are:
           non-duplicity files
-          duplicity partial files
 
         @rtype: list
         @return: list of duplicity metadata files
         """
         metafiles = {}
+        partials = {}
         need_passphrase = False
         for fn in filelist:
             pr = file_naming.parse(fn)
             if not pr:
                 continue
-            if pr.partial:
-                continue
             if pr.encrypted:
                 need_passphrase = True
             if pr.type in ["full-sig", "new-sig"] or pr.manifest:
                 base, ext = os.path.splitext(fn)
-                if ext in suffixes:
+                if ext not in suffixes:
+                    base = fn
+                if pr.partial:
+                    partials[base] = fn
+                else:
                     metafiles[base] = fn
-                else:
-                    metafiles[fn] = fn
-        return metafiles, need_passphrase
+        return metafiles, partials, need_passphrase
 
     def copy_raw(src_iter, filename):
         """
@@ -954,11 +955,11 @@
 
     # get remote metafile list
     remlist = globals.backend.list()
-    remote_metafiles, rem_needpass = get_metafiles(remlist)
+    remote_metafiles, ignored, rem_needpass = get_metafiles(remlist)
 
     # get local metafile list
     loclist = globals.archive_dir.listdir()
-    local_metafiles, loc_needpass = get_metafiles(loclist)
+    local_metafiles, local_partials, loc_needpass = get_metafiles(loclist)
 
     # we have the list of metafiles on both sides. remote is always
     # authoritative. figure out which are local spurious (should not
@@ -970,11 +971,18 @@
     local_spurious = []
 
     for key in remote_keys:
-        if not key in local_keys:
+        # If we lost our cache, re-get the remote file.  But don't do it if we
+        # already have a local partial.  The local partial will already be
+        # complete in this case (seems we got interrupted before we could move
+        # it to its final location).
+        if key not in local_keys and key not in local_partials:
             local_missing.append(key)
 
     for key in local_keys:
-        if not key in remote_keys:
+        # If we have a file locally that is unnecessary, delete it.  Also
+        # delete final versions of partial files because if we have both, it
+        # means the write of the final version got interrupted.
+        if key not in remote_keys or key in local_partials:
             local_spurious.append(key)
 
     # finally finish the process
@@ -986,8 +994,11 @@
         if not globals.dry_run:
             log.Notice(_("Synchronizing remote metadata to local cache..."))
             if local_missing and (rem_needpass or loc_needpass):
-                # password for the --encrypt-key
-                globals.gpg_profile.passphrase = get_passphrase(1, "sync")
+                if decrypt:
+                    # password for the --encrypt-key
+                    globals.gpg_profile.passphrase = get_passphrase(1, "sync")
+                else:
+                    local_missing = [] # don't download if we can't decrypt
             for fn in local_spurious:
                 remove_local(fn)
             for fn in local_missing:
@@ -1176,8 +1187,8 @@
     check_resources(action)
 
     # check archive synch with remote, fix if needed
-    if action not in ["collection-status"]:
-        sync_archive()
+    decrypt = action not in ["collection-status"]
+    sync_archive(decrypt)
 
     # get current collection status
     col_stats = collections.CollectionsStatus(globals.backend,
@@ -1249,7 +1260,7 @@
     elif action == "remove-all-but-n-full" or action == "remove-all-inc-of-but-n-full":
         remove_all_but_n_full(col_stats)
     elif action == "sync":
-        sync_archive(col_stats)
+        sync_archive(True)
     else:
         assert action == "inc" or action == "full", action
         # the passphrase for full and inc is used by --sign-key