duplicity-team team mailing list archive

[Kenneth Loafman <kenneth@xxxxxxxxxxx>]

Peter Schuller wrote:
> Some more incremental investigation:
> 
>> Traceback (most recent call last):
>>   File "./duplicity-bin", line 1197, in <module>
>>     with_tempdir(main)
>>   File "./duplicity-bin", line 1190, in with_tempdir
>>     fn()
>>   File "./duplicity-bin", line 1172, in main
>>     incremental_backup(sig_chain)
>>   File "./duplicity-bin", line 484, in incremental_backup
>>     globals.backend)
>>   File "./duplicity-bin", line 252, in write_multivol
>>     globals.restart.setLastSaved(mf)
>>   File "./duplicity-bin", line 1065, in setLastSaved
>>     vi = mf.volume_info_dict[self.start_vol]
>> KeyError: 0
> 
> In this case the relevant files are empty but existing:
> 
> -rw-r--r--  1 scode  scode  100902 Jul  7 18:41 duplicity-full-signatures.20090704T162538Z.sigtar.gz
> -rw-r--r--  1 scode  scode     199 Jul  7 18:41 duplicity-full.20090704T162538Z.manifest
> -rw-r--r--  1 scode  scode     181 Jul  7 18:41 duplicity-inc.20090704T162538Z.to.20090704T162600Z.manifest
> -rw-r--r--  1 scode  scode      20 Jul  7 18:41 duplicity-new-signatures.20090704T162538Z.to.20090704T162600Z.sigtar.gz
> -rw-------  1 scode  scode   41472 Jul  7 18:41 duplicity-new-signatures.20090704T162600Z.to.20090707T164117Z.sigtar.part
> -rw-------  1 scode  scode     254 Jul  7 18:41 duplicity-inc.20090704T162600Z.to.20090707T164117Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  7 18:41 duplicity-new-signatures.20090707T164117Z.to.20090707T164131Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  7 18:41 duplicity-inc.20090707T164117Z.to.20090707T164131Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  7 20:18 duplicity-new-signatures.20090707T164131Z.to.20090707T181854Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  7 20:18 duplicity-inc.20090707T164131Z.to.20090707T181854Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  7 20:19 duplicity-new-signatures.20090707T181854Z.to.20090707T181904Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  7 20:19 duplicity-inc.20090707T181854Z.to.20090707T181904Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  8 18:41 duplicity-new-signatures.20090707T181904Z.to.20090708T164157Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  8 18:41 duplicity-inc.20090707T181904Z.to.20090708T164157Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  8 18:46 duplicity-new-signatures.20090708T164157Z.to.20090708T164634Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  8 18:46 duplicity-inc.20090708T164157Z.to.20090708T164634Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  8 18:53 duplicity-new-signatures.20090708T164634Z.to.20090708T165356Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  8 18:53 duplicity-inc.20090708T164634Z.to.20090708T165356Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  8 19:12 duplicity-new-signatures.20090708T165356Z.to.20090708T171215Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  8 19:12 duplicity-inc.20090708T165356Z.to.20090708T171215Z.manifest.part
> -rw-------  1 scode  scode       0 Jul  8 23:37 duplicity-new-signatures.20090708T171215Z.to.20090708T213729Z.sigtar.part
> -rw-------  1 scode  scode       0 Jul  8 23:37 duplicity-inc.20090708T171215Z.to.20090708T213729Z.manifest.part
> 
> And indeed, the call to get_local_manifest() in duplicity-bin ends up
> dispatching to Manifest().from_string(), which receives an empty
> string, read from file.
> 
> As a result, the self.add_volume_info() in the loop.
> 
> If I understand things correctly (and I've always been fuzzy on these
> parts of the code), a "manifest" represents volume information for
> some set. In practice this is a set of volumes associated with a
> "backup", either a full backup or an incremental which is part of a
> chain.
> 
> In this case, the general issue is that the code fails for any
> manifest that is completely empty because we cannot obtain meta-data
> information for the "first" volume (start_vol) in
> Restart.setLastSaved().
> 
> Question: Is the bug that we should handle empty manifests, or is the
> bug that a manifast should never exist that is empty (other than as
> transient data structures in memory)?
> 
> Reading later on we have:
> 
>         # Checkpoint after each volume so restart has a place to restart.
>         # Note that until after the first volume, all files are temporary.
>         if vol_num == 1:
>             sig_outfp.to_partial()
>             man_outfp.to_partial()
>         else:
>             sig_outfp.flush()
>             man_outfp.flush()
> 
> This suggests to me the bug is the latter (since it's supposed to be
> invisible) rather than the former, though at this time I'm not sure
> why the attempt to make it invisible doesn't work.

You're on FreeBSD, right?  Are the semantics for file handling any
different than Linux?  The reason I ask is that there may be a problem
with the way I flush the file using the following sequence:
        self.fileobj.flush()
        os.fsync(self.fileobj.fileno())
which should do the job completely, however, it looks like its not
working on your system at all.  Is this possibly the problem?

It's possible that instead of flush(), we need to close and reopen,
which is what i was trying to avoid because of the high overhead.