duplicity-team team mailing list archive
-
duplicity-team team
-
Mailing list archive
-
Message #02474
[Merge] lp:~ed.so/duplicity/0.6-dpbx.importfix into lp:duplicity
edso has proposed merging lp:~ed.so/duplicity/0.6-dpbx.importfix into lp:duplicity.
Requested reviews:
duplicity-team (duplicity-team)
For more details, see:
https://code.launchpad.net/~ed.so/duplicity/0.6-dpbx.importfix/+merge/238422
fix this showstopper with the dropbox backend
"
...
File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/dpbxbackend.py", line 162, in login
except rest.ErrorResponse, e:
NameError: global name 'rest' is not defined
"
--
The attached diff has been truncated due to its size.
https://code.launchpad.net/~ed.so/duplicity/0.6-dpbx.importfix/+merge/238422
Your team duplicity-team is requested to review the proposed merge of lp:~ed.so/duplicity/0.6-dpbx.importfix into lp:duplicity.
=== modified file '.bzrignore'
--- .bzrignore 2014-05-09 15:20:31 +0000
+++ .bzrignore 2014-10-15 12:12:04 +0000
@@ -2,10 +2,9 @@
.project
.pydevproject
.settings
-.tox
+config.py
+testfiles
+random_seed
build
-config.py
duplicity.egg-info
-duplicity.spec
-random_seed
-testfiles
+.tox
=== modified file 'CHANGELOG'
--- CHANGELOG 2014-10-12 17:13:31 +0000
+++ CHANGELOG 2014-10-15 12:12:04 +0000
@@ -1,5 +1,10 @@
+<<<<<<< TREE
New in v0.7.00 (2014/??/??)
+=======
+New in v0.6.25 (2014/??/??)
+>>>>>>> MERGE-SOURCE
---------------------------
+<<<<<<< TREE
* Merged in lp:~jflaker/duplicity/BugFix1325215
- The reference to "--progress_rate" in the man page as a parameter is
incorrect. Should be "--progress-rate".
@@ -101,32 +106,29 @@
delete a not-existing files, and it requires the backend not to raise an
exception in that case (is this somewhat wanted or could we do the same as
for _delete or _query?)
+=======
+Enhancements:
+* Merged in lp:~jflaker/duplicity/BugFix1325215
+ - The reference to "--progress_rate" in the man page as a parameter is
+ incorrect. Should be "--progress-rate".
+* Merged in lp:~hooloovoo/duplicity/updated-README-REPO
+ - Changes to README-REPO to reflect the restructuring of the directories.
+* Fixed bug 1375304 with patch supplied by Aleksandar Ivanovic
+>>>>>>> MERGE-SOURCE
* Merged in lp:~ed.so/duplicity/webdav200fix-0.7
- webdav backend fix "BackendException: Bad status code 200 reason OK. " when
restarting an interrupted backup and overwriting partially uploaded volumes.
-* Merged in lp:~mterry/duplicity/webdav-fixes
- - This branch fixes two issues I saw when testing the webdav backend:
- - 1) Errors like the following: "Attempt 1 failed. BackendException: File
- /tmp/duplicity-LQ1a0i-tempdir/mktemp-u2aiyX-2 not found locally after get
- from backend". These were caused by the _get() method not calling setdata()
- on the local path object, so the rest of the code thought it didn't exist.
- - 2) Some odd issues from stale responses/data. We have a couple places in
- webdavbackend.py where we close the connection before making a request
- because of this problem. But I've changed it to do it every time, more
- reliably, by putting a _close() call inside the request() method.
- - With this, the webdav backend seems fine to me.
-* Merged in lp:~antmak/duplicity/0.7-par2-fix
+* Merged in lp:~antmak/duplicity/0.6-par2-fix
- Useful fix for verbatim par2cmdline options (like "-t" in par2-tbb version)
+* Merged in lp:~jon-haggblad/duplicity/ftps-fix
+ - Minor bugfix for ftps backend where lftp needs host prefixed by "ftps://".
* Fixed bug 1327550: OverflowError: signed integer is greater than maximum
- Major and minor device numbers are supposed to be one byte each. Someone
has crafted a special system image using OpenVZ where the major and minor
device numbers are much larger (ploop devices). We treat them as (0,0).
-* Added sxbacked.py, Skylable backend. Waiting on man page updates.
-* Merged in lp:~ed.so/duplicity/manpage.verify
- - Clarify verify's functionality as wished for by a user surprised with a big
- bandwidth bill from rackspace.
-* Merged in lp:~jeffreydavidrogers/duplicity/duplicity
- - This change fixes two small typos in the duplicity man page.
+* Merged in lp:~jon-haggblad/duplicity/ftps-fix (2nd try)
+ - Minor bugfix for ftps backend where lftp needs host prefixed by "ftps://".
+
New in v0.6.24 (2014/05/09)
---------------------------
@@ -167,39 +169,25 @@
- Fixes https://bugs.launchpad.net/duplicity/+bug/426282
* Merged in lp:~fredrik-loch/duplicity/duplicity-S3-SSE
- Adds support for server side encryption as requested in Bug #996660
+* Merged in lp:~mterry/duplicity/modern-testing
+ - Enable/use more modern testing tools like nosetests and tox as well as more
+ common setup.py hooks like test and sdist.
+ - Specifically:
+ * move setup.py to toplevel where most tools and users expect it
+ * Move and adjust test files to work when running "nosetests" in toplevel
+ directory. Specifically, do a lot more of the test setup in
+ tests/__init__.py rather than the run-tests scripts
+ * Add small tox.ini file for using tox, which is a wrapper for using
+ virtualenv. Only enable for py26 and py27 right now, since modern
+ setuptools dropped support for <2.6 (and tox 1.7 recently dropped <2.6)
+ * Add setup.py hooks for test and sdist which are both standard targets
+ (sdist just outsources to dist/makedist right now)
* Merged in lp:~mterry/duplicity/drop-u1
- Ubuntu One is closing shop. So no need to support a u1 backend anymore.
* Merged in lp:~mterry/duplicity/fix-drop-u1
- Looks like when the drop-u1 branch got merged, its conflict got resolved
badly. Here is the right version of backend.py to use (and also drops
u1backend.py from POTFILES).
-* Merged in lp:~mterry/duplicity/drop-pexpect
- - Drop our local copy of pexpect in favor of a system version.
- - It's only used by the pexpect ssh backend (and if you're opting into that,
- you probably can expect that you will need pexpect) and the tests.
- - I've done a quick smoketest (backed up and restored using
- --ssh-backend=pexpect) and it seemed to work fine with a modern version
- of pexpect.
-* Merged in lp:~mterry/duplicity/2.6isms
- - Here's a whole stack of minor syntax modernizations that will become
- necessary in python3. They all work in python2.6.
- - I've added a new test to keep us honest and prevent backsliding on these
- modernizations. It runs 2to3 and will fail the test if 2to3 finds anything
- that needs fixing (with a specific set of exceptions carved out).
- - This branch has most of the easy 2to3 fixes, the ones with obvious and
- safe syntax changes.
- - We could just let 2to3 do them for us, but ideally we use 2to3 as little
- as possible, since it doesn't always know how to solve a given problem.
- I will propose a branch later that actually does use 2to3 to generate
- python3 versions of duplicity if they are requested. But this is a first
- step to clean up the code base.
-* Merged in lp:~mterry/duplicity/drop-static
- - Drop static.py.
- - This is some of the oldest code in duplicity! A bzr blame says it is
- unmodified (except for whitespace / comment changes) since revision 1.
- - But it's not needed anymore. Not really even since we updated to python2.4,
- which introduced the @staticmethod decorator. So this branch drops it and
- its test file.
* Merged in lp:~mterry/duplicity/encode-for-print
- Encode translated strings before passing them to 'print'.
- The print command can only apparently handle bytes. So when we pass it
@@ -238,45 +226,6 @@
in it is actually there.
* Fixed bug #1312328 WebDAV backend can't understand 200 OK response to DELETE
- Allow both 200 and 204 as valid response to delete
-* Merged in lp:~mterry/duplicity/py3-map-filter
- - In py3, map and filter return iterable objects, not lists. So in each case
- we use them, I've either imported the future version or switched to a list
- comprehension if we really wanted a list.
-* Merged in lp:~mterry/duplicity/backend-unification
- - Reorganize and simplify backend code. Specifically:
- - Formalize the expected API between backends and duplicity. See the new
- file duplicity/backends/README for the instructions I've given authors.
- - Add some tests for our backend wrapper class as well as some tests for
- individual backends. For several backends that have some commands do all
- the heavy lifting (hsi, tahoe, ftp), I've added fake little mock commands
- so that we can test them locally. This doesn't truly test our integration
- with those commands, but at least lets us test the backend glue code.
- - Removed a lot of duplicate and unused code which backends were using (or
- not using). This branch drops 700 lines of code (~20%)
- in duplicity/backends!
- - Simplified expectations of backends. Our wrapper code now does all the
- retrying, and all the exception handling. Backends can 'fire and forget'
- trusting our wrappers to give the user a reasonable error message.
- Obviously, backends can also add more details and make nicer error
- messages. But they don't *have* to.
- - Separate out the backend classes from our wrapper class. Now there is no
- possibility of namespace collision. All our API methods use one
- underscore. Anything else (zero or two underscores) are for the backend
- class's use.
- - Added the concept of a 'backend prefix' which is used by par2 and gio
- backends to provide generic support for "schema+" in urls -- like par2+
- or gio+. I've since marked the '--gio' flag as deprecated, in favor of
- 'gio+'. Now you can even nest such backends like
- par2+gio+file://blah/blah.
- - The switch to control which cloudfiles backend had a typo. I fixed this,
- but I'm not sure I should have? If we haven't had complaints, maybe we
- can just drop the old backend.
- - I manually tested all the backends we have (except hsi and tahoe -- but
- those are simple wrappers around commands and I did test those via mocks
- per above). I also added a bunch more manual backend tests to
- ./testing/manual/backendtest.py, which can now be run like the above to
- test all the files you have configured in config.py or you can pass it a
- URL which it will use for testing (useful for backend authors).
* Merged in lp:~mterry/duplicity/encode-exceptions
- Because exceptions often contain file paths, they have the same problem
with Python 2.x's implicit decoding using the 'ascii' encoding that we've
@@ -288,7 +237,6 @@
https://answers.launchpad.net/duplicity/+question/248020
-
New in v0.6.23 (2014/01/24)
---------------------------
Enhancements:
=== modified file 'COPYING'
--- COPYING 2014-06-28 14:24:05 +0000
+++ COPYING 2014-10-15 12:12:04 +0000
@@ -1,12 +1,12 @@
- GNU GENERAL PUBLIC LICENSE
- Version 2, June 1991
+ GNU GENERAL PUBLIC LICENSE
+ Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
- Preamble
+ Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
@@ -56,7 +56,7 @@
The precise terms and conditions for copying, distribution and
modification follow.
- GNU GENERAL PUBLIC LICENSE
+ GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
@@ -255,7 +255,7 @@
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
- NO WARRANTY
+ NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
@@ -277,9 +277,9 @@
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
- END OF TERMS AND CONDITIONS
+ END OF TERMS AND CONDITIONS
- How to Apply These Terms to Your New Programs
+ How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
=== modified file 'Changelog.GNU'
--- Changelog.GNU 2014-10-12 17:13:31 +0000
+++ Changelog.GNU 2014-10-15 12:12:04 +0000
@@ -12,70 +12,31 @@
2014-08-17 Kenneth Loafman <kenneth@xxxxxxxxxxx>
- * Added sxbacked.py, Skylable backend. Waiting on man page updates.
- * Merged in lp:~ed.so/duplicity/manpage.verify
- - Clarify verify's functionality as wished for by a user surprised with a big
- bandwidth bill from rackspace.
- * Merged in lp:~jeffreydavidrogers/duplicity/duplicity
- - This change fixes two small typos in the duplicity man page.
+ * Merged in lp:~jon-haggblad/duplicity/ftps-fix (2nd try)
+ - Minor bugfix for ftps backend where lftp needs host prefixed by "ftps://".
2014-06-28 Kenneth Loafman <kenneth@xxxxxxxxxxx>
- * Merged in lp:~3v1n0/duplicity/copy.com-backend
- - I've added a backend for Copy.com cloud storage, this supports all the
- required operations and works as it should from my tests.
- - You can use it by calling duplicity with something like:
- copy://account@xxxxxxxxx:your-password@xxxxxxxx/duplicity
- - The only thing I've concerns with is the optimized support for _delete_list
- which can't be enabled here because the test_delete_list tries also to
- delete a not-existing files, and it requires the backend not to raise an
- exception in that case (is this somewhat wanted or could we do the same as
- for _delete or _query?)
- * Merged in lp:~ed.so/duplicity/webdav200fix-0.7
+ * Merged in lp:~ed.so/duplicity/webdav200fix-0.6
- webdav backend fix "BackendException: Bad status code 200 reason OK. " when
restarting an interrupted backup and overwriting partially uploaded volumes.
- * Merged in lp:~mterry/duplicity/webdav-fixes
- - This branch fixes two issues I saw when testing the webdav backend:
- - 1) Errors like the following: "Attempt 1 failed. BackendException: File
- /tmp/duplicity-LQ1a0i-tempdir/mktemp-u2aiyX-2 not found locally after get
- from backend". These were caused by the _get() method not calling setdata()
- on the local path object, so the rest of the code thought it didn't exist.
- - 2) Some odd issues from stale responses/data. We have a couple places in
- webdavbackend.py where we close the connection before making a request
- because of this problem. But I've changed it to do it every time, more
- reliably, by putting a _close() call inside the request() method.
- - With this, the webdav backend seems fine to me.
- * Merged in lp:~antmak/duplicity/0.7-par2-fix
+ * Merged in lp:~antmak/duplicity/0.6-par2-fix
- Useful fix for verbatim par2cmdline options (like "-t" in par2-tbb version)
+ * Merged in lp:~jon-haggblad/duplicity/ftps-fix
+ - Minor bugfix for ftps backend where lftp needs host prefixed by "ftps://".
* Fixed bug 1327550: OverflowError: signed integer is greater than maximum
- Major and minor device numbers are supposed to be one byte each. Someone
has crafted a special system image using OpenVZ where the major and minor
device numbers are much larger (ploop devices). We treat them as (0,0).
-2014-05-11 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
- * Merged in lp:~mterry/duplicity/py2.6.0
- - Support python 2.6.0.
- - Without this branch, we only support python >= 2.6.5 because that's when
- python's urlparse.py module became its more modern incarnation. (I won't
- get into the wisdom of them making such a change in the middle of the
- 2.6 lifecycle.)
- - Also, the version of lockfile that I have (0.8) doesn't work with python
- 2.6.0 or 2.6.1 due to their implementation of
- threading.current_thread().ident returning None unexpectedly. So this
- branch tells lockfile not to worry about adding the current thread's
- identifier to the lock filename (we don't need a separate lock per thread,
- since our locking is per process).
- - I've tested with 2.6.0 and 2.7.6 (both extremes of our current support).
- * Update shebang line to python2 instead of python to avoid confusion.
+2014-05-09 Kenneth Loafman <kenneth@xxxxxxxxxxx>
+ * Prep for 0.6.24.
2014-05-07 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Applied expat fix from edso. See answer #12 in
https://answers.launchpad.net/duplicity/+question/248020
2014-04-30 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~mterry/duplicity/encode-exceptions
- Because exceptions often contain file paths, they have the same problem
with Python 2.x's implicit decoding using the 'ascii' encoding that we've
@@ -84,58 +45,11 @@
around the place.
- Bugs fixed: 1289288, 1311176, 1313966
-2014-04-29 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
- * Merged in lp:~mterry/duplicity/backend-unification
- - Reorganize and simplify backend code. Specifically:
- - Formalize the expected API between backends and duplicity. See the new
- file duplicity/backends/README for the instructions I've given authors.
- - Add some tests for our backend wrapper class as well as some tests for
- individual backends. For several backends that have some commands do all
- the heavy lifting (hsi, tahoe, ftp), I've added fake little mock commands
- so that we can test them locally. This doesn't truly test our integration
- with those commands, but at least lets us test the backend glue code.
- - Removed a lot of duplicate and unused code which backends were using (or
- not using). This branch drops 700 lines of code (~20%)
- in duplicity/backends!
- - Simplified expectations of backends. Our wrapper code now does all the
- retrying, and all the exception handling. Backends can 'fire and forget'
- trusting our wrappers to give the user a reasonable error message.
- Obviously, backends can also add more details and make nicer error
- messages. But they don't *have* to.
- - Separate out the backend classes from our wrapper class. Now there is no
- possibility of namespace collision. All our API methods use one
- underscore. Anything else (zero or two underscores) are for the backend
- class's use.
- - Added the concept of a 'backend prefix' which is used by par2 and gio
- backends to provide generic support for "schema+" in urls -- like par2+
- or gio+. I've since marked the '--gio' flag as deprecated, in favor of
- 'gio+'. Now you can even nest such backends like
- par2+gio+file://blah/blah.
- - The switch to control which cloudfiles backend had a typo. I fixed this,
- but I'm not sure I should have? If we haven't had complaints, maybe we
- can just drop the old backend.
- - I manually tested all the backends we have (except hsi and tahoe -- but
- those are simple wrappers around commands and I did test those via mocks
- per above). I also added a bunch more manual backend tests to
- ./testing/manual/backendtest.py, which can now be run like the above to
- test all the files you have configured in config.py or you can pass it a
- URL which it will use for testing (useful for backend authors).
-
-2014-04-26 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
- * Merged in lp:~mterry/duplicity/py3-map-filter
- - In py3, map and filter return iterable objects, not lists. So in each case
- we use them, I've either imported the future version or switched to a list
- comprehension if we really wanted a list.
-
2014-04-25 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Fixed bug #1312328 WebDAV backend can't understand 200 OK response to DELETE
- Allow both 200 and 204 as valid response to delete
2014-04-20 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~mterry/duplicity/more-test-reorg
- Here's another test reorganization / modernization branch. It does the
following things:
@@ -166,27 +80,6 @@
in it is actually there.
2014-04-19 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
- * Merged in lp:~mterry/duplicity/2.6isms
- - Here's a whole stack of minor syntax modernizations that will become
- necessary in python3. They all work in python2.6.
- - I've added a new test to keep us honest and prevent backsliding on these
- modernizations. It runs 2to3 and will fail the test if 2to3 finds anything
- that needs fixing (with a specific set of exceptions carved out).
- - This branch has most of the easy 2to3 fixes, the ones with obvious and
- safe syntax changes.
- - We could just let 2to3 do them for us, but ideally we use 2to3 as little
- as possible, since it doesn't always know how to solve a given problem.
- I will propose a branch later that actually does use 2to3 to generate
- python3 versions of duplicity if they are requested. But this is a first
- step to clean up the code base.
- * Merged in lp:~mterry/duplicity/drop-static
- - Drop static.py.
- - This is some of the oldest code in duplicity! A bzr blame says it is
- unmodified (except for whitespace / comment changes) since revision 1.
- - But it's not needed anymore. Not really even since we updated to python2.4,
- which introduced the @staticmethod decorator. So this branch drops it and
- its test file.
* Merged in lp:~mterry/duplicity/encode-for-print
- Encode translated strings before passing them to 'print'.
- The print command can only apparently handle bytes. So when we pass it
@@ -197,7 +90,6 @@
to the same encoding rules.
2014-04-17 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~fredrik-loch/duplicity/duplicity-S3-SSE
- Adds support for server side encryption as requested in Bug #996660
* Merged in lp:~mterry/duplicity/consolidate-tests
@@ -220,27 +112,14 @@
setuptools dropped support for <2.6 (and tox 1.7 recently dropped <2.6)
* Add setup.py hooks for test and sdist which are both standard targets
(sdist just outsources to dist/makedist right now)
- * Merged in lp:~mterry/duplicity/require-2.6
- - Require at least Python 2.6.
- - Our code base already requires 2.6, because 2.6-isms have crept in. Usually
- because we or a contributor didn't think to test with 2.4. And frankly,
- I'm not even sure how to test with 2.4 on a modern system.
* Merged in lp:~mterry/duplicity/drop-u1
- Ubuntu One is closing shop. So no need to support a u1 backend anymore.
* Merged in lp:~mterry/duplicity/fix-drop-u1
- Looks like when the drop-u1 branch got merged, its conflict got resolved
badly. Here is the right version of backend.py to use (and also drops
u1backend.py from POTFILES).
- * Merged in lp:~mterry/duplicity/drop-pexpect
- - Drop our local copy of pexpect in favor of a system version.
- - It's only used by the pexpect ssh backend (and if you're opting into that,
- you probably can expect that you will need pexpect) and the tests.
- - I've done a quick smoketest (backed up and restored using
- --ssh-backend=pexpect) and it seemed to work fine with a modern version
- of pexpect.
2014-03-10 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~germer/duplicity/par2
- This branch adds Par2 recovery files to duplicity. It is a wrapper backend
which will create the recovery files and upload them all together with the
@@ -250,7 +129,6 @@
- Fixes https://bugs.launchpad.net/duplicity/+bug/426282
2014-03-06 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~ed.so/duplicity/fix.dpbx
- Fix dpbx backend "NameError: global name 'rest' is not defined"
* Merged in lp:~prateek/duplicity/botoimportfix
@@ -258,7 +136,6 @@
complaints during the importing of backends.
2014-02-26 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~prateek/duplicity/s3-glacier
- Fixes https://bugs.launchpad.net/duplicity/+bug/1039511
- Adds support to detect when a file is on Glacier and initiates a restore
@@ -269,13 +146,11 @@
upload speed.
2014-02-24 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~mterry/duplicity/pexpect-fix
- duplicity has its own copy of pexpect. Use that instead of requiring one
from the system.
2014-02-05 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~mterry/duplicity/gpg-encode
- getpass.getpass(prompt) eventually calls str(prompt). Which is a no go,
if the prompt contains unicode. Here's a patch to always pass getpass() a
@@ -284,7 +159,6 @@
a test that passes the passphrase via stdin.
2014-01-31 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Applied two patches from mailing list message at:
https://lists.nongnu.org/archive/html/duplicity-talk/2014-01/msg00030.html
"Added command line options to use different prefixes for manifest/sig/archive files"
@@ -292,11 +166,9 @@
a workaround for https://bugs.launchpad.net/duplicity/+bug/1170113
2014-01-24 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Prep for 0.6.23 release.
2014-01-17 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~louis-bouchard/duplicity/add-allow-concurrency
- Implement locking mechanism to avoid concurrent execution under the same
cache directory. This is the default behavior.
@@ -305,18 +177,15 @@
- This functionality adds a dependency to python-lockfile
2014-01-13 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Restored patch of gdocsbackend.py from original author (thanks ede)
* Applied patch from bug 1266753: Boto backend removes local cache if
connection cannot be made
2014-01-02 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Restored missing line from patch of gdocsbackend.py
* Reverted changes to gdocsbackend.py
2013-12-30 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~mterry/duplicity/encoding
- This branch hopefully fixes two filename encoding issues:
- Users in bug 989496 were noticing a UnicodeEncodeError exception which
@@ -349,7 +218,6 @@
manually adjust those bits without being able to test each one.
2013-12-28 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~verb/duplicity/boto-min-version
- Update documentation and error messages to match the current actual version
requirements of boto backend.
@@ -363,7 +231,6 @@
* Nuke tabs
2013-11-24 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Merged in lp:~jkrauss/duplicity/pyrax
- Rackspace has deprecated python-cloudfiles in favor of their pyrax
library, which consolidates all Rackspace Cloud API functionality into
@@ -373,7 +240,6 @@
To revert to the cloudfiles backend use '--cf-backend=cloudfiles'
2013-11-19 Kenneth Loafman <kenneth@xxxxxxxxxxx>
-
* Applied patch to fix "Access GDrive through gdocs backend failing"
- see https://lists.nongnu.org/archive/html/duplicity-talk/2013-07/msg00007.html
=== modified file 'README'
--- README 2014-06-16 17:00:15 +0000
+++ README 2014-10-15 12:12:04 +0000
@@ -19,7 +19,7 @@
REQUIREMENTS:
- * Python v2.6 or later
+ * Python v2.4 or later
* librsync v0.9.6 or later
* GnuPG v1.x for encryption
* python-lockfile for concurrency locking
@@ -28,8 +28,8 @@
* for ftp over SSL -- lftp version 3.7.15 or later
* Boto 2.0 or later for single-processing S3 or GCS access (default)
* Boto 2.1.1 or later for multi-processing S3 access
+ * Python v2.6 or later for multi-processing S3 access
* Boto 2.7.0 or later for Glacier S3 access
- * python-urllib3 for Copy.com access
If you install from the source package, you will also need:
=== modified file 'bin/duplicity'
--- bin/duplicity 2014-05-11 11:50:12 +0000
+++ bin/duplicity 2014-10-15 12:12:04 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python
# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# duplicity -- Encrypted bandwidth efficient backup
@@ -289,6 +289,8 @@
def validate_block(orig_size, dest_filename):
info = backend.query_info([dest_filename])[dest_filename]
+ if 'size' not in info:
+ return # backend didn't know how to query size
size = info['size']
if size is None:
return # error querying file
@@ -1041,7 +1043,7 @@
log.Notice(_("Deleting local %s (not authoritative at backend).") % util.ufn(del_name))
try:
util.ignore_missing(os.unlink, del_name)
- except Exception as e:
+ except Exception, e:
log.Warn(_("Unable to delete %s: %s") % (util.ufn(del_name), util.uexc(e)))
def copy_to_local(fn):
@@ -1326,7 +1328,7 @@
# determine what action we're performing and process command line
action = commandline.ProcessCommandLine(sys.argv[1:])
- globals.lockfile = FileLock(os.path.join(globals.archive_dir.name, "lockfile"), threaded=False)
+ globals.lockfile = FileLock(os.path.join(globals.archive_dir.name, "lockfile"))
if globals.lockfile.is_locked():
log.FatalError(
"Another instance is already running with this archive directory\n"
@@ -1504,18 +1506,18 @@
# sys.exit() function. Python handles this by
# raising the SystemExit exception. Cleanup code
# goes here, if needed.
- except SystemExit as e:
+ except SystemExit, e:
# No traceback, just get out
util.release_lockfile()
sys.exit(e)
- except KeyboardInterrupt as e:
+ except KeyboardInterrupt, e:
# No traceback, just get out
log.Info(_("INT intercepted...exiting."))
util.release_lockfile()
sys.exit(4)
- except gpg.GPGError as e:
+ except gpg.GPGError, e:
# For gpg errors, don't show an ugly stack trace by
# default. But do with sufficient verbosity.
util.release_lockfile()
@@ -1525,7 +1527,7 @@
log.ErrorCode.gpg_failed,
e.__class__.__name__)
- except duplicity.errors.UserError as e:
+ except duplicity.errors.UserError, e:
util.release_lockfile()
# For user errors, don't show an ugly stack trace by
# default. But do with sufficient verbosity.
@@ -1535,7 +1537,7 @@
log.ErrorCode.user_error,
e.__class__.__name__)
- except duplicity.errors.BackendException as e:
+ except duplicity.errors.BackendException, e:
util.release_lockfile()
# For backend errors, don't show an ugly stack trace by
# default. But do with sufficient verbosity.
@@ -1545,7 +1547,7 @@
log.ErrorCode.user_error,
e.__class__.__name__)
- except Exception as e:
+ except Exception, e:
util.release_lockfile()
if "Forced assertion for testing" in str(e):
log.FatalError(u"%s: %s" % (e.__class__.__name__, util.uexc(e)),
=== modified file 'bin/duplicity.1'
--- bin/duplicity.1 2014-10-12 17:13:31 +0000
+++ bin/duplicity.1 2014-10-15 12:12:04 +0000
@@ -51,7 +51,7 @@
.SH REQUIREMENTS
Duplicity requires a POSIX-like operating system with a
.B python
-interpreter version 2.6+ installed.
+interpreter version 2.4+ installed.
It is best used under GNU/Linux.
Some backends also require additional components (probably available as packages for your specific platform):
@@ -72,10 +72,6 @@
.B Dropbox Python SDK
- https://www.dropbox.com/developers/reference/sdk
.TP
-.B "copy backend" (Copy.com)
-.B python-urllib3
-- https://github.com/shazow/urllib3
-.TP
.B "ftp backend"
.B NcFTP Client
- http://www.ncftp.com/
@@ -124,9 +120,6 @@
.B ssh pexpect backend
.B sftp/scp client binaries
OpenSSH - http://www.openssh.com/
-.br
-.B Python pexpect module
-- http://pexpect.sourceforge.net/pexpect.html
.TP
.BR "swift backend (OpenStack Object Storage)"
.B Python swiftclient module
@@ -258,8 +251,8 @@
Duplicity will abort if no old signatures can be found.
.TP
-.BI "verify " "[--compare-data] [--time <time>] [--file-to-restore <rel_path>] <url> <local_path>"
-Restore backup contents temporarily file by file and compare against the local path's contents.
+.BI "verify " "[--compare-data] [--time <time>] [--file-to-restore <relpath>] <url> <folder>"
+Verify compares the backup contents with the source folder.
duplicity will exit with a non-zero error level if any files are different.
On verbosity level info (4) or higher, a message for each file that has
changed will be logged.
@@ -518,7 +511,7 @@
.TP
.BI "--file-prefix, --file-prefix-manifest, --file-prefix-archive, --file-prefix-signature
-Adds a prefix to all files, manifest files, archive files, and/or signature files.
+Adds a prefix to all files, manifest files, archive files, and/or signature files.
The same set of prefixes must be passed in on backup and restore.
@@ -787,22 +780,22 @@
.TP
.BI "--s3-use-rrs"
-Store volumes using Reduced Redundancy Storage when uploading to Amazon S3.
-This will lower the cost of storage but also lower the durability of stored
-volumes to 99.99% instead the 99.999999999% durability offered by Standard
+Store volumes using Reduced Redundnacy Storage when uploading to Amazon S3.
+This will lower the cost of storage but also lower the durability of stored
+volumnes to 99.99% instead the 99.999999999% durability offered by Standard
Storage on S3.
.TP
.BI "--s3-use-multiprocessing"
Allow multipart volumne uploads to S3 through multiprocessing. This option
requires Python 2.6 and can be used to make uploads to S3 more efficient.
-If enabled, files duplicity uploads to S3 will be split into chunks and
+If enabled, files duplicity uploads to S3 will be split into chunks and
uploaded in parallel. Useful if you want to saturate your bandwidth
or if large files are failing during upload.
.TP
.BI "--s3-multipart-chunk-size"
-Chunk size (in MB) used for S3 multipart uploads. Make this smaller than
+Chunk size (in MB) used for S3 multipart uploads. Make this smaller than
.B --volsize
to maximize the use of your bandwidth. For example, a chunk size of 10MB
with a volsize of 30MB will result in 3 chunks per volume upload.
@@ -1069,9 +1062,6 @@
Make sure to read
.BR "A NOTE ON DROPBOX ACCESS" " first!"
.PP
-copy://user[:password]@copy.com/some_dir
-.PP
-.PP
file://[relative|/absolute]/local/path
.PP
ftp[s]://user[:password]@other.host[:port]/some_dir
@@ -1700,7 +1690,7 @@
.SH A NOTE ON FILENAME PREFIXES
-Filename prefixes can be used in conjunction with S3 lifecycle rules to transition
+Filename prefixes can be used in conjunction with S3 lifecycle rules to transition
archive files to Glacier, while keeping metadata (signature and manifest files) on S3.
Duplicity does not require access to archive files except when restoring from backup.
=== modified file 'bin/rdiffdir'
--- bin/rdiffdir 2014-05-11 11:50:12 +0000
+++ bin/rdiffdir 2014-10-15 12:12:04 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python
# rdiffdir -- Extend rdiff functionality to directories
# Version $version released $reldate
#
@@ -64,7 +64,7 @@
"include-filelist-stdin", "include-globbing-filelist",
"include-regexp=", "max-blocksize", "null-separator",
"verbosity=", "write-sig-to="])
- except getopt.error as e:
+ except getopt.error, e:
command_line_error("Bad command line option: %s" % (str(e),))
for opt, arg in optlist:
=== modified file 'dist/duplicity.spec.template'
--- dist/duplicity.spec.template 2014-04-16 20:45:09 +0000
+++ dist/duplicity.spec.template 2014-10-15 12:12:04 +0000
@@ -10,8 +10,8 @@
License: GPL
Group: Applications/Archiving
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
-requires: librsync >= 0.9.6, %{PYTHON_NAME} >= 2.6, gnupg >= 1.0.6
-BuildPrereq: %{PYTHON_NAME}-devel >= 2.6, librsync-devel >= 0.9.6
+requires: librsync >= 0.9.6, %{PYTHON_NAME} >= 2.4, gnupg >= 1.0.6
+BuildPrereq: %{PYTHON_NAME}-devel >= 2.4, librsync-devel >= 0.9.6
%description
Duplicity incrementally backs up files and directory by encrypting
=== modified file 'dist/makedist'
--- dist/makedist 2014-05-11 11:50:12 +0000
+++ dist/makedist 2014-10-15 12:12:04 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python
# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# Copyright 2002 Ben Escoto <ben@xxxxxxxxxxx>
=== modified file 'dist/makerpm'
--- dist/makerpm 2014-05-11 11:50:12 +0000
+++ dist/makerpm 2014-10-15 12:12:04 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python
# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# Copyright 2002 Ben Escoto <ben@xxxxxxxxxxx>
=== modified file 'duplicity/__init__.py'
--- duplicity/__init__.py 2014-04-16 20:45:09 +0000
+++ duplicity/__init__.py 2014-10-15 12:12:04 +0000
@@ -19,5 +19,12 @@
# along with duplicity; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+import __builtin__
import gettext
-gettext.install('duplicity', unicode=True, names=['ngettext'])
+
+t = gettext.translation('duplicity', fallback=True)
+t.install(unicode=True)
+
+# Once we can depend on python >=2.5, we can just use names='ngettext' above.
+# But for now, do the install manually.
+__builtin__.__dict__['ngettext'] = t.ungettext
=== modified file 'duplicity/_librsyncmodule.c'
--- duplicity/_librsyncmodule.c 2014-04-16 20:45:09 +0000
+++ duplicity/_librsyncmodule.c 2014-10-15 12:12:04 +0000
@@ -26,6 +26,15 @@
#include <librsync.h>
#define RS_JOB_BLOCKSIZE 65536
+/* Support Python 2.4 and 2.5 */
+#ifndef PyVarObject_HEAD_INIT
+ #define PyVarObject_HEAD_INIT(type, size) \
+ PyObject_HEAD_INIT(type) size,
+#endif
+#ifndef Py_TYPE
+ #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)
+#endif
+
static PyObject *librsyncError;
/* Sets python error string from result */
=== modified file 'duplicity/backend.py'
--- duplicity/backend.py 2014-06-28 14:43:36 +0000
+++ duplicity/backend.py 2014-10-15 12:12:04 +0000
@@ -24,7 +24,6 @@
intended to be used by the backends themselves.
"""
-import errno
import os
import sys
import socket
@@ -32,22 +31,20 @@
import re
import getpass
import gettext
-import types
import urllib
-import urlparse
+import urlparse_2_5 as urlparser
from duplicity import dup_temp
+from duplicity import dup_threading
from duplicity import file_naming
from duplicity import globals
from duplicity import log
-from duplicity import path
from duplicity import progress
from duplicity import util
from duplicity.util import exception_traceback
-from duplicity.errors import BackendException
-from duplicity.errors import FatalBackendException
+from duplicity.errors import BackendException, FatalBackendError
from duplicity.errors import TemporaryLoadException
from duplicity.errors import ConflictingScheme
from duplicity.errors import InvalidBackendURL
@@ -59,30 +56,8 @@
# todo: this should really NOT be done here
socket.setdefaulttimeout(globals.timeout)
+_forced_backend = None
_backends = {}
-_backend_prefixes = {}
-
-# These URL schemes have a backend with a notion of an RFC "network location".
-# The 'file' and 's3+http' schemes should not be in this list.
-# 'http' and 'https' are not actually used for duplicity backend urls, but are needed
-# in order to properly support urls returned from some webdav servers. adding them here
-# is a hack. we should instead not stomp on the url parsing module to begin with.
-#
-# This looks similar to urlparse's 'uses_netloc' list, but urlparse doesn't use
-# that list for parsing, only creating urls. And doesn't include our custom
-# schemes anyway. So we keep our own here for our own use.
-uses_netloc = ['ftp',
- 'ftps',
- 'hsi',
- 's3',
- 'scp', 'ssh', 'sftp',
- 'webdav', 'webdavs',
- 'gdocs',
- 'http', 'https',
- 'imap', 'imaps',
- 'mega',
- 'copy']
-
def import_backends():
"""
@@ -101,6 +76,8 @@
if fn.endswith("backend.py"):
fn = fn[:-3]
imp = "duplicity.backends.%s" % (fn,)
+ # ignore gio as it is explicitly loaded in commandline.parse_cmdline_options()
+ if fn == "giobackend": continue
try:
__import__(imp)
res = "Succeeded"
@@ -113,6 +90,14 @@
continue
+def force_backend(backend):
+ """
+ Forces the use of a particular backend, regardless of schema
+ """
+ global _forced_backend
+ _forced_backend = backend
+
+
def register_backend(scheme, backend_factory):
"""
Register a given backend factory responsible for URL:s with the
@@ -139,32 +124,6 @@
_backends[scheme] = backend_factory
-def register_backend_prefix(scheme, backend_factory):
- """
- Register a given backend factory responsible for URL:s with the
- given scheme prefix.
-
- The backend must be a callable which, when called with a URL as
- the single parameter, returns an object implementing the backend
- protocol (i.e., a subclass of Backend).
-
- Typically the callable will be the Backend subclass itself.
-
- This function is not thread-safe and is intended to be called
- during module importation or start-up.
- """
- global _backend_prefixes
-
- assert callable(backend_factory), "backend factory must be callable"
-
- if scheme in _backend_prefixes:
- raise ConflictingScheme("the prefix %s already has a backend "
- "associated with it"
- "" % (scheme,))
-
- _backend_prefixes[scheme] = backend_factory
-
-
def is_backend_url(url_string):
"""
@return Whether the given string looks like a backend URL.
@@ -178,9 +137,9 @@
return False
-def get_backend_object(url_string):
+def get_backend(url_string):
"""
- Find the right backend class instance for the given URL, or return None
+ Instantiate a backend suitable for the given URL, or return None
if the given string looks like a local path rather than a URL.
Raise InvalidBackendURL if the URL is not a valid URL.
@@ -188,45 +147,63 @@
if not is_backend_url(url_string):
return None
- global _backends, _backend_prefixes
-
pu = ParsedUrl(url_string)
+
+ # Implicit local path
assert pu.scheme, "should be a backend url according to is_backend_url"
- factory = None
-
- for prefix in _backend_prefixes:
- if url_string.startswith(prefix + '+'):
- factory = _backend_prefixes[prefix]
- pu = ParsedUrl(url_string.lstrip(prefix + '+'))
- break
-
- if factory is None:
- if not pu.scheme in _backends:
- raise UnsupportedBackendScheme(url_string)
- else:
- factory = _backends[pu.scheme]
-
- try:
- return factory(pu)
- except ImportError:
- raise BackendException(_("Could not initialize backend: %s") % str(sys.exc_info()[1]))
-
-
-def get_backend(url_string):
- """
- Instantiate a backend suitable for the given URL, or return None
- if the given string looks like a local path rather than a URL.
-
- Raise InvalidBackendURL if the URL is not a valid URL.
- """
- if globals.use_gio:
- url_string = 'gio+' + url_string
- obj = get_backend_object(url_string)
- if obj:
- obj = BackendWrapper(obj)
- return obj
-
+ global _backends, _forced_backend
+
+ if _forced_backend:
+ return _forced_backend(pu)
+ elif not pu.scheme in _backends:
+ raise UnsupportedBackendScheme(url_string)
+ else:
+ try:
+ return _backends[pu.scheme](pu)
+ except ImportError:
+ raise BackendException(_("Could not initialize backend: %s") % str(sys.exc_info()[1]))
+
+_urlparser_initialized = False
+_urlparser_initialized_lock = dup_threading.threading_module().Lock()
+
+def _ensure_urlparser_initialized():
+ """
+ Ensure that the appropriate clobbering of variables in the
+ urlparser module has been done. In the future, the need for this
+ clobbering to begin with should preferably be eliminated.
+ """
+ def init():
+ global _urlparser_initialized
+
+ if not _urlparser_initialized:
+ # These URL schemes have a backend with a notion of an RFC "network location".
+ # The 'file' and 's3+http' schemes should not be in this list.
+ # 'http' and 'https' are not actually used for duplicity backend urls, but are needed
+ # in order to properly support urls returned from some webdav servers. adding them here
+ # is a hack. we should instead not stomp on the url parsing module to begin with.
+ #
+ # todo: eliminate the need for backend specific hacking here completely.
+ urlparser.uses_netloc = ['ftp',
+ 'ftps',
+ 'hsi',
+ 'rsync',
+ 's3',
+ 'u1',
+ 'scp', 'ssh', 'sftp',
+ 'webdav', 'webdavs',
+ 'gdocs',
+ 'http', 'https',
+ 'imap', 'imaps',
+ 'mega']
+
+ # Do not transform or otherwise parse the URL path component.
+ urlparser.uses_query = []
+ urlparser.uses_fragm = []
+
+ _urlparser_initialized = True
+
+ dup_threading.with_lock(_urlparser_initialized_lock, init)
class ParsedUrl:
"""
@@ -240,19 +217,16 @@
Raise InvalidBackendURL on invalid URL's
"""
def __init__(self, url_string):
+ _ensure_urlparser_initialized()
self.url_string = url_string
- # Python < 2.6.5 still examine urlparse.uses_netlock when parsing urls,
- # so stuff our custom list in there before we parse.
- urlparse.uses_netloc = uses_netloc
-
# While useful in some cases, the fact is that the urlparser makes
# all the properties in the URL deferred or lazy. This means that
# problems don't get detected till called. We'll try to trap those
# problems here, so they will be caught early.
try:
- pu = urlparse.urlparse(url_string)
+ pu = urlparser.urlparse(url_string)
except Exception:
raise InvalidBackendURL("Syntax error in: %s" % url_string)
@@ -303,29 +277,18 @@
if not ( self.scheme in ['rsync'] and re.search('::[^:]*$', self.url_string)):
raise InvalidBackendURL("Syntax error (port) in: %s A%s B%s C%s" % (url_string, (self.scheme in ['rsync']), re.search('::[^:]+$', self.netloc), self.netloc ) )
- # Our URL system uses two slashes more than urlparse's does when using
- # non-netloc URLs. And we want to make sure that if urlparse assuming
- # a netloc where we don't want one, that we correct it.
- if self.scheme not in uses_netloc:
- if self.netloc:
- self.path = '//' + self.netloc + self.path
- self.netloc = ''
- self.hostname = None
- elif not self.path.startswith('//') and self.path.startswith('/'):
- self.path = '//' + self.path
-
# This happens for implicit local paths.
if not self.scheme:
return
# Our backends do not handle implicit hosts.
- if self.scheme in uses_netloc and not self.hostname:
+ if self.scheme in urlparser.uses_netloc and not self.hostname:
raise InvalidBackendURL("Missing hostname in a backend URL which "
"requires an explicit hostname: %s"
"" % (url_string))
# Our backends do not handle implicit relative paths.
- if self.scheme not in uses_netloc and not self.path.startswith('//'):
+ if self.scheme not in urlparser.uses_netloc and not self.path.startswith('//'):
raise InvalidBackendURL("missing // - relative paths not supported "
"for scheme %s: %s"
"" % (self.scheme, url_string))
@@ -343,74 +306,165 @@
# Replace the full network location with the stripped copy.
return parsed_url.geturl().replace(parsed_url.netloc, straight_netloc, 1)
-def _get_code_from_exception(backend, operation, e):
- if isinstance(e, BackendException) and e.code != log.ErrorCode.backend_error:
- return e.code
- elif hasattr(backend, '_error_code'):
- return backend._error_code(operation, e) or log.ErrorCode.backend_error
- elif hasattr(e, 'errno'):
- # A few backends return such errors (local, paramiko, etc)
- if e.errno == errno.EACCES:
- return log.ErrorCode.backend_permission_denied
- elif e.errno == errno.ENOENT:
- return log.ErrorCode.backend_not_found
- elif e.errno == errno.ENOSPC:
- return log.ErrorCode.backend_no_space
- return log.ErrorCode.backend_error
-
-def retry(operation, fatal=True):
- # Decorators with arguments introduce a new level of indirection. So we
- # have to return a decorator function (which itself returns a function!)
- def outer_retry(fn):
- def inner_retry(self, *args):
- for n in range(1, globals.num_retries + 1):
+
+# Decorator for backend operation functions to simplify writing one that
+# retries. Make sure to add a keyword argument 'raise_errors' to your function
+# and if it is true, raise an exception on an error. If false, fatal-log it.
+def retry(fn):
+ def iterate(*args):
+ for n in range(1, globals.num_retries):
+ try:
+ kwargs = {"raise_errors" : True}
+ return fn(*args, **kwargs)
+ except Exception, e:
+ log.Warn(_("Attempt %s failed: %s: %s")
+ % (n, e.__class__.__name__, util.uexc(e)))
+ log.Debug(_("Backtrace of previous error: %s")
+ % exception_traceback())
+ if isinstance(e, TemporaryLoadException):
+ time.sleep(30) # wait longer before trying again
+ else:
+ time.sleep(10) # wait a bit before trying again
+ # Now try one last time, but fatal-log instead of raising errors
+ kwargs = {"raise_errors" : False}
+ return fn(*args, **kwargs)
+ return iterate
+
+# same as above, a bit dumber and always dies fatally if last trial fails
+# hence no need for the raise_errors var ;), we really catch everything here
+# as we don't know what the underlying code comes up with and we really *do*
+# want to retry globals.num_retries times under all circumstances
+def retry_fatal(fn):
+ def _retry_fatal(self, *args):
+ try:
+ n = 0
+ for n in range(1, globals.num_retries):
try:
+ self.retry_count = n
return fn(self, *args)
- except FatalBackendException as e:
+ except FatalBackendError, e:
# die on fatal errors
raise e
- except Exception as e:
+ except Exception, e:
# retry on anything else
+ log.Warn(_("Attempt %s failed. %s: %s")
+ % (n, e.__class__.__name__, util.uexc(e)))
log.Debug(_("Backtrace of previous error: %s")
% exception_traceback())
- at_end = n == globals.num_retries
- code = _get_code_from_exception(self.backend, operation, e)
- if code == log.ErrorCode.backend_not_found:
- # If we tried to do something, but the file just isn't there,
- # no need to retry.
- at_end = True
- if at_end and fatal:
- def make_filename(f):
- if isinstance(f, path.ROPath):
- return util.escape(f.name)
- else:
- return util.escape(f)
- extra = ' '.join([operation] + [make_filename(x) for x in args if x])
- log.FatalError(_("Giving up after %s attempts. %s: %s")
- % (n, e.__class__.__name__,
- util.uexc(e)), code=code, extra=extra)
- else:
- log.Warn(_("Attempt %s failed. %s: %s")
- % (n, e.__class__.__name__, util.uexc(e)))
- if not at_end:
- if isinstance(e, TemporaryLoadException):
- time.sleep(90) # wait longer before trying again
- else:
- time.sleep(30) # wait a bit before trying again
- if hasattr(self.backend, '_retry_cleanup'):
- self.backend._retry_cleanup()
-
- return inner_retry
- return outer_retry
-
+ time.sleep(10) # wait a bit before trying again
+ # final trial, die on exception
+ self.retry_count = n+1
+ return fn(self, *args)
+ except Exception, e:
+ log.Debug(_("Backtrace of previous error: %s")
+ % exception_traceback())
+ log.FatalError(_("Giving up after %s attempts. %s: %s")
+ % (self.retry_count, e.__class__.__name__, util.uexc(e)),
+ log.ErrorCode.backend_error)
+ self.retry_count = 0
+
+ return _retry_fatal
class Backend(object):
"""
- See README in backends directory for information on how to write a backend.
+ Represents a generic duplicity backend, capable of storing and
+ retrieving files.
+
+ Concrete sub-classes are expected to implement:
+
+ - put
+ - get
+ - list
+ - delete
+ - close (if needed)
+
+ Optional:
+
+ - move
"""
+
def __init__(self, parsed_url):
self.parsed_url = parsed_url
+ def put(self, source_path, remote_filename = None):
+ """
+ Transfer source_path (Path object) to remote_filename (string)
+
+ If remote_filename is None, get the filename from the last
+ path component of pathname.
+ """
+ raise NotImplementedError()
+
+ def move(self, source_path, remote_filename = None):
+ """
+ Move source_path (Path object) to remote_filename (string)
+
+ Same as put(), but unlinks source_path in the process. This allows the
+ local backend to do this more efficiently using rename.
+ """
+ self.put(source_path, remote_filename)
+ source_path.delete()
+
+ def get(self, remote_filename, local_path):
+ """Retrieve remote_filename and place in local_path"""
+ raise NotImplementedError()
+
+ def list(self):
+ """
+ Return list of filenames (byte strings) present in backend
+ """
+ def tobytes(filename):
+ "Convert a (maybe unicode) filename to bytes"
+ if isinstance(filename, unicode):
+ # There shouldn't be any encoding errors for files we care
+ # about, since duplicity filenames are ascii. But user files
+ # may be in the same directory. So just replace characters.
+ return filename.encode(sys.getfilesystemencoding(), 'replace')
+ else:
+ return filename
+
+ if hasattr(self, '_list'):
+ # Make sure that duplicity internals only ever see byte strings
+ # for filenames, no matter what the backend thinks it is talking.
+ return map(tobytes, self._list())
+ else:
+ raise NotImplementedError()
+
+ def delete(self, filename_list):
+ """
+ Delete each filename in filename_list, in order if possible.
+ """
+ raise NotImplementedError()
+
+ # Should never cause FatalError.
+ # Returns a dictionary of dictionaries. The outer dictionary maps
+ # filenames to metadata dictionaries. Supported metadata are:
+ #
+ # 'size': if >= 0, size of file
+ # if -1, file is not found
+ # if None, error querying file
+ #
+ # Returned dictionary is guaranteed to contain a metadata dictionary for
+ # each filename, but not all metadata are guaranteed to be present.
+ def query_info(self, filename_list, raise_errors=True):
+ """
+ Return metadata about each filename in filename_list
+ """
+ info = {}
+ if hasattr(self, '_query_list_info'):
+ info = self._query_list_info(filename_list)
+ elif hasattr(self, '_query_file_info'):
+ for filename in filename_list:
+ info[filename] = self._query_file_info(filename)
+
+ # Fill out any missing entries (may happen if backend has no support
+ # or its query_list support is lazy)
+ for filename in filename_list:
+ if filename not in info:
+ info[filename] = {}
+
+ return info
+
""" use getpass by default, inherited backends may overwrite this behaviour """
use_getpass = True
@@ -449,7 +503,27 @@
else:
return commandline
- def __subprocess_popen(self, commandline):
+ """
+ DEPRECATED:
+ run_command(_persist) - legacy wrappers for subprocess_popen(_persist)
+ """
+ def run_command(self, commandline):
+ return self.subprocess_popen(commandline)
+ def run_command_persist(self, commandline):
+ return self.subprocess_popen_persist(commandline)
+
+ """
+ DEPRECATED:
+ popen(_persist) - legacy wrappers for subprocess_popen(_persist)
+ """
+ def popen(self, commandline):
+ result, stdout, stderr = self.subprocess_popen(commandline)
+ return stdout
+ def popen_persist(self, commandline):
+ result, stdout, stderr = self.subprocess_popen_persist(commandline)
+ return stdout
+
+ def _subprocess_popen(self, commandline):
"""
For internal use.
Execute the given command line, interpreted as a shell command.
@@ -461,10 +535,6 @@
return p.returncode, stdout, stderr
- """ a dictionary for breaking exceptions, syntax is
- { 'command' : [ code1, code2 ], ... } see ftpbackend for an example """
- popen_breaks = {}
-
def subprocess_popen(self, commandline):
"""
Execute the given command line with error check.
@@ -474,179 +544,54 @@
"""
private = self.munge_password(commandline)
log.Info(_("Reading results of '%s'") % private)
- result, stdout, stderr = self.__subprocess_popen(commandline)
+ result, stdout, stderr = self._subprocess_popen(commandline)
if result != 0:
+ raise BackendException("Error running '%s'" % private)
+ return result, stdout, stderr
+
+ """ a dictionary for persist breaking exceptions, syntax is
+ { 'command' : [ code1, code2 ], ... } see ftpbackend for an example """
+ popen_persist_breaks = {}
+
+ def subprocess_popen_persist(self, commandline):
+ """
+ Execute the given command line with error check.
+ Retries globals.num_retries times with 30s delay.
+ Returns int Exitcode, string StdOut, string StdErr
+
+ Raise a BackendException on failure.
+ """
+ private = self.munge_password(commandline)
+
+ for n in range(1, globals.num_retries+1):
+ # sleep before retry
+ if n > 1:
+ time.sleep(30)
+ log.Info(_("Reading results of '%s'") % private)
+ result, stdout, stderr = self._subprocess_popen(commandline)
+ if result == 0:
+ return result, stdout, stderr
+
try:
m = re.search("^\s*([\S]+)", commandline)
cmd = m.group(1)
- ignores = self.popen_breaks[ cmd ]
+ ignores = self.popen_persist_breaks[ cmd ]
ignores.index(result)
""" ignore a predefined set of error codes """
return 0, '', ''
except (KeyError, ValueError):
- raise BackendException("Error running '%s': returned %d, with output:\n%s" %
- (private, result, stdout + '\n' + stderr))
- return result, stdout, stderr
-
-
-class BackendWrapper(object):
- """
- Represents a generic duplicity backend, capable of storing and
- retrieving files.
- """
-
- def __init__(self, backend):
- self.backend = backend
-
- def __do_put(self, source_path, remote_filename):
- if hasattr(self.backend, '_put'):
- log.Info(_("Writing %s") % util.ufn(remote_filename))
- self.backend._put(source_path, remote_filename)
- else:
- raise NotImplementedError()
-
- @retry('put', fatal=True)
- def put(self, source_path, remote_filename=None):
- """
- Transfer source_path (Path object) to remote_filename (string)
-
- If remote_filename is None, get the filename from the last
- path component of pathname.
- """
- if not remote_filename:
- remote_filename = source_path.get_filename()
- self.__do_put(source_path, remote_filename)
-
- @retry('move', fatal=True)
- def move(self, source_path, remote_filename=None):
- """
- Move source_path (Path object) to remote_filename (string)
-
- Same as put(), but unlinks source_path in the process. This allows the
- local backend to do this more efficiently using rename.
- """
- if not remote_filename:
- remote_filename = source_path.get_filename()
- if hasattr(self.backend, '_move'):
- if self.backend._move(source_path, remote_filename) is not False:
- source_path.setdata()
- return
- self.__do_put(source_path, remote_filename)
- source_path.delete()
-
- @retry('get', fatal=True)
- def get(self, remote_filename, local_path):
- """Retrieve remote_filename and place in local_path"""
- if hasattr(self.backend, '_get'):
- self.backend._get(remote_filename, local_path)
- local_path.setdata()
- if not local_path.exists():
- raise BackendException(_("File %s not found locally after get "
- "from backend") % util.ufn(local_path.name))
- else:
- raise NotImplementedError()
-
- @retry('list', fatal=True)
- def list(self):
- """
- Return list of filenames (byte strings) present in backend
- """
- def tobytes(filename):
- "Convert a (maybe unicode) filename to bytes"
- if isinstance(filename, unicode):
- # There shouldn't be any encoding errors for files we care
- # about, since duplicity filenames are ascii. But user files
- # may be in the same directory. So just replace characters.
- return filename.encode(sys.getfilesystemencoding(), 'replace')
- else:
- return filename
-
- if hasattr(self.backend, '_list'):
- # Make sure that duplicity internals only ever see byte strings
- # for filenames, no matter what the backend thinks it is talking.
- return [tobytes(x) for x in self.backend._list()]
- else:
- raise NotImplementedError()
-
- def delete(self, filename_list):
- """
- Delete each filename in filename_list, in order if possible.
- """
- assert type(filename_list) is not types.StringType
- if hasattr(self.backend, '_delete_list'):
- self._do_delete_list(filename_list)
- elif hasattr(self.backend, '_delete'):
- for filename in filename_list:
- self._do_delete(filename)
- else:
- raise NotImplementedError()
-
- @retry('delete', fatal=False)
- def _do_delete_list(self, filename_list):
- self.backend._delete_list(filename_list)
-
- @retry('delete', fatal=False)
- def _do_delete(self, filename):
- self.backend._delete(filename)
-
- # Should never cause FatalError.
- # Returns a dictionary of dictionaries. The outer dictionary maps
- # filenames to metadata dictionaries. Supported metadata are:
- #
- # 'size': if >= 0, size of file
- # if -1, file is not found
- # if None, error querying file
- #
- # Returned dictionary is guaranteed to contain a metadata dictionary for
- # each filename, and all metadata are guaranteed to be present.
- def query_info(self, filename_list):
- """
- Return metadata about each filename in filename_list
- """
- info = {}
- if hasattr(self.backend, '_query_list'):
- info = self._do_query_list(filename_list)
- if info is None:
- info = {}
- elif hasattr(self.backend, '_query'):
- for filename in filename_list:
- info[filename] = self._do_query(filename)
-
- # Fill out any missing entries (may happen if backend has no support
- # or its query_list support is lazy)
- for filename in filename_list:
- if filename not in info or info[filename] is None:
- info[filename] = {}
- for metadata in ['size']:
- info[filename].setdefault(metadata, None)
-
- return info
-
- @retry('query', fatal=False)
- def _do_query_list(self, filename_list):
- info = self.backend._query_list(filename_list)
- if info is None:
- info = {}
- return info
-
- @retry('query', fatal=False)
- def _do_query(self, filename):
- try:
- return self.backend._query(filename)
- except Exception as e:
- code = _get_code_from_exception(self.backend, 'query', e)
- if code == log.ErrorCode.backend_not_found:
- return {'size': -1}
- else:
- raise e
-
- def close(self):
- """
- Close the backend, releasing any resources held and
- invalidating any file objects obtained from the backend.
- """
- if hasattr(self.backend, '_close'):
- self.backend._close()
+ pass
+
+ log.Warn(ngettext("Running '%s' failed with code %d (attempt #%d)",
+ "Running '%s' failed with code %d (attempt #%d)", n) %
+ (private, result, n))
+ if stdout or stderr:
+ log.Warn(_("Error is:\n%s") % stderr + (stderr and stdout and "\n") + stdout)
+
+ log.Warn(ngettext("Giving up trying to execute '%s' after %d attempt",
+ "Giving up trying to execute '%s' after %d attempts",
+ globals.num_retries) % (private, globals.num_retries))
+ raise BackendException("Error running '%s'" % private)
def get_fileobj_read(self, filename, parseresults = None):
"""
@@ -663,6 +608,37 @@
tdp.setdata()
return tdp.filtered_open_with_delete("rb")
+ def get_fileobj_write(self, filename,
+ parseresults = None,
+ sizelist = None):
+ """
+ Return fileobj opened for writing, which will cause the file
+ to be written to the backend on close().
+
+ The file will be encoded as specified in parseresults (or as
+ read from the filename), and stored in a temp file until it
+ can be copied over and deleted.
+
+ If sizelist is not None, it should be set to an empty list.
+ The number of bytes will be inserted into the list.
+ """
+ if not parseresults:
+ parseresults = file_naming.parse(filename)
+ assert parseresults, u"Filename %s not correctly parsed" % util.ufn(filename)
+ tdp = dup_temp.new_tempduppath(parseresults)
+
+ def close_file_hook():
+ """This is called when returned fileobj is closed"""
+ self.put(tdp, filename)
+ if sizelist is not None:
+ tdp.setdata()
+ sizelist.append(tdp.getsize())
+ tdp.delete()
+
+ fh = dup_temp.FileobjHooked(tdp.filtered_open("wb"))
+ fh.addhook(close_file_hook)
+ return fh
+
def get_data(self, filename, parseresults = None):
"""
Retrieve a file from backend, process it, return contents.
@@ -671,3 +647,18 @@
buf = fin.read()
assert not fin.close()
return buf
+
+ def put_data(self, buffer, filename, parseresults = None):
+ """
+ Put buffer into filename on backend after processing.
+ """
+ fout = self.get_fileobj_write(filename, parseresults)
+ fout.write(buffer)
+ assert not fout.close()
+
+ def close(self):
+ """
+ Close the backend, releasing any resources held and
+ invalidating any file objects obtained from the backend.
+ """
+ pass
=== removed file 'duplicity/backends/README'
--- duplicity/backends/README 2014-05-06 18:10:52 +0000
+++ duplicity/backends/README 1970-01-01 00:00:00 +0000
@@ -1,79 +0,0 @@
-= How to write a backend, in five easy steps! =
-
-There are five main methods you want to implement:
-
-__init__ - Initial setup
-_get
- - Get one file
- - Retried if an exception is thrown
-_put
- - Upload one file
- - Retried if an exception is thrown
-_list
- - List all files in the backend
- - Return a list of filenames
- - Retried if an exception is thrown
-_delete
- - Delete one file
- - Retried if an exception is thrown
-
-There are other methods you may optionally implement:
-
-_delete_list
- - Delete list of files
- - This is used in preference of _delete if defined
- - Must gracefully handle individual file errors itself
- - Retried if an exception is thrown
-_query
- - Query metadata of one file
- - Return a dict with a 'size' key, and a file size value (-1 for not found)
- - Retried if an exception is thrown
-_query_list
- - Query metadata of a list of files
- - Return a dict of filenames mapping to a dict with a 'size' key,
- and a file size value (-1 for not found)
- - This is used in preference of _query if defined
- - Must gracefully handle individual file errors itself
- - Retried if an exception is thrown
-_retry_cleanup
- - If the backend wants to do any bookkeeping or connection resetting inbetween
- retries, do it here.
-_error_code
- - Passed an exception thrown by your backend, return a log.ErrorCode that
- corresponds to that exception
-_move
- - If your backend can more optimally move a local file into its backend,
- implement this. If it's not implemented or returns False, _put will be
- called instead (and duplicity will delete the source file after).
- - Retried if an exception is thrown
-_close
- - If your backend needs to clean up after itself, do that here.
-
-== Subclassing ==
-
-Always subclass from duplicity.backend.Backend
-
-== Registering ==
-
-You can register your class as a single backend like so:
-
-duplicity.backend.register_backend("foo", FooBackend)
-
-This will allow a URL like so: foo://hostname/path
-
-Or you can register your class as a meta backend like so:
-duplicity.backend.register_backend_prefix("bar", BarBackend)
-
-Which will allow a URL like so: bar+foo://hostname/path and your class will
-be passed the inner URL to either interpret how you like or create a new
-inner backend instance with duplicity.backend.get_backend_object(url).
-
-== Naming ==
-
-Any method that duplicity calls will start with one underscore. Please use
-zero or two underscores in your method names to avoid conflicts.
-
-== Testing ==
-
-Use "./testing/manual/backendtest foo://hostname/path" to test your new
-backend. It will load your backend from your current branch.
=== modified file 'duplicity/backends/_boto_multi.py'
--- duplicity/backends/_boto_multi.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/_boto_multi.py 2014-10-15 12:12:04 +0000
@@ -33,8 +33,8 @@
from duplicity.filechunkio import FileChunkIO
from duplicity import progress
-from ._boto_single import BotoBackend as BotoSingleBackend
-from ._boto_single import get_connection
+from _boto_single import BotoBackend as BotoSingleBackend
+from _boto_single import get_connection
BOTO_MIN_VERSION = "2.1.1"
@@ -63,7 +63,7 @@
try:
args = self.queue.get(True, 1)
progress.report_transfer(args[0], args[1])
- except Queue.Empty as e:
+ except Queue.Empty, e:
pass
@@ -98,8 +98,8 @@
self._pool = multiprocessing.Pool(processes=number_of_procs)
- def _close(self):
- BotoSingleBackend._close(self)
+ def close(self):
+ BotoSingleBackend.close(self)
log.Debug("Closing pool")
self._pool.terminate()
self._pool.join()
@@ -210,7 +210,7 @@
conn = None
bucket = None
del conn
- except Exception as e:
+ except Exception, e:
traceback.print_exc()
if num_retries:
log.Debug("%s: Upload of chunk %d failed. Retrying %d more times..." % (
=== modified file 'duplicity/backends/_boto_single.py'
--- duplicity/backends/_boto_single.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/_boto_single.py 2014-10-15 12:12:04 +0000
@@ -25,7 +25,9 @@
import duplicity.backend
from duplicity import globals
from duplicity import log
-from duplicity.errors import FatalBackendException, BackendException
+from duplicity.errors import * #@UnusedWildImport
+from duplicity.util import exception_traceback
+from duplicity.backend import retry
from duplicity import progress
BOTO_MIN_VERSION = "2.1.1"
@@ -135,7 +137,7 @@
# This folds the null prefix and all null parts, which means that:
# //MyBucket/ and //MyBucket are equivalent.
# //MyBucket//My///My/Prefix/ and //MyBucket/My/Prefix are equivalent.
- self.url_parts = [x for x in parsed_url.path.split('/') if x != '']
+ self.url_parts = filter(lambda x: x != '', parsed_url.path.split('/'))
if self.url_parts:
self.bucket_name = self.url_parts.pop(0)
@@ -161,7 +163,7 @@
self.resetConnection()
self._listed_keys = {}
- def _close(self):
+ def close(self):
del self._listed_keys
self._listed_keys = {}
self.bucket = None
@@ -183,69 +185,137 @@
self.conn = get_connection(self.scheme, self.parsed_url, self.storage_uri)
self.bucket = self.conn.lookup(self.bucket_name)
- def _retry_cleanup(self):
- self.resetConnection()
-
- def _put(self, source_path, remote_filename):
+ def put(self, source_path, remote_filename=None):
from boto.s3.connection import Location
if globals.s3_european_buckets:
if not globals.s3_use_new_style:
- raise FatalBackendException("European bucket creation was requested, but not new-style "
- "bucket addressing (--s3-use-new-style)",
- code=log.ErrorCode.s3_bucket_not_style)
-
- if self.bucket is None:
+ log.FatalError("European bucket creation was requested, but not new-style "
+ "bucket addressing (--s3-use-new-style)",
+ log.ErrorCode.s3_bucket_not_style)
+ #Network glitch may prevent first few attempts of creating/looking up a bucket
+ for n in range(1, globals.num_retries+1):
+ if self.bucket:
+ break
+ if n > 1:
+ time.sleep(30)
+ self.resetConnection()
try:
- self.bucket = self.conn.get_bucket(self.bucket_name, validate=True)
- except Exception as e:
- if "NoSuchBucket" in str(e):
- if globals.s3_european_buckets:
- self.bucket = self.conn.create_bucket(self.bucket_name,
- location=Location.EU)
+ try:
+ self.bucket = self.conn.get_bucket(self.bucket_name, validate=True)
+ except Exception, e:
+ if "NoSuchBucket" in str(e):
+ if globals.s3_european_buckets:
+ self.bucket = self.conn.create_bucket(self.bucket_name,
+ location=Location.EU)
+ else:
+ self.bucket = self.conn.create_bucket(self.bucket_name)
else:
- self.bucket = self.conn.create_bucket(self.bucket_name)
+ raise e
+ except Exception, e:
+ log.Warn("Failed to create bucket (attempt #%d) '%s' failed (reason: %s: %s)"
+ "" % (n, self.bucket_name,
+ e.__class__.__name__,
+ str(e)))
+
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+ key = self.bucket.new_key(self.key_prefix + remote_filename)
+
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry (new connection to a **hopeful** new host, so no need to wait so long)
+ time.sleep(10)
+
+ if globals.s3_use_rrs:
+ storage_class = 'REDUCED_REDUNDANCY'
+ else:
+ storage_class = 'STANDARD'
+ log.Info("Uploading %s/%s to %s Storage" % (self.straight_url, remote_filename, storage_class))
+ try:
+ if globals.s3_use_sse:
+ headers = {
+ 'Content-Type': 'application/octet-stream',
+ 'x-amz-storage-class': storage_class,
+ 'x-amz-server-side-encryption': 'AES256'
+ }
else:
- raise
-
- key = self.bucket.new_key(self.key_prefix + remote_filename)
-
- if globals.s3_use_rrs:
- storage_class = 'REDUCED_REDUNDANCY'
- else:
- storage_class = 'STANDARD'
- log.Info("Uploading %s/%s to %s Storage" % (self.straight_url, remote_filename, storage_class))
- if globals.s3_use_sse:
- headers = {
- 'Content-Type': 'application/octet-stream',
- 'x-amz-storage-class': storage_class,
- 'x-amz-server-side-encryption': 'AES256'
- }
- else:
- headers = {
- 'Content-Type': 'application/octet-stream',
- 'x-amz-storage-class': storage_class
- }
-
- upload_start = time.time()
- self.upload(source_path.name, key, headers)
- upload_end = time.time()
- total_s = abs(upload_end-upload_start) or 1 # prevent a zero value!
- rough_upload_speed = os.path.getsize(source_path.name)/total_s
- log.Debug("Uploaded %s/%s to %s Storage at roughly %f bytes/second" % (self.straight_url, remote_filename, storage_class, rough_upload_speed))
-
- def _get(self, remote_filename, local_path):
+ headers = {
+ 'Content-Type': 'application/octet-stream',
+ 'x-amz-storage-class': storage_class
+ }
+
+ upload_start = time.time()
+ self.upload(source_path.name, key, headers)
+ upload_end = time.time()
+ total_s = abs(upload_end-upload_start) or 1 # prevent a zero value!
+ rough_upload_speed = os.path.getsize(source_path.name)/total_s
+ self.resetConnection()
+ log.Debug("Uploaded %s/%s to %s Storage at roughly %f bytes/second" % (self.straight_url, remote_filename, storage_class, rough_upload_speed))
+ return
+ except Exception, e:
+ log.Warn("Upload '%s/%s' failed (attempt #%d, reason: %s: %s)"
+ "" % (self.straight_url,
+ remote_filename,
+ n,
+ e.__class__.__name__,
+ str(e)))
+ log.Debug("Backtrace of previous error: %s" % (exception_traceback(),))
+ self.resetConnection()
+ log.Warn("Giving up trying to upload %s/%s after %d attempts" %
+ (self.straight_url, remote_filename, globals.num_retries))
+ raise BackendException("Error uploading %s/%s" % (self.straight_url, remote_filename))
+
+ def get(self, remote_filename, local_path):
key_name = self.key_prefix + remote_filename
self.pre_process_download(remote_filename, wait=True)
key = self._listed_keys[key_name]
- self.resetConnection()
- key.get_contents_to_filename(local_path.name)
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry (new connection to a **hopeful** new host, so no need to wait so long)
+ time.sleep(10)
+ log.Info("Downloading %s/%s" % (self.straight_url, remote_filename))
+ try:
+ self.resetConnection()
+ key.get_contents_to_filename(local_path.name)
+ local_path.setdata()
+ return
+ except Exception, e:
+ log.Warn("Download %s/%s failed (attempt #%d, reason: %s: %s)"
+ "" % (self.straight_url,
+ remote_filename,
+ n,
+ e.__class__.__name__,
+ str(e)), 1)
+ log.Debug("Backtrace of previous error: %s" % (exception_traceback(),))
+
+ log.Warn("Giving up trying to download %s/%s after %d attempts" %
+ (self.straight_url, remote_filename, globals.num_retries))
+ raise BackendException("Error downloading %s/%s" % (self.straight_url, remote_filename))
def _list(self):
if not self.bucket:
raise BackendException("No connection to backend")
- return self.list_filenames_in_bucket()
-
- def list_filenames_in_bucket(self):
+
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry
+ time.sleep(30)
+ self.resetConnection()
+ log.Info("Listing %s" % self.straight_url)
+ try:
+ return self._list_filenames_in_bucket()
+ except Exception, e:
+ log.Warn("List %s failed (attempt #%d, reason: %s: %s)"
+ "" % (self.straight_url,
+ n,
+ e.__class__.__name__,
+ str(e)), 1)
+ log.Debug("Backtrace of previous error: %s" % (exception_traceback(),))
+ log.Warn("Giving up trying to list %s after %d attempts" %
+ (self.straight_url, globals.num_retries))
+ raise BackendException("Error listng %s" % self.straight_url)
+
+ def _list_filenames_in_bucket(self):
# We add a 'd' to the prefix to make sure it is not null (for boto) and
# to optimize the listing of our filenames, which always begin with 'd'.
# This will cause a failure in the regression tests as below:
@@ -266,37 +336,76 @@
pass
return filename_list
- def _delete(self, filename):
- self.bucket.delete_key(self.key_prefix + filename)
+ def delete(self, filename_list):
+ for filename in filename_list:
+ self.bucket.delete_key(self.key_prefix + filename)
+ log.Debug("Deleted %s/%s" % (self.straight_url, filename))
- def _query(self, filename):
- key = self.bucket.lookup(self.key_prefix + filename)
- if key is None:
- return {'size': -1}
- return {'size': key.size}
+ @retry
+ def _query_file_info(self, filename, raise_errors=False):
+ try:
+ key = self.bucket.lookup(self.key_prefix + filename)
+ if key is None:
+ return {'size': -1}
+ return {'size': key.size}
+ except Exception, e:
+ log.Warn("Query %s/%s failed: %s"
+ "" % (self.straight_url,
+ filename,
+ str(e)))
+ self.resetConnection()
+ if raise_errors:
+ raise e
+ else:
+ return {'size': None}
def upload(self, filename, key, headers):
- key.set_contents_from_filename(filename, headers,
- cb=progress.report_transfer,
- num_cb=(max(2, 8 * globals.volsize / (1024 * 1024)))
- ) # Max num of callbacks = 8 times x megabyte
- key.close()
+ key.set_contents_from_filename(filename, headers,
+ cb=progress.report_transfer,
+ num_cb=(max(2, 8 * globals.volsize / (1024 * 1024)))
+ ) # Max num of callbacks = 8 times x megabyte
+ key.close()
- def pre_process_download(self, remote_filename, wait=False):
+ def pre_process_download(self, files_to_download, wait=False):
# Used primarily to move files in Glacier to S3
- key_name = self.key_prefix + remote_filename
- if not self._listed_keys.get(key_name, False):
- self._listed_keys[key_name] = list(self.bucket.list(key_name))[0]
- key = self._listed_keys[key_name]
+ if isinstance(files_to_download, basestring):
+ files_to_download = [files_to_download]
- if key.storage_class == "GLACIER":
- # We need to move the file out of glacier
- if not self.bucket.get_key(key.key).ongoing_restore:
- log.Info("File %s is in Glacier storage, restoring to S3" % remote_filename)
- key.restore(days=1) # Shouldn't need this again after 1 day
- if wait:
- log.Info("Waiting for file %s to restore from Glacier" % remote_filename)
- while self.bucket.get_key(key.key).ongoing_restore:
- time.sleep(60)
+ for remote_filename in files_to_download:
+ success = False
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry (new connection to a **hopeful** new host, so no need to wait so long)
+ time.sleep(10)
self.resetConnection()
- log.Info("File %s was successfully restored from Glacier" % remote_filename)
+ try:
+ key_name = self.key_prefix + remote_filename
+ if not self._listed_keys.get(key_name, False):
+ self._listed_keys[key_name] = list(self.bucket.list(key_name))[0]
+ key = self._listed_keys[key_name]
+
+ if key.storage_class == "GLACIER":
+ # We need to move the file out of glacier
+ if not self.bucket.get_key(key.key).ongoing_restore:
+ log.Info("File %s is in Glacier storage, restoring to S3" % remote_filename)
+ key.restore(days=1) # Shouldn't need this again after 1 day
+ if wait:
+ log.Info("Waiting for file %s to restore from Glacier" % remote_filename)
+ while self.bucket.get_key(key.key).ongoing_restore:
+ time.sleep(60)
+ self.resetConnection()
+ log.Info("File %s was successfully restored from Glacier" % remote_filename)
+ success = True
+ break
+ except Exception, e:
+ log.Warn("Restoration from Glacier for file %s/%s failed (attempt #%d, reason: %s: %s)"
+ "" % (self.straight_url,
+ remote_filename,
+ n,
+ e.__class__.__name__,
+ str(e)), 1)
+ log.Debug("Backtrace of previous error: %s" % (exception_traceback(),))
+ if not success:
+ log.Warn("Giving up trying to restore %s/%s after %d attempts" %
+ (self.straight_url, remote_filename, globals.num_retries))
+ raise BackendException("Error restoring %s/%s from Glacier to S3" % (self.straight_url, remote_filename))
=== modified file 'duplicity/backends/_cf_cloudfiles.py'
--- duplicity/backends/_cf_cloudfiles.py 2014-04-29 23:49:01 +0000
+++ duplicity/backends/_cf_cloudfiles.py 2014-10-15 12:12:04 +0000
@@ -19,11 +19,14 @@
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import os
+import time
import duplicity.backend
+from duplicity import globals
from duplicity import log
-from duplicity import util
-from duplicity.errors import BackendException
+from duplicity.errors import * #@UnusedWildImport
+from duplicity.util import exception_traceback
+from duplicity.backend import retry
class CloudFilesBackend(duplicity.backend.Backend):
"""
@@ -41,17 +44,17 @@
self.resp_exc = ResponseError
conn_kwargs = {}
- if 'CLOUDFILES_USERNAME' not in os.environ:
+ if not os.environ.has_key('CLOUDFILES_USERNAME'):
raise BackendException('CLOUDFILES_USERNAME environment variable'
'not set.')
- if 'CLOUDFILES_APIKEY' not in os.environ:
+ if not os.environ.has_key('CLOUDFILES_APIKEY'):
raise BackendException('CLOUDFILES_APIKEY environment variable not set.')
conn_kwargs['username'] = os.environ['CLOUDFILES_USERNAME']
conn_kwargs['api_key'] = os.environ['CLOUDFILES_APIKEY']
- if 'CLOUDFILES_AUTHURL' in os.environ:
+ if os.environ.has_key('CLOUDFILES_AUTHURL'):
conn_kwargs['authurl'] = os.environ['CLOUDFILES_AUTHURL']
else:
conn_kwargs['authurl'] = consts.default_authurl
@@ -60,43 +63,130 @@
try:
conn = Connection(**conn_kwargs)
- except Exception as e:
+ except Exception, e:
log.FatalError("Connection failed, please check your credentials: %s %s"
- % (e.__class__.__name__, util.uexc(e)),
+ % (e.__class__.__name__, str(e)),
log.ErrorCode.connection_failed)
self.container = conn.create_container(container)
- def _error_code(self, operation, e):
+ def put(self, source_path, remote_filename = None):
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
+ for n in range(1, globals.num_retries+1):
+ log.Info("Uploading '%s/%s' " % (self.container, remote_filename))
+ try:
+ sobject = self.container.create_object(remote_filename)
+ sobject.load_from_filename(source_path.name)
+ return
+ except self.resp_exc, error:
+ log.Warn("Upload of '%s' failed (attempt %d): CloudFiles returned: %s %s"
+ % (remote_filename, n, error.status, error.reason))
+ except Exception, e:
+ log.Warn("Upload of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up uploading '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error uploading '%s'" % remote_filename)
+
+ def get(self, remote_filename, local_path):
+ for n in range(1, globals.num_retries+1):
+ log.Info("Downloading '%s/%s'" % (self.container, remote_filename))
+ try:
+ sobject = self.container.create_object(remote_filename)
+ f = open(local_path.name, 'w')
+ for chunk in sobject.stream():
+ f.write(chunk)
+ local_path.setdata()
+ return
+ except self.resp_exc, resperr:
+ log.Warn("Download of '%s' failed (attempt %s): CloudFiles returned: %s %s"
+ % (remote_filename, n, resperr.status, resperr.reason))
+ except Exception, e:
+ log.Warn("Download of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up downloading '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error downloading '%s/%s'"
+ % (self.container, remote_filename))
+
+ def _list(self):
+ for n in range(1, globals.num_retries+1):
+ log.Info("Listing '%s'" % (self.container))
+ try:
+ # Cloud Files will return a max of 10,000 objects. We have
+ # to make multiple requests to get them all.
+ objs = self.container.list_objects()
+ keys = objs
+ while len(objs) == 10000:
+ objs = self.container.list_objects(marker=keys[-1])
+ keys += objs
+ return keys
+ except self.resp_exc, resperr:
+ log.Warn("Listing of '%s' failed (attempt %s): CloudFiles returned: %s %s"
+ % (self.container, n, resperr.status, resperr.reason))
+ except Exception, e:
+ log.Warn("Listing of '%s' failed (attempt %s): %s: %s"
+ % (self.container, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up listing of '%s' after %s attempts"
+ % (self.container, globals.num_retries))
+ raise BackendException("Error listing '%s'"
+ % (self.container))
+
+ def delete_one(self, remote_filename):
+ for n in range(1, globals.num_retries+1):
+ log.Info("Deleting '%s/%s'" % (self.container, remote_filename))
+ try:
+ self.container.delete_object(remote_filename)
+ return
+ except self.resp_exc, resperr:
+ if n > 1 and resperr.status == 404:
+ # We failed on a timeout, but delete succeeded on the server
+ log.Warn("Delete of '%s' missing after retry - must have succeded earler" % remote_filename )
+ return
+ log.Warn("Delete of '%s' failed (attempt %s): CloudFiles returned: %s %s"
+ % (remote_filename, n, resperr.status, resperr.reason))
+ except Exception, e:
+ log.Warn("Delete of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up deleting '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error deleting '%s/%s'"
+ % (self.container, remote_filename))
+
+ def delete(self, filename_list):
+ for file in filename_list:
+ self.delete_one(file)
+ log.Debug("Deleted '%s/%s'" % (self.container, file))
+
+ @retry
+ def _query_file_info(self, filename, raise_errors=False):
from cloudfiles.errors import NoSuchObject
- if isinstance(e, NoSuchObject):
- return log.ErrorCode.backend_not_found
- elif isinstance(e, self.resp_exc):
- if e.status == 404:
- return log.ErrorCode.backend_not_found
-
- def _put(self, source_path, remote_filename):
- sobject = self.container.create_object(remote_filename)
- sobject.load_from_filename(source_path.name)
-
- def _get(self, remote_filename, local_path):
- sobject = self.container.create_object(remote_filename)
- with open(local_path.name, 'wb') as f:
- for chunk in sobject.stream():
- f.write(chunk)
-
- def _list(self):
- # Cloud Files will return a max of 10,000 objects. We have
- # to make multiple requests to get them all.
- objs = self.container.list_objects()
- keys = objs
- while len(objs) == 10000:
- objs = self.container.list_objects(marker=keys[-1])
- keys += objs
- return keys
-
- def _delete(self, filename):
- self.container.delete_object(filename)
-
- def _query(self, filename):
- sobject = self.container.get_object(filename)
- return {'size': sobject.size}
+ try:
+ sobject = self.container.get_object(filename)
+ return {'size': sobject.size}
+ except NoSuchObject:
+ return {'size': -1}
+ except Exception, e:
+ log.Warn("Error querying '%s/%s': %s"
+ "" % (self.container,
+ filename,
+ str(e)))
+ if raise_errors:
+ raise e
+ else:
+ return {'size': None}
+
+duplicity.backend.register_backend("cf+http", CloudFilesBackend)
=== modified file 'duplicity/backends/_cf_pyrax.py'
--- duplicity/backends/_cf_pyrax.py 2014-04-29 23:49:01 +0000
+++ duplicity/backends/_cf_pyrax.py 2014-10-15 12:12:04 +0000
@@ -19,12 +19,14 @@
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import os
+import time
import duplicity.backend
+from duplicity import globals
from duplicity import log
-from duplicity import util
-from duplicity.errors import BackendException
-
+from duplicity.errors import * # @UnusedWildImport
+from duplicity.util import exception_traceback
+from duplicity.backend import retry
class PyraxBackend(duplicity.backend.Backend):
"""
@@ -43,63 +45,150 @@
conn_kwargs = {}
- if 'CLOUDFILES_USERNAME' not in os.environ:
+ if not os.environ.has_key('CLOUDFILES_USERNAME'):
raise BackendException('CLOUDFILES_USERNAME environment variable'
'not set.')
- if 'CLOUDFILES_APIKEY' not in os.environ:
+ if not os.environ.has_key('CLOUDFILES_APIKEY'):
raise BackendException('CLOUDFILES_APIKEY environment variable not set.')
conn_kwargs['username'] = os.environ['CLOUDFILES_USERNAME']
conn_kwargs['api_key'] = os.environ['CLOUDFILES_APIKEY']
- if 'CLOUDFILES_REGION' in os.environ:
+ if os.environ.has_key('CLOUDFILES_REGION'):
conn_kwargs['region'] = os.environ['CLOUDFILES_REGION']
container = parsed_url.path.lstrip('/')
try:
pyrax.set_credentials(**conn_kwargs)
- except Exception as e:
+ except Exception, e:
log.FatalError("Connection failed, please check your credentials: %s %s"
- % (e.__class__.__name__, util.uexc(e)),
+ % (e.__class__.__name__, str(e)),
log.ErrorCode.connection_failed)
self.client_exc = pyrax.exceptions.ClientException
self.nso_exc = pyrax.exceptions.NoSuchObject
+ self.cloudfiles = pyrax.cloudfiles
self.container = pyrax.cloudfiles.create_container(container)
- def _error_code(self, operation, e):
- if isinstance(e, self.nso_exc):
- return log.ErrorCode.backend_not_found
- elif isinstance(e, self.client_exc):
- if e.status == 404:
- return log.ErrorCode.backend_not_found
- elif hasattr(e, 'http_status'):
- if e.http_status == 404:
- return log.ErrorCode.backend_not_found
-
- def _put(self, source_path, remote_filename):
- self.container.upload_file(source_path.name, remote_filename)
-
- def _get(self, remote_filename, local_path):
- sobject = self.container.get_object(remote_filename)
- with open(local_path.name, 'wb') as f:
- f.write(sobject.get())
+ def put(self, source_path, remote_filename = None):
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
+ for n in range(1, globals.num_retries + 1):
+ log.Info("Uploading '%s/%s' " % (self.container, remote_filename))
+ try:
+ self.container.upload_file(source_path.name, remote_filename)
+ return
+ except self.client_exc, error:
+ log.Warn("Upload of '%s' failed (attempt %d): pyrax returned: %s %s"
+ % (remote_filename, n, error.__class__.__name__, error.message))
+ except Exception, e:
+ log.Warn("Upload of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up uploading '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error uploading '%s'" % remote_filename)
+
+ def get(self, remote_filename, local_path):
+ for n in range(1, globals.num_retries + 1):
+ log.Info("Downloading '%s/%s'" % (self.container, remote_filename))
+ try:
+ sobject = self.container.get_object(remote_filename)
+ f = open(local_path.name, 'w')
+ f.write(sobject.get())
+ local_path.setdata()
+ return
+ except self.nso_exc:
+ return
+ except self.client_exc, resperr:
+ log.Warn("Download of '%s' failed (attempt %s): pyrax returned: %s %s"
+ % (remote_filename, n, resperr.__class__.__name__, resperr.message))
+ except Exception, e:
+ log.Warn("Download of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up downloading '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error downloading '%s/%s'"
+ % (self.container, remote_filename))
def _list(self):
- # Cloud Files will return a max of 10,000 objects. We have
- # to make multiple requests to get them all.
- objs = self.container.get_object_names()
- keys = objs
- while len(objs) == 10000:
- objs = self.container.get_object_names(marker = keys[-1])
- keys += objs
- return keys
-
- def _delete(self, filename):
- self.container.delete_object(filename)
-
- def _query(self, filename):
- sobject = self.container.get_object(filename)
- return {'size': sobject.total_bytes}
+ for n in range(1, globals.num_retries + 1):
+ log.Info("Listing '%s'" % (self.container))
+ try:
+ # Cloud Files will return a max of 10,000 objects. We have
+ # to make multiple requests to get them all.
+ objs = self.container.get_object_names()
+ keys = objs
+ while len(objs) == 10000:
+ objs = self.container.get_object_names(marker = keys[-1])
+ keys += objs
+ return keys
+ except self.client_exc, resperr:
+ log.Warn("Listing of '%s' failed (attempt %s): pyrax returned: %s %s"
+ % (self.container, n, resperr.__class__.__name__, resperr.message))
+ except Exception, e:
+ log.Warn("Listing of '%s' failed (attempt %s): %s: %s"
+ % (self.container, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up listing of '%s' after %s attempts"
+ % (self.container, globals.num_retries))
+ raise BackendException("Error listing '%s'"
+ % (self.container))
+
+ def delete_one(self, remote_filename):
+ for n in range(1, globals.num_retries + 1):
+ log.Info("Deleting '%s/%s'" % (self.container, remote_filename))
+ try:
+ self.container.delete_object(remote_filename)
+ return
+ except self.client_exc, resperr:
+ if n > 1 and resperr.status == 404:
+ # We failed on a timeout, but delete succeeded on the server
+ log.Warn("Delete of '%s' missing after retry - must have succeded earler" % remote_filename)
+ return
+ log.Warn("Delete of '%s' failed (attempt %s): pyrax returned: %s %s"
+ % (remote_filename, n, resperr.__class__.__name__, resperr.message))
+ except Exception, e:
+ log.Warn("Delete of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up deleting '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error deleting '%s/%s'"
+ % (self.container, remote_filename))
+
+ def delete(self, filename_list):
+ for file_ in filename_list:
+ self.delete_one(file_)
+ log.Debug("Deleted '%s/%s'" % (self.container, file_))
+
+ @retry
+ def _query_file_info(self, filename, raise_errors = False):
+ try:
+ sobject = self.container.get_object(filename)
+ return {'size': sobject.total_bytes}
+ except self.nso_exc:
+ return {'size': -1}
+ except Exception, e:
+ log.Warn("Error querying '%s/%s': %s"
+ "" % (self.container,
+ filename,
+ str(e)))
+ if raise_errors:
+ raise e
+ else:
+ return {'size': None}
+
+duplicity.backend.register_backend("cf+http", PyraxBackend)
=== modified file 'duplicity/backends/_ssh_paramiko.py'
--- duplicity/backends/_ssh_paramiko.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/_ssh_paramiko.py 2014-10-15 12:12:04 +0000
@@ -28,6 +28,7 @@
import os
import errno
import sys
+import time
import getpass
import logging
from binascii import hexlify
@@ -35,7 +36,8 @@
import duplicity.backend
from duplicity import globals
from duplicity import log
-from duplicity.errors import BackendException
+from duplicity import util
+from duplicity.errors import *
read_blocksize=65635 # for doing scp retrievals, where we need to read ourselves
@@ -133,7 +135,7 @@
try:
if os.path.isfile("/etc/ssh/ssh_known_hosts"):
self.client.load_system_host_keys("/etc/ssh/ssh_known_hosts")
- except Exception as e:
+ except Exception, e:
raise BackendException("could not load /etc/ssh/ssh_known_hosts, maybe corrupt?")
try:
# use load_host_keys() to signal it's writable to paramiko
@@ -143,7 +145,7 @@
self.client.load_host_keys(file)
else:
self.client._host_keys_filename = file
- except Exception as e:
+ except Exception, e:
raise BackendException("could not load ~/.ssh/known_hosts, maybe corrupt?")
""" the next block reorganizes all host parameters into a
@@ -210,7 +212,7 @@
allow_agent=True,
look_for_keys=True,
key_filename=self.config['identityfile'])
- except Exception as e:
+ except Exception, e:
raise BackendException("ssh connection to %s@%s:%d failed: %s" % (
self.config['user'],
self.config['hostname'],
@@ -228,9 +230,10 @@
else:
try:
self.sftp=self.client.open_sftp()
- except Exception as e:
+ except Exception, e:
raise BackendException("sftp negotiation failed: %s" % e)
+
# move to the appropriate directory, possibly after creating it and its parents
dirs = self.remote_dir.split(os.sep)
if len(dirs) > 0:
@@ -242,104 +245,170 @@
continue
try:
attrs=self.sftp.stat(d)
- except IOError as e:
+ except IOError, e:
if e.errno == errno.ENOENT:
try:
self.sftp.mkdir(d)
- except Exception as e:
+ except Exception, e:
raise BackendException("sftp mkdir %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e))
else:
raise BackendException("sftp stat %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e))
try:
self.sftp.chdir(d)
- except Exception as e:
+ except Exception, e:
raise BackendException("sftp chdir to %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e))
- def _put(self, source_path, remote_filename):
- if globals.use_scp:
- f=file(source_path.name,'rb')
- try:
- chan=self.client.get_transport().open_session()
- chan.settimeout(globals.timeout)
- chan.exec_command("scp -t '%s'" % self.remote_dir) # scp in sink mode uses the arg as base directory
- except Exception as e:
- raise BackendException("scp execution failed: %s" % e)
- # scp protocol: one 0x0 after startup, one after the Create meta, one after saving
- # if there's a problem: 0x1 or 0x02 and some error text
- response=chan.recv(1)
- if (response!="\0"):
- raise BackendException("scp remote error: %s" % chan.recv(-1))
- fstat=os.stat(source_path.name)
- chan.send('C%s %d %s\n' %(oct(fstat.st_mode)[-4:], fstat.st_size, remote_filename))
- response=chan.recv(1)
- if (response!="\0"):
- raise BackendException("scp remote error: %s" % chan.recv(-1))
- chan.sendall(f.read()+'\0')
- f.close()
- response=chan.recv(1)
- if (response!="\0"):
- raise BackendException("scp remote error: %s" % chan.recv(-1))
- chan.close()
- else:
- self.sftp.put(source_path.name,remote_filename)
-
- def _get(self, remote_filename, local_path):
- if globals.use_scp:
- try:
- chan=self.client.get_transport().open_session()
- chan.settimeout(globals.timeout)
- chan.exec_command("scp -f '%s/%s'" % (self.remote_dir,remote_filename))
- except Exception as e:
- raise BackendException("scp execution failed: %s" % e)
-
- chan.send('\0') # overall ready indicator
- msg=chan.recv(-1)
- m=re.match(r"C([0-7]{4})\s+(\d+)\s+(\S.*)$",msg)
- if (m==None or m.group(3)!=remote_filename):
- raise BackendException("scp get %s failed: incorrect response '%s'" % (remote_filename,msg))
- chan.recv(1) # dispose of the newline trailing the C message
-
- size=int(m.group(2))
- togo=size
- f=file(local_path.name,'wb')
- chan.send('\0') # ready for data
- try:
- while togo>0:
- if togo>read_blocksize:
- blocksize = read_blocksize
- else:
- blocksize = togo
- buff=chan.recv(blocksize)
- f.write(buff)
- togo-=len(buff)
- except Exception as e:
- raise BackendException("scp get %s failed: %s" % (remote_filename,e))
-
- msg=chan.recv(1) # check the final status
- if msg!='\0':
- raise BackendException("scp get %s failed: %s" % (remote_filename,chan.recv(-1)))
- f.close()
- chan.send('\0') # send final done indicator
- chan.close()
- else:
- self.sftp.get(remote_filename,local_path.name)
+ def put(self, source_path, remote_filename = None):
+ """transfers a single file to the remote side.
+ In scp mode unavoidable quoting issues will make this fail if the remote directory or file name
+ contain single quotes."""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry
+ time.sleep(self.retry_delay)
+ try:
+ if (globals.use_scp):
+ f=file(source_path.name,'rb')
+ try:
+ chan=self.client.get_transport().open_session()
+ chan.settimeout(globals.timeout)
+ chan.exec_command("scp -t '%s'" % self.remote_dir) # scp in sink mode uses the arg as base directory
+ except Exception, e:
+ raise BackendException("scp execution failed: %s" % e)
+ # scp protocol: one 0x0 after startup, one after the Create meta, one after saving
+ # if there's a problem: 0x1 or 0x02 and some error text
+ response=chan.recv(1)
+ if (response!="\0"):
+ raise BackendException("scp remote error: %s" % chan.recv(-1))
+ fstat=os.stat(source_path.name)
+ chan.send('C%s %d %s\n' %(oct(fstat.st_mode)[-4:], fstat.st_size, remote_filename))
+ response=chan.recv(1)
+ if (response!="\0"):
+ raise BackendException("scp remote error: %s" % chan.recv(-1))
+ chan.sendall(f.read()+'\0')
+ f.close()
+ response=chan.recv(1)
+ if (response!="\0"):
+ raise BackendException("scp remote error: %s" % chan.recv(-1))
+ chan.close()
+ return
+ else:
+ try:
+ self.sftp.put(source_path.name,remote_filename)
+ return
+ except Exception, e:
+ raise BackendException("sftp put of %s (as %s) failed: %s" % (source_path.name,remote_filename,e))
+ except Exception, e:
+ log.Warn("%s (Try %d of %d) Will retry in %d seconds." % (e,n,globals.num_retries,self.retry_delay))
+ raise BackendException("Giving up trying to upload '%s' after %d attempts" % (remote_filename,n))
+
+
+ def get(self, remote_filename, local_path):
+ """retrieves a single file from the remote side.
+ In scp mode unavoidable quoting issues will make this fail if the remote directory or file names
+ contain single quotes."""
+
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry
+ time.sleep(self.retry_delay)
+ try:
+ if (globals.use_scp):
+ try:
+ chan=self.client.get_transport().open_session()
+ chan.settimeout(globals.timeout)
+ chan.exec_command("scp -f '%s/%s'" % (self.remote_dir,remote_filename))
+ except Exception, e:
+ raise BackendException("scp execution failed: %s" % e)
+
+ chan.send('\0') # overall ready indicator
+ msg=chan.recv(-1)
+ m=re.match(r"C([0-7]{4})\s+(\d+)\s+(\S.*)$",msg)
+ if (m==None or m.group(3)!=remote_filename):
+ raise BackendException("scp get %s failed: incorrect response '%s'" % (remote_filename,msg))
+ chan.recv(1) # dispose of the newline trailing the C message
+
+ size=int(m.group(2))
+ togo=size
+ f=file(local_path.name,'wb')
+ chan.send('\0') # ready for data
+ try:
+ while togo>0:
+ if togo>read_blocksize:
+ blocksize = read_blocksize
+ else:
+ blocksize = togo
+ buff=chan.recv(blocksize)
+ f.write(buff)
+ togo-=len(buff)
+ except Exception, e:
+ raise BackendException("scp get %s failed: %s" % (remote_filename,e))
+
+ msg=chan.recv(1) # check the final status
+ if msg!='\0':
+ raise BackendException("scp get %s failed: %s" % (remote_filename,chan.recv(-1)))
+ f.close()
+ chan.send('\0') # send final done indicator
+ chan.close()
+ return
+ else:
+ try:
+ self.sftp.get(remote_filename,local_path.name)
+ return
+ except Exception, e:
+ raise BackendException("sftp get of %s (to %s) failed: %s" % (remote_filename,local_path.name,e))
+ local_path.setdata()
+ except Exception, e:
+ log.Warn("%s (Try %d of %d) Will retry in %d seconds." % (e,n,globals.num_retries,self.retry_delay))
+ raise BackendException("Giving up trying to download '%s' after %d attempts" % (remote_filename,n))
def _list(self):
- # In scp mode unavoidable quoting issues will make this fail if the
- # directory name contains single quotes.
- if globals.use_scp:
- output = self.runremote("ls -1 '%s'" % self.remote_dir, False, "scp dir listing ")
- return output.splitlines()
- else:
- return self.sftp.listdir()
-
- def _delete(self, filename):
- # In scp mode unavoidable quoting issues will cause failures if
- # filenames containing single quotes are encountered.
- if globals.use_scp:
- self.runremote("rm '%s/%s'" % (self.remote_dir, filename), False, "scp rm ")
- else:
- self.sftp.remove(filename)
+ """lists the contents of the one-and-only duplicity dir on the remote side.
+ In scp mode unavoidable quoting issues will make this fail if the directory name
+ contains single quotes."""
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry
+ time.sleep(self.retry_delay)
+ try:
+ if (globals.use_scp):
+ output=self.runremote("ls -1 '%s'" % self.remote_dir,False,"scp dir listing ")
+ return output.splitlines()
+ else:
+ try:
+ return self.sftp.listdir()
+ except Exception, e:
+ raise BackendException("sftp listing of %s failed: %s" % (self.sftp.getcwd(),e))
+ except Exception, e:
+ log.Warn("%s (Try %d of %d) Will retry in %d seconds." % (e,n,globals.num_retries,self.retry_delay))
+ raise BackendException("Giving up trying to list '%s' after %d attempts" % (self.remote_dir,n))
+
+ def delete(self, filename_list):
+ """deletes all files in the list on the remote side. In scp mode unavoidable quoting issues
+ will cause failures if filenames containing single quotes are encountered."""
+ for fn in filename_list:
+ # Try to delete each file several times before giving up completely.
+ for n in range(1, globals.num_retries+1):
+ try:
+ if (globals.use_scp):
+ self.runremote("rm '%s/%s'" % (self.remote_dir,fn),False,"scp rm ")
+ else:
+ try:
+ self.sftp.remove(fn)
+ except Exception, e:
+ raise BackendException("sftp rm %s failed: %s" % (fn,e))
+
+ # If we get here, we deleted this file successfully. Move on to the next one.
+ break
+ except Exception, e:
+ if n == globals.num_retries:
+ log.FatalError(util.uexc(e), log.ErrorCode.backend_error)
+ else:
+ log.Warn("%s (Try %d of %d) Will retry in %d seconds." % (e,n,globals.num_retries,self.retry_delay))
+ time.sleep(self.retry_delay)
def runremote(self,cmd,ignoreexitcode=False,errorprefix=""):
"""small convenience function that opens a shell channel, runs remote command and returns
@@ -348,7 +417,7 @@
chan=self.client.get_transport().open_session()
chan.settimeout(globals.timeout)
chan.exec_command(cmd)
- except Exception as e:
+ except Exception, e:
raise BackendException("%sexecution failed: %s" % (errorprefix,e))
output=chan.recv(-1)
res=chan.recv_exit_status()
@@ -366,7 +435,11 @@
sshconfig = paramiko.SSHConfig()
try:
sshconfig.parse(open(file))
- except Exception as e:
+ except Exception, e:
raise BackendException("could not load '%s', maybe corrupt?" % (file))
return sshconfig.lookup(host)
+
+duplicity.backend.register_backend("sftp", SSHParamikoBackend)
+duplicity.backend.register_backend("scp", SSHParamikoBackend)
+duplicity.backend.register_backend("ssh", SSHParamikoBackend)
=== modified file 'duplicity/backends/_ssh_pexpect.py'
--- duplicity/backends/_ssh_pexpect.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/_ssh_pexpect.py 2014-10-15 12:12:04 +0000
@@ -24,20 +24,19 @@
# have the same syntax. Also these strings will be executed by the
# shell, so shouldn't have strange characters in them.
-from future_builtins import map
-
import re
import string
+import time
import os
import duplicity.backend
from duplicity import globals
from duplicity import log
-from duplicity.errors import BackendException
+from duplicity import pexpect
+from duplicity.errors import * #@UnusedWildImport
class SSHPExpectBackend(duplicity.backend.Backend):
- """This backend copies files using scp. List not supported. Filenames
- should not need any quoting or this will break."""
+ """This backend copies files using scp. List not supported"""
def __init__(self, parsed_url):
"""scpBackend initializer"""
duplicity.backend.Backend.__init__(self, parsed_url)
@@ -77,72 +76,77 @@
def run_scp_command(self, commandline):
""" Run an scp command, responding to password prompts """
- import pexpect
- log.Info("Running '%s'" % commandline)
- child = pexpect.spawn(commandline, timeout = None)
- if globals.ssh_askpass:
- state = "authorizing"
- else:
- state = "copying"
- while 1:
- if state == "authorizing":
- match = child.expect([pexpect.EOF,
- "(?i)timeout, server not responding",
- "(?i)pass(word|phrase .*):",
- "(?i)permission denied",
- "authenticity"])
- log.Debug("State = %s, Before = '%s'" % (state, child.before.strip()))
- if match == 0:
- log.Warn("Failed to authenticate")
- break
- elif match == 1:
- log.Warn("Timeout waiting to authenticate")
- break
- elif match == 2:
- child.sendline(self.password)
- state = "copying"
- elif match == 3:
- log.Warn("Invalid SSH password")
- break
- elif match == 4:
- log.Warn("Remote host authentication failed (missing known_hosts entry?)")
- break
- elif state == "copying":
- match = child.expect([pexpect.EOF,
- "(?i)timeout, server not responding",
- "stalled",
- "authenticity",
- "ETA"])
- log.Debug("State = %s, Before = '%s'" % (state, child.before.strip()))
- if match == 0:
- break
- elif match == 1:
- log.Warn("Timeout waiting for response")
- break
- elif match == 2:
- state = "stalled"
- elif match == 3:
- log.Warn("Remote host authentication failed (missing known_hosts entry?)")
- break
- elif state == "stalled":
- match = child.expect([pexpect.EOF,
- "(?i)timeout, server not responding",
- "ETA"])
- log.Debug("State = %s, Before = '%s'" % (state, child.before.strip()))
- if match == 0:
- break
- elif match == 1:
- log.Warn("Stalled for too long, aborted copy")
- break
- elif match == 2:
- state = "copying"
- child.close(force = True)
- if child.exitstatus != 0:
- raise BackendException("Error running '%s'" % commandline)
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry
+ time.sleep(self.retry_delay)
+ log.Info("Running '%s' (attempt #%d)" % (commandline, n))
+ child = pexpect.spawn(commandline, timeout = None)
+ if globals.ssh_askpass:
+ state = "authorizing"
+ else:
+ state = "copying"
+ while 1:
+ if state == "authorizing":
+ match = child.expect([pexpect.EOF,
+ "(?i)timeout, server not responding",
+ "(?i)pass(word|phrase .*):",
+ "(?i)permission denied",
+ "authenticity"])
+ log.Debug("State = %s, Before = '%s'" % (state, child.before.strip()))
+ if match == 0:
+ log.Warn("Failed to authenticate")
+ break
+ elif match == 1:
+ log.Warn("Timeout waiting to authenticate")
+ break
+ elif match == 2:
+ child.sendline(self.password)
+ state = "copying"
+ elif match == 3:
+ log.Warn("Invalid SSH password")
+ break
+ elif match == 4:
+ log.Warn("Remote host authentication failed (missing known_hosts entry?)")
+ break
+ elif state == "copying":
+ match = child.expect([pexpect.EOF,
+ "(?i)timeout, server not responding",
+ "stalled",
+ "authenticity",
+ "ETA"])
+ log.Debug("State = %s, Before = '%s'" % (state, child.before.strip()))
+ if match == 0:
+ break
+ elif match == 1:
+ log.Warn("Timeout waiting for response")
+ break
+ elif match == 2:
+ state = "stalled"
+ elif match == 3:
+ log.Warn("Remote host authentication failed (missing known_hosts entry?)")
+ break
+ elif state == "stalled":
+ match = child.expect([pexpect.EOF,
+ "(?i)timeout, server not responding",
+ "ETA"])
+ log.Debug("State = %s, Before = '%s'" % (state, child.before.strip()))
+ if match == 0:
+ break
+ elif match == 1:
+ log.Warn("Stalled for too long, aborted copy")
+ break
+ elif match == 2:
+ state = "copying"
+ child.close(force = True)
+ if child.exitstatus == 0:
+ return
+ log.Warn("Running '%s' failed (attempt #%d)" % (commandline, n))
+ log.Warn("Giving up trying to execute '%s' after %d attempts" % (commandline, globals.num_retries))
+ raise BackendException("Error running '%s'" % commandline)
def run_sftp_command(self, commandline, commands):
""" Run an sftp command, responding to password prompts, passing commands from list """
- import pexpect
maxread = 2000 # expected read buffer size
responses = [pexpect.EOF,
"(?i)timeout, server not responding",
@@ -155,69 +159,76 @@
"Couldn't delete file",
"open(.*): Failure"]
max_response_len = max([len(p) for p in responses[1:]])
- log.Info("Running '%s'" % (commandline))
- child = pexpect.spawn(commandline, timeout = None, maxread=maxread)
- cmdloc = 0
- passprompt = 0
- while 1:
- msg = ""
- match = child.expect(responses,
- searchwindowsize=maxread+max_response_len)
- log.Debug("State = sftp, Before = '%s'" % (child.before.strip()))
- if match == 0:
- break
- elif match == 1:
- msg = "Timeout waiting for response"
- break
- if match == 2:
- if cmdloc < len(commands):
- command = commands[cmdloc]
- log.Info("sftp command: '%s'" % (command,))
- child.sendline(command)
- cmdloc += 1
- else:
- command = 'quit'
- child.sendline(command)
- res = child.before
- elif match == 3:
- passprompt += 1
- child.sendline(self.password)
- if (passprompt>1):
- raise BackendException("Invalid SSH password.")
- elif match == 4:
- if not child.before.strip().startswith("mkdir"):
- msg = "Permission denied"
- break
- elif match == 5:
- msg = "Host key authenticity could not be verified (missing known_hosts entry?)"
- break
- elif match == 6:
- if not child.before.strip().startswith("rm"):
- msg = "Remote file or directory does not exist in command='%s'" % (commandline,)
- break
- elif match == 7:
- if not child.before.strip().startswith("Removing"):
+ for n in range(1, globals.num_retries+1):
+ if n > 1:
+ # sleep before retry
+ time.sleep(self.retry_delay)
+ log.Info("Running '%s' (attempt #%d)" % (commandline, n))
+ child = pexpect.spawn(commandline, timeout = None, maxread=maxread)
+ cmdloc = 0
+ passprompt = 0
+ while 1:
+ msg = ""
+ match = child.expect(responses,
+ searchwindowsize=maxread+max_response_len)
+ log.Debug("State = sftp, Before = '%s'" % (child.before.strip()))
+ if match == 0:
+ break
+ elif match == 1:
+ msg = "Timeout waiting for response"
+ break
+ if match == 2:
+ if cmdloc < len(commands):
+ command = commands[cmdloc]
+ log.Info("sftp command: '%s'" % (command,))
+ child.sendline(command)
+ cmdloc += 1
+ else:
+ command = 'quit'
+ child.sendline(command)
+ res = child.before
+ elif match == 3:
+ passprompt += 1
+ child.sendline(self.password)
+ if (passprompt>1):
+ raise BackendException("Invalid SSH password.")
+ elif match == 4:
+ if not child.before.strip().startswith("mkdir"):
+ msg = "Permission denied"
+ break
+ elif match == 5:
+ msg = "Host key authenticity could not be verified (missing known_hosts entry?)"
+ break
+ elif match == 6:
+ if not child.before.strip().startswith("rm"):
+ msg = "Remote file or directory does not exist in command='%s'" % (commandline,)
+ break
+ elif match == 7:
+ if not child.before.strip().startswith("Removing"):
+ msg = "Could not delete file in command='%s'" % (commandline,)
+ break;
+ elif match == 8:
msg = "Could not delete file in command='%s'" % (commandline,)
- break;
- elif match == 8:
- msg = "Could not delete file in command='%s'" % (commandline,)
- break
- elif match == 9:
- msg = "Could not open file in command='%s'" % (commandline,)
- break
- child.close(force = True)
- if child.exitstatus == 0:
- return res
- else:
- raise BackendException("Error running '%s': %s" % (commandline, msg))
+ break
+ elif match == 9:
+ msg = "Could not open file in command='%s'" % (commandline,)
+ break
+ child.close(force = True)
+ if child.exitstatus == 0:
+ return res
+ log.Warn("Running '%s' with commands:\n %s\n failed (attempt #%d): %s" % (commandline, "\n ".join(commands), n, msg))
+ raise BackendException("Giving up trying to execute '%s' with commands:\n %s\n after %d attempts" % (commandline, "\n ".join(commands), globals.num_retries))
- def _put(self, source_path, remote_filename):
+ def put(self, source_path, remote_filename = None):
if globals.use_scp:
- self.put_scp(source_path, remote_filename)
+ self.put_scp(source_path, remote_filename = remote_filename)
else:
- self.put_sftp(source_path, remote_filename)
+ self.put_sftp(source_path, remote_filename = remote_filename)
- def put_sftp(self, source_path, remote_filename):
+ def put_sftp(self, source_path, remote_filename = None):
+ """Use sftp to copy source_dir/filename to remote computer"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
commands = ["put \"%s\" \"%s.%s.part\"" %
(source_path.name, self.remote_prefix, remote_filename),
"rename \"%s.%s.part\" \"%s%s\"" %
@@ -227,36 +238,53 @@
self.host_string))
self.run_sftp_command(commandline, commands)
- def put_scp(self, source_path, remote_filename):
+ def put_scp(self, source_path, remote_filename = None):
+ """Use scp to copy source_dir/filename to remote computer"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
commandline = "%s %s %s %s:%s%s" % \
(self.scp_command, globals.ssh_options, source_path.name, self.host_string,
self.remote_prefix, remote_filename)
self.run_scp_command(commandline)
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
if globals.use_scp:
self.get_scp(remote_filename, local_path)
else:
self.get_sftp(remote_filename, local_path)
def get_sftp(self, remote_filename, local_path):
+ """Use sftp to get a remote file"""
commands = ["get \"%s%s\" \"%s\"" %
(self.remote_prefix, remote_filename, local_path.name)]
commandline = ("%s %s %s" % (self.sftp_command,
globals.ssh_options,
self.host_string))
self.run_sftp_command(commandline, commands)
+ local_path.setdata()
+ if not local_path.exists():
+ raise BackendException("File %s not found locally after get "
+ "from backend" % local_path.name)
def get_scp(self, remote_filename, local_path):
+ """Use scp to get a remote file"""
commandline = "%s %s %s:%s%s %s" % \
(self.scp_command, globals.ssh_options, self.host_string, self.remote_prefix,
remote_filename, local_path.name)
self.run_scp_command(commandline)
+ local_path.setdata()
+ if not local_path.exists():
+ raise BackendException("File %s not found locally after get "
+ "from backend" % local_path.name)
def _list(self):
- # Note that this command can get confused when dealing with
- # files with newlines in them, as the embedded newlines cannot
- # be distinguished from the file boundaries.
+ """
+ List files available for scp
+
+ Note that this command can get confused when dealing with
+ files with newlines in them, as the embedded newlines cannot
+ be distinguished from the file boundaries.
+ """
dirs = self.remote_dir.split(os.sep)
if len(dirs) > 0:
if not dirs[0] :
@@ -273,10 +301,18 @@
l = self.run_sftp_command(commandline, commands).split('\n')[1:]
- return [x for x in map(string.strip, l) if x]
+ return filter(lambda x: x, map(string.strip, l))
- def _delete(self, filename):
+ def delete(self, filename_list):
+ """
+ Runs sftp rm to delete files. Files must not require quoting.
+ """
commands = ["cd \"%s\"" % (self.remote_dir,)]
- commands.append("rm \"%s\"" % filename)
+ for fn in filename_list:
+ commands.append("rm \"%s\"" % fn)
commandline = ("%s %s %s" % (self.sftp_command, globals.ssh_options, self.host_string))
self.run_sftp_command(commandline, commands)
+
+duplicity.backend.register_backend("ssh", SSHPExpectBackend)
+duplicity.backend.register_backend("scp", SSHPExpectBackend)
+duplicity.backend.register_backend("sftp", SSHPExpectBackend)
=== modified file 'duplicity/backends/botobackend.py'
--- duplicity/backends/botobackend.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/botobackend.py 2014-10-15 12:12:04 +0000
@@ -22,12 +22,18 @@
import duplicity.backend
from duplicity import globals
+import sys
if globals.s3_use_multiprocessing:
- from ._boto_multi import BotoBackend
+ if sys.version_info[:2] < (2, 6):
+ print "Sorry, S3 multiprocessing requires version 2.6 or later of python"
+ sys.exit(1)
+ from _boto_multi import BotoBackend as BotoMultiUploadBackend
+ duplicity.backend.register_backend("gs", BotoMultiUploadBackend)
+ duplicity.backend.register_backend("s3", BotoMultiUploadBackend)
+ duplicity.backend.register_backend("s3+http", BotoMultiUploadBackend)
else:
- from ._boto_single import BotoBackend
-
-duplicity.backend.register_backend("gs", BotoBackend)
-duplicity.backend.register_backend("s3", BotoBackend)
-duplicity.backend.register_backend("s3+http", BotoBackend)
+ from _boto_single import BotoBackend as BotoSingleUploadBackend
+ duplicity.backend.register_backend("gs", BotoSingleUploadBackend)
+ duplicity.backend.register_backend("s3", BotoSingleUploadBackend)
+ duplicity.backend.register_backend("s3+http", BotoSingleUploadBackend)
=== modified file 'duplicity/backends/cfbackend.py'
--- duplicity/backends/cfbackend.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/cfbackend.py 2014-10-15 12:12:04 +0000
@@ -18,13 +18,10 @@
# along with duplicity; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-import duplicity.backend
from duplicity import globals
if (globals.cf_backend and
globals.cf_backend.lower().strip() == 'pyrax'):
- from ._cf_pyrax import PyraxBackend as CFBackend
+ import _cf_pyrax
else:
- from ._cf_cloudfiles import CloudFilesBackend as CFBackend
-
-duplicity.backend.register_backend("cf+http", CFBackend)
+ import _cf_cloudfiles
=== removed file 'duplicity/backends/copycombackend.py'
--- duplicity/backends/copycombackend.py 2014-06-16 15:51:30 +0000
+++ duplicity/backends/copycombackend.py 1970-01-01 00:00:00 +0000
@@ -1,319 +0,0 @@
-# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4; coding: utf-8 -*-
-#
-# Copyright 2014 Marco Trevisan (Treviño) <mail@xxxxxxxxx>
-#
-# This file is part of duplicity.
-#
-# Duplicity is free software; you can redistribute it and/or modify it
-# under the terms of the GNU General Public License as published by the
-# Free Software Foundation; either version 2 of the License, or (at your
-# option) any later version.
-#
-# Duplicity is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with duplicity; if not, write to the Free Software Foundation,
-# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-import os.path
-import sys
-
-import duplicity.backend
-from duplicity import log
-from duplicity.errors import BackendException
-
-class CoPyCloud:
- API_URI = 'https://api.copy.com'
- DEFAULT_ENCODING = 'latin-1'
- DEFAULT_HEADERS = {'X-Client-Type': 'api', 'X-Api-Version': '1.0',
- 'X-Authorization': '', 'Accept': 'application/json' }
-
- PART_MAX_SIZE = 1024*1024
- PARTS_HEADER_FMT = '!IIIIII'
- PARTS_HEADER_SIG = 0xba5eba11
- PARTS_HEADER_VERSION = 1
- PART_ITEM_FMT = '!IIII73sIIII'
- PART_ITEM_SIG = 0xcab005e5
- PART_ITEM_VERSION = 1
-
- class Error(Exception):
- def __init__(self, message):
- Exception.__init__(self, message)
-
- def __init__(self, username, password):
- import urllib3
- self.http = urllib3.connection_from_url(self.API_URI, block=True, maxsize=1)
- res = self.__post_req('auth_user', {'username': username, 'password' : password})
-
- if not res or 'auth_token' not in res:
- raise CoPyCloud.Error("Invalid Login")
-
- self.DEFAULT_HEADERS['X-Authorization'] = res['auth_token'].encode('ascii','ignore')
-
-
- def __req(self, req_type, method, params={}, headers={}):
- import json
- headers.update(self.DEFAULT_HEADERS)
- method = '/'+method if method[0] != '/' else method
-
- if isinstance(params, dict):
- res = self.http.request_encode_body(req_type, method, {'data': json.dumps(params)},
- headers, encode_multipart=False)
- else:
- res = self.http.urlopen(req_type, method, params, headers)
-
- if res.status != 200:
- raise CoPyCloud.Error("Got HTTP error "+str(res.status))
-
- try:
- if 'content-type' in res.headers and res.headers['content-type'] == 'application/json':
- jd = json.loads(res.data.decode(self.DEFAULT_ENCODING), self.DEFAULT_ENCODING)
-
- if jd and 'result' in jd and jd['result'] == 'error':
- raise CoPyCloud.Error("Error %s: %s" % (jd['error_code'], jd['error_string']))
-
- return jd
- except ValueError:
- pass
-
- return res.data
-
- def __post_req(self, method, params={}, headers={}):
- return self.__req('POST', method, params, headers)
-
- def __get_req(self, method, headers={}):
- return self.__req('GET', method, headers)
-
- def __binary_parts_req(self, method, parts, share_id=0, headers={}):
- if not len(parts):
- return
-
- import struct
- invalid_parts = []
- header_size = struct.calcsize(self.PARTS_HEADER_FMT)
- item_base_size = struct.calcsize(self.PART_ITEM_FMT)
- items_data_size = sum([p['size'] if 'data' in p else 0 for p in parts])
- buf = bytearray(header_size + item_base_size * len(parts) + items_data_size)
- error_code = 0
- padding = 0
- pos = 0
-
- struct.pack_into(self.PARTS_HEADER_FMT, buf, pos, self.PARTS_HEADER_SIG, header_size,
- self.PARTS_HEADER_VERSION, len(buf) - header_size, len(parts), error_code)
- pos += header_size
-
- for part in parts:
- data_size = part['size'] if 'data' in part else 0
- part_size = item_base_size + data_size
- fingerprint = bytes(part['fingerprint'].encode(self.DEFAULT_ENCODING))
- struct.pack_into(self.PART_ITEM_FMT, buf, pos, self.PART_ITEM_SIG, part_size,
- self.PART_ITEM_VERSION, share_id, fingerprint, part['size'],
- data_size, error_code, padding)
-
- pos += item_base_size
-
- if data_size > 0:
- buf[pos:pos+data_size] = part['data']
- pos += data_size
-
- ret = self.__post_req(method, buf, {'Content-Type': 'application/octet-stream'})
-
- pos = 0
- r = (sig, header_size, version, parts_size, parts_num, error) = \
- struct.unpack_from(self.PARTS_HEADER_FMT, ret, pos)
- pos += header_size
-
- if sig != self.PARTS_HEADER_SIG:
- raise CoPyCloud.Error("Invalid binary header signature from server")
- if error != 0:
- raise CoPyCloud.Error("Invalid binary response from server: "+str(ret[pos:]))
- if header_size != struct.calcsize(self.PARTS_HEADER_FMT):
- raise CoPyCloud.Error("Invalid binary header size from server")
- if version != self.PARTS_HEADER_VERSION:
- raise CoPyCloud.Error("Binary header version mismatch")
- if parts_num != len(parts):
- raise CoPyCloud.Error("Part count mismatch")
-
- for part in parts:
- (sig, item_size, version, share_id, fingerprint, remote_size, data_size, error, padding) = \
- struct.unpack_from(self.PART_ITEM_FMT, ret, pos)
-
- if sig != self.PART_ITEM_SIG:
- raise CoPyCloud.Error("Invalid binary part item header signature from server")
- if version != self.PART_ITEM_VERSION:
- raise CoPyCloud.Error("Binary part item version mismatch")
- if fingerprint[:-1] != bytes(part['fingerprint'].encode(self.DEFAULT_ENCODING)):
- raise CoPyCloud.Error("Part %u fingerprint mismatch" % part['offset'])
-
- if 'data' in part:
- if error != 0:
- offset = pos + item_base_size
- raise CoPyCloud.Error("Invalid binary part item: "+str(ret[offset:offset+data_size]))
- if item_size != item_base_size:
- raise CoPyCloud.Error("Invalid binary part item size received from server")
- if remote_size != part['size']:
- raise CoPyCloud.Error("Part %u local/remote size mismatch" % part['offset'])
- else:
- if error != 0 or remote_size != part['size']:
- invalid_parts.append(part)
-
- pos += item_base_size
-
- return invalid_parts
-
- def __update_objects(self, parameters):
- p = [parameters] if isinstance(parameters, dict) else parameters
- self.__post_req('update_objects', {'meta': p})
-
- def __sanitize_path(self, path):
- path = '/' if not path or not len(path) else path
- return '/'+path if path[0] != '/' else path
-
- def __get_file_parts(self, f):
- import hashlib
- parts = []
- size = os.path.getsize(f.name)
-
- while f.tell() < size:
- offset = f.tell()
- part_data = f.read(self.PART_MAX_SIZE)
- fingerprint = hashlib.md5(part_data).hexdigest() + hashlib.sha1(part_data).hexdigest()
- parts.append({'fingerprint': fingerprint, 'offset': offset, 'size': len(part_data)})
-
- if f.tell() != size:
- raise CoPyCloud.Error("Impossible to generate full parts for file "+f.name)
-
- return parts
-
- def __fill_file_parts(self, f, parts):
- for part in parts:
- f.seek(part['offset'])
- part['data'] = f.read(part['size'])
-
-
- def list_files(self, path=None, max_items=sys.maxsize):
- path = path = self.__sanitize_path(path)
- parameters = {'path': path, 'max_items': max_items}
- res = self.__post_req('list_objects', parameters)
-
- if not res or 'children' not in res:
- raise CoPyCloud.Error("Impossible to retrieve the files")
-
- if 'object' in res and 'type' in res['object'] and res['object']['type'] == 'file':
- return res['object']
-
- return res['children']
-
- def remove(self, paths):
- if isinstance(paths, basestring):
- if not len(paths):
- raise CoPyCloud.Error("Impossible to remove a file with an empty path")
- paths = [paths]
- if paths is None:
- raise CoPyCloud.Error("Impossible to remove files with invalid path")
-
- self.__update_objects([{'action': 'remove', 'path': self.__sanitize_path(p)} for p in paths])
-
- def download(self, path):
- if not path or not len(path):
- raise CoPyCloud.Error("Impossible to download a file with an empty path")
-
- if not len(self.list_files(path, max_items=1)):
- raise CoPyCloud.Error("Impossible to download '"+path+"'")
-
- return self.__post_req('download_object', {'path': path})
-
- def upload(self, source, dest, parallel=5, share_id=0):
- try:
- f = open(source, 'rb')
- except Exception as e:
- raise CoPyCloud.Error("Impossible to open source file "+ str(e))
-
- parts = self.__get_file_parts(f)
- parts_chunks = [parts[i:i+parallel] for i in range(0, len(parts), parallel)]
-
- for parts_chunk in parts_chunks:
- missing_parts = self.__binary_parts_req('has_object_parts', parts_chunk)
-
- if len(missing_parts):
- self.__fill_file_parts(f, missing_parts)
- self.__binary_parts_req('send_object_parts', missing_parts)
-
- for part in parts_chunk:
- del(part['data'])
-
- update_params = {'action': 'create', 'object_type': 'file', 'path': self.__sanitize_path(dest),
- 'size': os.path.getsize(f.name), 'parts': parts}
- self.__update_objects(update_params)
-
- f.close()
-
-
-class CopyComBackend(duplicity.backend.Backend):
- """Copy.com duplicity backend"""
-
- def __init__(self, parsed_url):
- """Connect to Copy.com"""
- duplicity.backend.Backend.__init__(self, parsed_url)
- try:
- self.copy = CoPyCloud(parsed_url.username, self.get_password())
- except CoPyCloud.Error as e:
- raise BackendException(e)
-
- [self.folder] = parsed_url.path[1:].split('/')
-
- def _list(self):
- """List files in folder"""
- try:
- return [os.path.basename(f['path']) for f in self.copy.list_files(self.folder)]
- except CoPyCloud.Error as e:
- raise BackendException(e)
-
- def _query(self, filename):
- try:
- file_info = self.copy.list_files(os.path.join(self.folder, filename))
- return {'size': int(file_info['size']) if 'size' in file_info else -1}
- except CoPyCloud.Error as e:
- raise BackendException(e)
-
- def _put(self, source_path, remote_filename):
- """Upload local file to cloud"""
- try:
- self.copy.upload(source_path.get_canonical(), os.path.join(self.folder, remote_filename))
- except CoPyCloud.Error as e:
- raise BackendException(e)
-
- def _get(self, remote_filename, local_path):
- """Save cloud file locally"""
- try:
- raw = self.copy.download(os.path.join(self.folder, remote_filename))
- except CoPyCloud.Error as e:
- raise BackendException(e)
-
- f = local_path.open(mode='wb')
- f.write(raw)
- f.close()
-
- def _delete(self, filename):
- """Delete a file"""
- try:
- self.copy.remove(os.path.join(self.folder, filename))
- except CoPyCloud.Error as e:
- raise BackendException(e)
-
-''' This must be disabled here, because if a file in list does not exist, the
- Copy server will stop deleting the subsequent stuff, raising an error,
- making test_delete_list to fail.
- def _delete_list(self, filenames):
- """Delete list of files"""
- try:
- self.copy.remove([os.path.join(self.folder, f) for f in filenames])
- except CoPyCloud.Error as e:
- if 'Error 1024' in e:
- pass # "Ignore file can't be located error, in this case"
-'''
-
-duplicity.backend.register_backend('copy', CopyComBackend)
=== modified file 'duplicity/backends/dpbxbackend.py'
--- duplicity/backends/dpbxbackend.py 2014-05-10 10:49:20 +0000
+++ duplicity/backends/dpbxbackend.py 2014-10-15 12:12:04 +0000
@@ -29,14 +29,16 @@
import urllib
import re
import locale, sys
-from functools import reduce
import traceback, StringIO
+from exceptions import Exception
import duplicity.backend
+from duplicity import globals
from duplicity import log
-from duplicity import util
-from duplicity.errors import BackendException
+from duplicity.errors import *
+from duplicity import tempdir
+from duplicity.backend import retry_fatal
# This application key is registered in my name (jno at pisem dot net).
@@ -59,13 +61,13 @@
_TOKEN_CACHE_FILE = os.path.expanduser("~/.dropbox.token_store.txt")
def log_exception(e):
- log.Error('Exception [%s]:'%(e,))
- f = StringIO.StringIO()
- traceback.print_exc(file=f)
- f.seek(0)
- for s in f.readlines():
- log.Error('| '+s.rstrip())
- f.close()
+ log.Error('Exception [%s]:'%(e,))
+ f = StringIO.StringIO()
+ traceback.print_exc(file=f)
+ f.seek(0)
+ for s in f.readlines():
+ log.Error('| '+s.rstrip())
+ f.close()
def command(login_required=True):
"""a decorator for handling authentication and exceptions"""
@@ -73,19 +75,19 @@
def wrapper(self, *args):
from dropbox import rest
if login_required and not self.sess.is_linked():
- raise BackendException("dpbx Cannot login: check your credentials", log.ErrorCode.dpbx_nologin)
- return
+ log.FatalError("dpbx Cannot login: check your credentials",log.ErrorCode.dpbx_nologin)
+ return
try:
return f(self, *args)
- except TypeError as e:
+ except TypeError, e:
log_exception(e)
- raise BackendException('dpbx type error "%s"' % (e,))
- except rest.ErrorResponse as e:
- msg = e.user_error_msg or util.uexc(e)
+ log.FatalError('dpbx type error "%s"' % (str(e),), log.ErrorCode.backend_code_error)
+ except rest.ErrorResponse, e:
+ msg = e.user_error_msg or str(e)
log.Error('dpbx error: %s' % (msg,), log.ErrorCode.backend_command_error)
raise e
- except Exception as e:
+ except Exception, e:
log_exception(e)
log.Error('dpbx code error "%s"' % (e,), log.ErrorCode.backend_code_error)
raise e
@@ -99,12 +101,13 @@
def __init__(self, parsed_url):
duplicity.backend.Backend.__init__(self, parsed_url)
+ global client, rest, session
from dropbox import client, rest, session
class StoredSession(session.DropboxSession):
"""a wrapper around DropboxSession that stores a token to a file on disk"""
TOKEN_FILE = _TOKEN_CACHE_FILE
-
+
def load_creds(self):
try:
f = open(self.TOKEN_FILE)
@@ -114,21 +117,21 @@
log.Info( "[loaded access token]" )
except IOError:
pass # don't worry if it's not there
-
+
def write_creds(self, token):
open(self.TOKEN_FILE, 'w').close() # create/reset file
- os.chmod(self.TOKEN_FILE, 0o600) # set it -rw------ (NOOP in Windows?)
+ os.chmod(self.TOKEN_FILE,0600) # set it -rw------ (NOOP in Windows?)
# now write the content
f = open(self.TOKEN_FILE, 'w')
f.write("|".join([token.key, token.secret]))
f.close()
-
+
def delete_creds(self):
os.unlink(self.TOKEN_FILE)
-
+
def link(self):
if not sys.stdout.isatty() or not sys.stdin.isatty() :
- log.FatalError('dpbx error: cannot interact, but need human attention', log.ErrorCode.backend_command_error)
+ log.FatalError('dpbx error: cannot interact, but need human attention', log.ErrorCode.backend_command_error)
request_token = self.obtain_request_token()
url = self.build_authorize_url(request_token)
print
@@ -136,10 +139,10 @@
print "url:", url
print "Please authorize in the browser. After you're done, press enter."
raw_input()
-
+
self.obtain_access_token(request_token)
self.write_creds(self.token)
-
+
def unlink(self):
self.delete_creds()
session.DropboxSession.unlink(self)
@@ -155,29 +158,32 @@
def login(self):
if not self.sess.is_linked():
- try: # to login to the box
- self.sess.link()
- except rest.ErrorResponse as e:
- log.FatalError('dpbx Error: %s\n' % util.uexc(e), log.ErrorCode.dpbx_nologin)
- if not self.sess.is_linked(): # stil not logged in
- log.FatalError("dpbx Cannot login: check your credentials",log.ErrorCode.dpbx_nologin)
-
- def _error_code(self, operation, e):
- from dropbox import rest
- if isinstance(e, rest.ErrorResponse):
- if e.status == 404:
- return log.ErrorCode.backend_not_found
-
+ try: # to login to the box
+ self.sess.link()
+ except rest.ErrorResponse, e:
+ log.FatalError('dpbx Error: %s\n' % str(e), log.ErrorCode.dpbx_nologin)
+ if not self.sess.is_linked(): # stil not logged in
+ log.FatalError("dpbx Cannot login: check your credentials",log.ErrorCode.dpbx_nologin)
+
+ @retry_fatal
@command()
- def _put(self, source_path, remote_filename):
+ def put(self, source_path, remote_filename = None):
+ """Transfer source_path to remote_filename"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/'))
remote_path = os.path.join(remote_dir, remote_filename).rstrip()
+
from_file = open(source_path.name, "rb")
+
resp = self.api_client.put_file(remote_path, from_file)
log.Debug( 'dpbx,put(%s,%s): %s'%(source_path.name, remote_path, resp))
+ @retry_fatal
@command()
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
+ """Get remote filename, saving it to local_path"""
remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
to_file = open( local_path.name, 'wb' )
@@ -190,8 +196,10 @@
local_path.setdata()
+ @retry_fatal
@command()
- def _list(self):
+ def _list(self,none=None):
+ """List files in directory"""
# Do a long listing to avoid connection reset
remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
resp = self.api_client.metadata(remote_dir)
@@ -206,37 +214,43 @@
l.append(name.encode(encoding))
return l
+ @retry_fatal
@command()
- def _delete(self, filename):
+ def delete(self, filename_list):
+ """Delete files in filename_list"""
+ if not filename_list :
+ log.Debug('dpbx.delete(): no op')
+ return
remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
- remote_name = os.path.join( remote_dir, filename )
- resp = self.api_client.file_delete( remote_name )
- log.Debug('dpbx.delete(%s): %s'%(remote_name,resp))
+ for filename in filename_list:
+ remote_name = os.path.join( remote_dir, filename )
+ resp = self.api_client.file_delete( remote_name )
+ log.Debug('dpbx.delete(%s): %s'%(remote_name,resp))
@command()
- def _close(self):
- """close backend session? no! just "flush" the data"""
- info = self.api_client.account_info()
- log.Debug('dpbx.close():')
- for k in info :
- log.Debug(':: %s=[%s]' % (k, info[k]))
- entries = []
- more = True
- cursor = None
- while more :
- info = self.api_client.delta(cursor)
- if info.get('reset', False) :
- log.Debug("delta returned True value for \"reset\", no matter")
- cursor = info.get('cursor', None)
- more = info.get('more', False)
- entr = info.get('entries', [])
- entries += entr
- for path, meta in entries:
- mm = meta and 'ok' or 'DELETE'
- log.Info(':: :: [%s] %s' % (path, mm))
- if meta :
- for k in meta :
- log.Debug(':: :: :: %s=[%s]' % (k, meta[k]))
+ def close(self):
+ """close backend session? no! just "flush" the data"""
+ info = self.api_client.account_info()
+ log.Debug('dpbx.close():')
+ for k in info :
+ log.Debug(':: %s=[%s]'%(k,info[k]))
+ entries = []
+ more = True
+ cursor = None
+ while more :
+ info = self.api_client.delta(cursor)
+ if info.get('reset',False) :
+ log.Debug("delta returned True value for \"reset\", no matter")
+ cursor = info.get('cursor',None)
+ more = info.get('more',False)
+ entr = info.get('entries',[])
+ entries += entr
+ for path,meta in entries:
+ mm = meta and 'ok' or 'DELETE'
+ log.Info(':: :: [%s] %s'%(path,mm))
+ if meta :
+ for k in meta :
+ log.Debug(':: :: :: %s=[%s]'%(k,meta[k]))
def _mkdir(self, path):
"""create a new directory"""
@@ -244,7 +258,7 @@
log.Debug('dpbx._mkdir(%s): %s'%(path,resp))
def etacsufbo(s):
- return ''.join(reduce(lambda x,y:(x and len(x[-1])==1)and(x.append(y+
- x.pop(-1))and x or x)or(x+[y]),s,[]))
+ return ''.join(reduce(lambda x,y:(x and len(x[-1])==1)and(x.append(y+
+ x.pop(-1))and x or x)or(x+[y]),s,[]))
duplicity.backend.register_backend("dpbx", DPBXBackend)
=== modified file 'duplicity/backends/ftpbackend.py'
--- duplicity/backends/ftpbackend.py 2014-04-26 12:54:37 +0000
+++ duplicity/backends/ftpbackend.py 2014-10-15 12:12:04 +0000
@@ -25,6 +25,7 @@
import duplicity.backend
from duplicity import globals
from duplicity import log
+from duplicity.errors import * #@UnusedWildImport
from duplicity import tempdir
class FTPBackend(duplicity.backend.Backend):
@@ -64,7 +65,7 @@
# This squelches the "file not found" result from ncftpls when
# the ftp backend looks for a collection that does not exist.
# version 3.2.2 has error code 5, 1280 is some legacy value
- self.popen_breaks[ 'ncftpls' ] = [ 5, 1280 ]
+ self.popen_persist_breaks[ 'ncftpls' ] = [ 5, 1280 ]
# Use an explicit directory name.
if self.url_string[-1] != '/':
@@ -87,28 +88,37 @@
if parsed_url.port != None and parsed_url.port != 21:
self.flags += " -P '%s'" % (parsed_url.port)
- def _put(self, source_path, remote_filename):
+ def put(self, source_path, remote_filename = None):
+ """Transfer source_path to remote_filename"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
commandline = "ncftpput %s -m -V -C '%s' '%s'" % \
(self.flags, source_path.name, remote_path)
- self.subprocess_popen(commandline)
+ self.run_command_persist(commandline)
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
+ """Get remote filename, saving it to local_path"""
remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
commandline = "ncftpget %s -V -C '%s' '%s' '%s'" % \
(self.flags, self.parsed_url.hostname, remote_path.lstrip('/'), local_path.name)
- self.subprocess_popen(commandline)
+ self.run_command_persist(commandline)
+ local_path.setdata()
def _list(self):
+ """List files in directory"""
# Do a long listing to avoid connection reset
commandline = "ncftpls %s -l '%s'" % (self.flags, self.url_string)
- _, l, _ = self.subprocess_popen(commandline)
+ l = self.popen_persist(commandline).split('\n')
+ l = filter(lambda x: x, l)
# Look for our files as the last element of a long list line
- return [x.split()[-1] for x in l.split('\n') if x and not x.startswith("total ")]
+ return [x.split()[-1] for x in l if not x.startswith("total ")]
- def _delete(self, filename):
- commandline = "ncftpls %s -l -X 'DELE %s' '%s'" % \
- (self.flags, filename, self.url_string)
- self.subprocess_popen(commandline)
+ def delete(self, filename_list):
+ """Delete files in filename_list"""
+ for filename in filename_list:
+ commandline = "ncftpls %s -l -X 'DELE %s' '%s'" % \
+ (self.flags, filename, self.url_string)
+ self.popen_persist(commandline)
duplicity.backend.register_backend("ftp", FTPBackend)
=== modified file 'duplicity/backends/ftpsbackend.py'
--- duplicity/backends/ftpsbackend.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/ftpsbackend.py 2014-10-15 12:12:04 +0000
@@ -28,6 +28,7 @@
import duplicity.backend
from duplicity import globals
from duplicity import log
+from duplicity.errors import *
from duplicity import tempdir
class FTPSBackend(duplicity.backend.Backend):
@@ -78,35 +79,49 @@
os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout)
os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries)
os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt)
- os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))
+ os.write(self.tempfile, "open %s ftps://%s\n" % (self.portflag, self.parsed_url.hostname))
# allow .netrc auth by only setting user/pass when user was actually given
if self.parsed_url.username:
os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password))
os.close(self.tempfile)
- def _put(self, source_path, remote_filename):
+ self.flags = "-f %s" % self.tempname
+
+ def put(self, source_path, remote_filename = None):
+ """Transfer source_path to remote_filename"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
commandline = "lftp -c 'source %s;put \'%s\' -o \'%s\''" % \
(self.tempname, source_path.name, remote_path)
- self.subprocess_popen(commandline)
+ l = self.run_command_persist(commandline)
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
+ """Get remote filename, saving it to local_path"""
remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
commandline = "lftp -c 'source %s;get %s -o %s'" % \
(self.tempname, remote_path.lstrip('/'), local_path.name)
- self.subprocess_popen(commandline)
+ self.run_command_persist(commandline)
+ local_path.setdata()
def _list(self):
+ """List files in directory"""
# Do a long listing to avoid connection reset
remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
commandline = "lftp -c 'source %s;ls \'%s\''" % (self.tempname, remote_dir)
- _, l, _ = self.subprocess_popen(commandline)
+ l = self.popen_persist(commandline).split('\n')
+ l = filter(lambda x: x, l)
# Look for our files as the last element of a long list line
- return [x.split()[-1] for x in l.split('\n') if x]
+ return [x.split()[-1] for x in l]
- def _delete(self, filename):
- remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
- commandline = "lftp -c 'source %s;cd \'%s\';rm \'%s\''" % (self.tempname, remote_dir, filename)
- self.subprocess_popen(commandline)
+ def delete(self, filename_list):
+ """Delete files in filename_list"""
+ filelist = ""
+ for filename in filename_list:
+ filelist += "\'%s\' " % filename
+ if filelist.rstrip():
+ remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
+ commandline = "lftp -c 'source %s;cd \'%s\';rm %s'" % (self.tempname, remote_dir, filelist.rstrip())
+ self.popen_persist(commandline)
duplicity.backend.register_backend("ftps", FTPSBackend)
=== modified file 'duplicity/backends/gdocsbackend.py'
--- duplicity/backends/gdocsbackend.py 2014-04-21 19:21:45 +0000
+++ duplicity/backends/gdocsbackend.py 2014-10-15 12:12:04 +0000
@@ -23,7 +23,9 @@
import urllib
import duplicity.backend
-from duplicity.errors import BackendException
+from duplicity.backend import retry
+from duplicity import log
+from duplicity.errors import * #@UnusedWildImport
class GDocsBackend(duplicity.backend.Backend):
@@ -51,14 +53,14 @@
self.client = gdata.docs.client.DocsClient(source='duplicity $version')
self.client.ssl = True
self.client.http_client.debug = False
- self._authorize(parsed_url.username + '@' + parsed_url.hostname, self.get_password())
+ self.__authorize(parsed_url.username + '@' + parsed_url.hostname, self.get_password())
# Fetch destination folder entry (and crete hierarchy if required).
folder_names = string.split(parsed_url.path[1:], '/')
parent_folder = None
parent_folder_id = GDocsBackend.ROOT_FOLDER_ID
for folder_name in folder_names:
- entries = self._fetch_entries(parent_folder_id, 'folder', folder_name)
+ entries = self.__fetch_entries(parent_folder_id, 'folder', folder_name)
if entries is not None:
if len(entries) == 1:
parent_folder = entries[0]
@@ -75,54 +77,106 @@
raise BackendException("Error while fetching destination folder '%s'." % folder_name)
self.folder = parent_folder
- def _put(self, source_path, remote_filename):
- self._delete(remote_filename)
-
- # Set uploader instance. Note that resumable uploads are required in order to
- # enable uploads for all file types.
- # (see http://googleappsdeveloper.blogspot.com/2011/05/upload-all-file-types-to-any-google.html)
- file = source_path.open()
- uploader = gdata.client.ResumableUploader(
- self.client, file, GDocsBackend.BACKUP_DOCUMENT_TYPE, os.path.getsize(file.name),
- chunk_size=gdata.client.ResumableUploader.DEFAULT_CHUNK_SIZE,
- desired_class=gdata.docs.data.Resource)
- if uploader:
- # Chunked upload.
- entry = gdata.docs.data.Resource(title=atom.data.Title(text=remote_filename))
- uri = self.folder.get_resumable_create_media_link().href + '?convert=false'
- entry = uploader.UploadFile(uri, entry=entry)
- if not entry:
- raise BackendException("Failed to upload file '%s' to remote folder '%s'"
- % (source_path.get_filename(), self.folder.title.text))
- else:
- raise BackendException("Failed to initialize upload of file '%s' to remote folder '%s'"
- % (source_path.get_filename(), self.folder.title.text))
- assert not file.close()
-
- def _get(self, remote_filename, local_path):
- entries = self._fetch_entries(self.folder.resource_id.text,
- GDocsBackend.BACKUP_DOCUMENT_TYPE,
- remote_filename)
- if len(entries) == 1:
- entry = entries[0]
- self.client.DownloadResource(entry, local_path.name)
- else:
- raise BackendException("Failed to find file '%s' in remote folder '%s'"
- % (remote_filename, self.folder.title.text))
-
- def _list(self):
- entries = self._fetch_entries(self.folder.resource_id.text,
- GDocsBackend.BACKUP_DOCUMENT_TYPE)
- return [entry.title.text for entry in entries]
-
- def _delete(self, filename):
- entries = self._fetch_entries(self.folder.resource_id.text,
- GDocsBackend.BACKUP_DOCUMENT_TYPE,
- filename)
- for entry in entries:
- self.client.delete(entry.get_edit_link().href + '?delete=true', force=True)
-
- def _authorize(self, email, password, captcha_token=None, captcha_response=None):
+ @retry
+ def put(self, source_path, remote_filename=None, raise_errors=False):
+ """Transfer source_path to remote_filename"""
+ # Default remote file name.
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
+ # Upload!
+ try:
+ # If remote file already exists in destination folder, remove it.
+ entries = self.__fetch_entries(self.folder.resource_id.text,
+ GDocsBackend.BACKUP_DOCUMENT_TYPE,
+ remote_filename)
+ for entry in entries:
+ self.client.delete(entry.get_edit_link().href + '?delete=true', force=True)
+
+ # Set uploader instance. Note that resumable uploads are required in order to
+ # enable uploads for all file types.
+ # (see http://googleappsdeveloper.blogspot.com/2011/05/upload-all-file-types-to-any-google.html)
+ file = source_path.open()
+ uploader = gdata.client.ResumableUploader(
+ self.client, file, GDocsBackend.BACKUP_DOCUMENT_TYPE, os.path.getsize(file.name),
+ chunk_size=gdata.client.ResumableUploader.DEFAULT_CHUNK_SIZE,
+ desired_class=gdata.docs.data.Resource)
+ if uploader:
+ # Chunked upload.
+ entry = gdata.docs.data.Resource(title=atom.data.Title(text=remote_filename))
+ uri = self.folder.get_resumable_create_media_link().href + '?convert=false'
+ entry = uploader.UploadFile(uri, entry=entry)
+ if not entry:
+ self.__handle_error("Failed to upload file '%s' to remote folder '%s'"
+ % (source_path.get_filename(), self.folder.title.text), raise_errors)
+ else:
+ self.__handle_error("Failed to initialize upload of file '%s' to remote folder '%s'"
+ % (source_path.get_filename(), self.folder.title.text), raise_errors)
+ assert not file.close()
+ except Exception, e:
+ self.__handle_error("Failed to upload file '%s' to remote folder '%s': %s"
+ % (source_path.get_filename(), self.folder.title.text, str(e)), raise_errors)
+
+ @retry
+ def get(self, remote_filename, local_path, raise_errors=False):
+ """Get remote filename, saving it to local_path"""
+ try:
+ entries = self.__fetch_entries(self.folder.resource_id.text,
+ GDocsBackend.BACKUP_DOCUMENT_TYPE,
+ remote_filename)
+ if len(entries) == 1:
+ entry = entries[0]
+ self.client.DownloadResource(entry, local_path.name)
+ local_path.setdata()
+ return
+ else:
+ self.__handle_error("Failed to find file '%s' in remote folder '%s'"
+ % (remote_filename, self.folder.title.text), raise_errors)
+ except Exception, e:
+ self.__handle_error("Failed to download file '%s' in remote folder '%s': %s"
+ % (remote_filename, self.folder.title.text, str(e)), raise_errors)
+
+ @retry
+ def _list(self, raise_errors=False):
+ """List files in folder"""
+ try:
+ entries = self.__fetch_entries(self.folder.resource_id.text,
+ GDocsBackend.BACKUP_DOCUMENT_TYPE)
+ return [entry.title.text for entry in entries]
+ except Exception, e:
+ self.__handle_error("Failed to fetch list of files in remote folder '%s': %s"
+ % (self.folder.title.text, str(e)), raise_errors)
+
+ @retry
+ def delete(self, filename_list, raise_errors=False):
+ """Delete files in filename_list"""
+ for filename in filename_list:
+ try:
+ entries = self.__fetch_entries(self.folder.resource_id.text,
+ GDocsBackend.BACKUP_DOCUMENT_TYPE,
+ filename)
+ if len(entries) > 0:
+ success = True
+ for entry in entries:
+ if not self.client.delete(entry.get_edit_link().href + '?delete=true', force=True):
+ success = False
+ if not success:
+ self.__handle_error("Failed to remove file '%s' in remote folder '%s'"
+ % (filename, self.folder.title.text), raise_errors)
+ else:
+ log.Warn("Failed to fetch file '%s' in remote folder '%s'"
+ % (filename, self.folder.title.text))
+ except Exception, e:
+ self.__handle_error("Failed to remove file '%s' in remote folder '%s': %s"
+ % (filename, self.folder.title.text, str(e)), raise_errors)
+
+ def __handle_error(self, message, raise_errors=True):
+ if raise_errors:
+ raise BackendException(message)
+ else:
+ log.FatalError(message, log.ErrorCode.backend_error)
+
+ def __authorize(self, email, password, captcha_token=None, captcha_response=None):
try:
self.client.client_login(email,
password,
@@ -130,20 +184,22 @@
service='writely',
captcha_token=captcha_token,
captcha_response=captcha_response)
- except gdata.client.CaptchaChallenge as challenge:
+ except gdata.client.CaptchaChallenge, challenge:
print('A captcha challenge in required. Please visit ' + challenge.captcha_url)
answer = None
while not answer:
answer = raw_input('Answer to the challenge? ')
- self._authorize(email, password, challenge.captcha_token, answer)
+ self.__authorize(email, password, challenge.captcha_token, answer)
except gdata.client.BadAuthentication:
- raise BackendException('Invalid user credentials given. Be aware that accounts '
- 'that use 2-step verification require creating an application specific '
- 'access code for using this Duplicity backend. Follow the instruction in '
- 'http://www.google.com/support/accounts/bin/static.py?page=guide.cs&guide=1056283&topic=1056286 '
- 'and create your application-specific password to run duplicity backups.')
+ self.__handle_error('Invalid user credentials given. Be aware that accounts '
+ 'that use 2-step verification require creating an application specific '
+ 'access code for using this Duplicity backend. Follow the instrucction in '
+ 'http://www.google.com/support/accounts/bin/static.py?page=guide.cs&guide=1056283&topic=1056286 '
+ 'and create your application-specific password to run duplicity backups.')
+ except Exception, e:
+ self.__handle_error('Error while authenticating client: %s.' % str(e))
- def _fetch_entries(self, folder_id, type, title=None):
+ def __fetch_entries(self, folder_id, type, title=None):
# Build URI.
uri = '/feeds/default/private/full/%s/contents' % folder_id
if type == 'folder':
@@ -155,31 +211,34 @@
if title:
uri += '&title=' + urllib.quote(title) + '&title-exact=true'
- # Fetch entries.
- entries = self.client.get_all_resources(uri=uri)
-
- # When filtering by entry title, API is returning (don't know why) documents in other
- # folders (apart from folder_id) matching the title, so some extra filtering is required.
- if title:
- result = []
- for entry in entries:
- resource_type = entry.get_resource_type()
- if (not type) \
- or (type == 'folder' and resource_type == 'folder') \
- or (type == GDocsBackend.BACKUP_DOCUMENT_TYPE and resource_type != 'folder'):
-
- if folder_id != GDocsBackend.ROOT_FOLDER_ID:
- for link in entry.in_collections():
- folder_entry = self.client.get_entry(link.href, None, None,
- desired_class=gdata.docs.data.Resource)
- if folder_entry and (folder_entry.resource_id.text == folder_id):
- result.append(entry)
- elif len(entry.in_collections()) == 0:
- result.append(entry)
- else:
- result = entries
-
- # Done!
- return result
+ try:
+ # Fetch entries.
+ entries = self.client.get_all_resources(uri=uri)
+
+ # When filtering by entry title, API is returning (don't know why) documents in other
+ # folders (apart from folder_id) matching the title, so some extra filtering is required.
+ if title:
+ result = []
+ for entry in entries:
+ resource_type = entry.get_resource_type()
+ if (not type) \
+ or (type == 'folder' and resource_type == 'folder') \
+ or (type == GDocsBackend.BACKUP_DOCUMENT_TYPE and resource_type != 'folder'):
+
+ if folder_id != GDocsBackend.ROOT_FOLDER_ID:
+ for link in entry.in_collections():
+ folder_entry = self.client.get_entry(link.href, None, None,
+ desired_class=gdata.docs.data.Resource)
+ if folder_entry and (folder_entry.resource_id.text == folder_id):
+ result.append(entry)
+ elif len(entry.in_collections()) == 0:
+ result.append(entry)
+ else:
+ result = entries
+
+ # Done!
+ return result
+ except Exception, e:
+ self.__handle_error('Error while fetching remote entries: %s.' % str(e))
duplicity.backend.register_backend('gdocs', GDocsBackend)
=== modified file 'duplicity/backends/giobackend.py'
--- duplicity/backends/giobackend.py 2014-04-29 23:49:01 +0000
+++ duplicity/backends/giobackend.py 2014-10-15 12:12:04 +0000
@@ -19,13 +19,18 @@
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import os
+import types
import subprocess
import atexit
import signal
+from gi.repository import Gio #@UnresolvedImport
+from gi.repository import GLib #@UnresolvedImport
import duplicity.backend
+from duplicity.backend import retry
from duplicity import log
from duplicity import util
+from duplicity.errors import * #@UnusedWildImport
def ensure_dbus():
# GIO requires a dbus session bus which can start the gvfs daemons
@@ -41,39 +46,36 @@
atexit.register(os.kill, int(parts[1]), signal.SIGTERM)
os.environ[parts[0]] = parts[1]
+class DupMountOperation(Gio.MountOperation):
+ """A simple MountOperation that grabs the password from the environment
+ or the user.
+ """
+ def __init__(self, backend):
+ Gio.MountOperation.__init__(self)
+ self.backend = backend
+ self.connect('ask-password', self.ask_password_cb)
+ self.connect('ask-question', self.ask_question_cb)
+
+ def ask_password_cb(self, *args, **kwargs):
+ self.set_password(self.backend.get_password())
+ self.reply(Gio.MountOperationResult.HANDLED)
+
+ def ask_question_cb(self, *args, **kwargs):
+ # Obviously just always answering with the first choice is a naive
+ # approach. But there's no easy way to allow for answering questions
+ # in duplicity's typical run-from-cron mode with environment variables.
+ # And only a couple gvfs backends ask questions: 'sftp' does about
+ # new hosts and 'afc' does if the device is locked. 0 should be a
+ # safe choice.
+ self.set_choice(0)
+ self.reply(Gio.MountOperationResult.HANDLED)
+
class GIOBackend(duplicity.backend.Backend):
"""Use this backend when saving to a GIO URL.
This is a bit of a meta-backend, in that it can handle multiple schemas.
URLs look like schema://user@server/path.
"""
def __init__(self, parsed_url):
- from gi.repository import Gio #@UnresolvedImport
- from gi.repository import GLib #@UnresolvedImport
-
- class DupMountOperation(Gio.MountOperation):
- """A simple MountOperation that grabs the password from the environment
- or the user.
- """
- def __init__(self, backend):
- Gio.MountOperation.__init__(self)
- self.backend = backend
- self.connect('ask-password', self.ask_password_cb)
- self.connect('ask-question', self.ask_question_cb)
-
- def ask_password_cb(self, *args, **kwargs):
- self.set_password(self.backend.get_password())
- self.reply(Gio.MountOperationResult.HANDLED)
-
- def ask_question_cb(self, *args, **kwargs):
- # Obviously just always answering with the first choice is a naive
- # approach. But there's no easy way to allow for answering questions
- # in duplicity's typical run-from-cron mode with environment variables.
- # And only a couple gvfs backends ask questions: 'sftp' does about
- # new hosts and 'afc' does if the device is locked. 0 should be a
- # safe choice.
- self.set_choice(0)
- self.reply(Gio.MountOperationResult.HANDLED)
-
duplicity.backend.Backend.__init__(self, parsed_url)
ensure_dbus()
@@ -84,86 +86,118 @@
op = DupMountOperation(self)
loop = GLib.MainLoop()
self.remote_file.mount_enclosing_volume(Gio.MountMountFlags.NONE,
- op, None,
- self.__done_with_mount, loop)
+ op, None, self.done_with_mount,
+ loop)
loop.run() # halt program until we're done mounting
# Now make the directory if it doesn't exist
try:
self.remote_file.make_directory_with_parents(None)
- except GLib.GError as e:
+ except GLib.GError, e:
if e.code != Gio.IOErrorEnum.EXISTS:
raise
- def __done_with_mount(self, fileobj, result, loop):
- from gi.repository import Gio #@UnresolvedImport
- from gi.repository import GLib #@UnresolvedImport
+ def done_with_mount(self, fileobj, result, loop):
try:
fileobj.mount_enclosing_volume_finish(result)
- except GLib.GError as e:
+ except GLib.GError, e:
# check for NOT_SUPPORTED because some schemas (e.g. file://) validly don't
if e.code != Gio.IOErrorEnum.ALREADY_MOUNTED and e.code != Gio.IOErrorEnum.NOT_SUPPORTED:
log.FatalError(_("Connection failed, please check your password: %s")
% util.uexc(e), log.ErrorCode.connection_failed)
loop.quit()
- def __copy_progress(self, *args, **kwargs):
- pass
-
- def __copy_file(self, source, target):
- from gi.repository import Gio #@UnresolvedImport
- source.copy(target,
- Gio.FileCopyFlags.OVERWRITE | Gio.FileCopyFlags.NOFOLLOW_SYMLINKS,
- None, self.__copy_progress, None)
-
- def _error_code(self, operation, e):
- from gi.repository import Gio #@UnresolvedImport
- from gi.repository import GLib #@UnresolvedImport
+ def handle_error(self, raise_error, e, op, file1=None, file2=None):
+ if raise_error:
+ raise e
+ code = log.ErrorCode.backend_error
if isinstance(e, GLib.GError):
- if e.code == Gio.IOErrorEnum.FAILED and operation == 'delete':
- # Sometimes delete will return a generic failure on a file not
- # found (notably the FTP does that)
- return log.ErrorCode.backend_not_found
- elif e.code == Gio.IOErrorEnum.PERMISSION_DENIED:
- return log.ErrorCode.backend_permission_denied
+ if e.code == Gio.IOErrorEnum.PERMISSION_DENIED:
+ code = log.ErrorCode.backend_permission_denied
elif e.code == Gio.IOErrorEnum.NOT_FOUND:
- return log.ErrorCode.backend_not_found
+ code = log.ErrorCode.backend_not_found
elif e.code == Gio.IOErrorEnum.NO_SPACE:
- return log.ErrorCode.backend_no_space
-
- def _put(self, source_path, remote_filename):
- from gi.repository import Gio #@UnresolvedImport
+ code = log.ErrorCode.backend_no_space
+ extra = ' '.join([util.escape(x) for x in [file1, file2] if x])
+ extra = ' '.join([op, extra])
+ log.FatalError(util.uexc(e), code, extra)
+
+ def copy_progress(self, *args, **kwargs):
+ pass
+
+ @retry
+ def copy_file(self, op, source, target, raise_errors=False):
+ log.Info(_("Writing %s") % util.ufn(target.get_parse_name()))
+ try:
+ source.copy(target,
+ Gio.FileCopyFlags.OVERWRITE | Gio.FileCopyFlags.NOFOLLOW_SYMLINKS,
+ None, self.copy_progress, None)
+ except Exception, e:
+ self.handle_error(raise_errors, e, op, source.get_parse_name(),
+ target.get_parse_name())
+
+ def put(self, source_path, remote_filename = None):
+ """Copy file to remote"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
source_file = Gio.File.new_for_path(source_path.name)
target_file = self.remote_file.get_child(remote_filename)
- self.__copy_file(source_file, target_file)
+ self.copy_file('put', source_file, target_file)
- def _get(self, filename, local_path):
- from gi.repository import Gio #@UnresolvedImport
+ def get(self, filename, local_path):
+ """Get file and put in local_path (Path object)"""
source_file = self.remote_file.get_child(filename)
target_file = Gio.File.new_for_path(local_path.name)
- self.__copy_file(source_file, target_file)
+ self.copy_file('get', source_file, target_file)
+ local_path.setdata()
- def _list(self):
- from gi.repository import Gio #@UnresolvedImport
+ @retry
+ def _list(self, raise_errors=False):
+ """List files in that directory"""
files = []
- enum = self.remote_file.enumerate_children(Gio.FILE_ATTRIBUTE_STANDARD_NAME,
- Gio.FileQueryInfoFlags.NOFOLLOW_SYMLINKS,
- None)
- info = enum.next_file(None)
- while info:
- files.append(info.get_name())
+ try:
+ enum = self.remote_file.enumerate_children(Gio.FILE_ATTRIBUTE_STANDARD_NAME,
+ Gio.FileQueryInfoFlags.NOFOLLOW_SYMLINKS,
+ None)
info = enum.next_file(None)
+ while info:
+ files.append(info.get_name())
+ info = enum.next_file(None)
+ except Exception, e:
+ self.handle_error(raise_errors, e, 'list',
+ self.remote_file.get_parse_name())
return files
- def _delete(self, filename):
- target_file = self.remote_file.get_child(filename)
- target_file.delete(None)
-
- def _query(self, filename):
- from gi.repository import Gio #@UnresolvedImport
- target_file = self.remote_file.get_child(filename)
- info = target_file.query_info(Gio.FILE_ATTRIBUTE_STANDARD_SIZE,
- Gio.FileQueryInfoFlags.NONE, None)
- return {'size': info.get_size()}
-
-duplicity.backend.register_backend_prefix('gio', GIOBackend)
+ @retry
+ def delete(self, filename_list, raise_errors=False):
+ """Delete all files in filename list"""
+ assert type(filename_list) is not types.StringType
+ for filename in filename_list:
+ target_file = self.remote_file.get_child(filename)
+ try:
+ target_file.delete(None)
+ except Exception, e:
+ if isinstance(e, GLib.GError):
+ if e.code == Gio.IOErrorEnum.NOT_FOUND:
+ continue
+ self.handle_error(raise_errors, e, 'delete',
+ target_file.get_parse_name())
+ return
+
+ @retry
+ def _query_file_info(self, filename, raise_errors=False):
+ """Query attributes on filename"""
+ target_file = self.remote_file.get_child(filename)
+ attrs = Gio.FILE_ATTRIBUTE_STANDARD_SIZE
+ try:
+ info = target_file.query_info(attrs, Gio.FileQueryInfoFlags.NONE,
+ None)
+ return {'size': info.get_size()}
+ except Exception, e:
+ if isinstance(e, GLib.GError):
+ if e.code == Gio.IOErrorEnum.NOT_FOUND:
+ return {'size': -1} # early exit, no need to retry
+ if raise_errors:
+ raise e
+ else:
+ return {'size': None}
=== modified file 'duplicity/backends/hsibackend.py'
--- duplicity/backends/hsibackend.py 2014-04-26 12:54:37 +0000
+++ duplicity/backends/hsibackend.py 2014-10-15 12:12:04 +0000
@@ -20,7 +20,9 @@
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import os
+
import duplicity.backend
+from duplicity.errors import * #@UnusedWildImport
hsi_command = "hsi"
class HSIBackend(duplicity.backend.Backend):
@@ -33,23 +35,36 @@
else:
self.remote_prefix = ""
- def _put(self, source_path, remote_filename):
+ def put(self, source_path, remote_filename = None):
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
commandline = '%s "put %s : %s%s"' % (hsi_command,source_path.name,self.remote_prefix,remote_filename)
- self.subprocess_popen(commandline)
+ try:
+ self.run_command(commandline)
+ except Exception:
+ print commandline
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
commandline = '%s "get %s : %s%s"' % (hsi_command, local_path.name, self.remote_prefix, remote_filename)
- self.subprocess_popen(commandline)
+ self.run_command(commandline)
+ local_path.setdata()
+ if not local_path.exists():
+ raise BackendException("File %s not found" % local_path.name)
- def _list(self):
+ def list(self):
commandline = '%s "ls -l %s"' % (hsi_command, self.remote_dir)
l = os.popen3(commandline)[2].readlines()[3:]
for i in range(0,len(l)):
l[i] = l[i].split()[-1]
- return [x for x in l if x]
+ print filter(lambda x: x, l)
+ return filter(lambda x: x, l)
- def _delete(self, filename):
- commandline = '%s "rm %s%s"' % (hsi_command, self.remote_prefix, filename)
- self.subprocess_popen(commandline)
+ def delete(self, filename_list):
+ assert len(filename_list) > 0
+ for fn in filename_list:
+ commandline = '%s "rm %s%s"' % (hsi_command, self.remote_prefix, fn)
+ self.run_command(commandline)
duplicity.backend.register_backend("hsi", HSIBackend)
+
+
=== modified file 'duplicity/backends/imapbackend.py'
--- duplicity/backends/imapbackend.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/imapbackend.py 2014-10-15 12:12:04 +0000
@@ -44,7 +44,7 @@
(self.__class__.__name__, parsed_url.scheme, parsed_url.hostname, parsed_url.username))
# Store url for reconnection on error
- self.url = parsed_url
+ self._url = parsed_url
# Set the username
if ( parsed_url.username is None ):
@@ -54,19 +54,19 @@
# Set the password
if ( not parsed_url.password ):
- if 'IMAP_PASSWORD' in os.environ:
+ if os.environ.has_key('IMAP_PASSWORD'):
password = os.environ.get('IMAP_PASSWORD')
else:
password = getpass.getpass("Enter account password: ")
else:
password = parsed_url.password
- self.username = username
- self.password = password
- self.resetConnection()
+ self._username = username
+ self._password = password
+ self._resetConnection()
- def resetConnection(self):
- parsed_url = self.url
+ def _resetConnection(self):
+ parsed_url = self._url
try:
imap_server = os.environ['IMAP_SERVER']
except KeyError:
@@ -74,32 +74,32 @@
# Try to close the connection cleanly
try:
- self.conn.close()
+ self._conn.close()
except Exception:
pass
if (parsed_url.scheme == "imap"):
cl = imaplib.IMAP4
- self.conn = cl(imap_server, 143)
+ self._conn = cl(imap_server, 143)
elif (parsed_url.scheme == "imaps"):
cl = imaplib.IMAP4_SSL
- self.conn = cl(imap_server, 993)
+ self._conn = cl(imap_server, 993)
log.Debug("Type of imap class: %s" % (cl.__name__))
self.remote_dir = re.sub(r'^/', r'', parsed_url.path, 1)
# Login
if (not(globals.imap_full_address)):
- self.conn.login(self.username, self.password)
- self.conn.select(globals.imap_mailbox)
+ self._conn.login(self._username, self._password)
+ self._conn.select(globals.imap_mailbox)
log.Info("IMAP connected")
else:
- self.conn.login(self.username + "@" + parsed_url.hostname, self.password)
- self.conn.select(globals.imap_mailbox)
+ self._conn.login(self._username + "@" + parsed_url.hostname, self._password)
+ self._conn.select(globals.imap_mailbox)
log.Info("IMAP connected")
- def prepareBody(self,f,rname):
+ def _prepareBody(self,f,rname):
mp = email.MIMEMultipart.MIMEMultipart()
# I am going to use the remote_dir as the From address so that
@@ -117,7 +117,9 @@
return mp.as_string()
- def _put(self, source_path, remote_filename):
+ def put(self, source_path, remote_filename = None):
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
f=source_path.open("rb")
allowedTimeout = globals.timeout
if (allowedTimeout == 0):
@@ -125,12 +127,12 @@
allowedTimeout = 2880
while allowedTimeout > 0:
try:
- self.conn.select(remote_filename)
- body=self.prepareBody(f,remote_filename)
+ self._conn.select(remote_filename)
+ body=self._prepareBody(f,remote_filename)
# If we don't select the IMAP folder before
# append, the message goes into the INBOX.
- self.conn.select(globals.imap_mailbox)
- self.conn.append(globals.imap_mailbox, None, None, body)
+ self._conn.select(globals.imap_mailbox)
+ self._conn.append(globals.imap_mailbox, None, None, body)
break
except (imaplib.IMAP4.abort, socket.error, socket.sslerror):
allowedTimeout -= 1
@@ -138,7 +140,7 @@
time.sleep(30)
while allowedTimeout > 0:
try:
- self.resetConnection()
+ self._resetConnection()
break
except (imaplib.IMAP4.abort, socket.error, socket.sslerror):
allowedTimeout -= 1
@@ -147,15 +149,15 @@
log.Info("IMAP mail with '%s' subject stored" % remote_filename)
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
allowedTimeout = globals.timeout
if (allowedTimeout == 0):
# Allow a total timeout of 1 day
allowedTimeout = 2880
while allowedTimeout > 0:
try:
- self.conn.select(globals.imap_mailbox)
- (result,list) = self.conn.search(None, 'Subject', remote_filename)
+ self._conn.select(globals.imap_mailbox)
+ (result,list) = self._conn.search(None, 'Subject', remote_filename)
if result != "OK":
raise Exception(list[0])
@@ -163,7 +165,7 @@
if list[0] == '':
raise Exception("no mail with subject %s")
- (result,list) = self.conn.fetch(list[0],"(RFC822)")
+ (result,list) = self._conn.fetch(list[0],"(RFC822)")
if result != "OK":
raise Exception(list[0])
@@ -183,7 +185,7 @@
time.sleep(30)
while allowedTimeout > 0:
try:
- self.resetConnection()
+ self._resetConnection()
break
except (imaplib.IMAP4.abort, socket.error, socket.sslerror):
allowedTimeout -= 1
@@ -197,7 +199,7 @@
def _list(self):
ret = []
- (result,list) = self.conn.select(globals.imap_mailbox)
+ (result,list) = self._conn.select(globals.imap_mailbox)
if result != "OK":
raise BackendException(list[0])
@@ -205,14 +207,14 @@
# address
# Search returns an error if you haven't selected an IMAP folder.
- (result,list) = self.conn.search(None, 'FROM', self.remote_dir)
+ (result,list) = self._conn.search(None, 'FROM', self.remote_dir)
if result!="OK":
raise Exception(list[0])
if list[0]=='':
return ret
nums=list[0].split(" ")
set="%s:%s"%(nums[0],nums[-1])
- (result,list) = self.conn.fetch(set,"(BODY[HEADER])")
+ (result,list) = self._conn.fetch(set,"(BODY[HEADER])")
if result!="OK":
raise Exception(list[0])
@@ -230,32 +232,34 @@
log.Info("IMAP LIST: %s %s" % (subj,header_from))
return ret
- def imapf(self,fun,*args):
+ def _imapf(self,fun,*args):
(ret,list)=fun(*args)
if ret != "OK":
raise Exception(list[0])
return list
- def delete_single_mail(self,i):
- self.imapf(self.conn.store,i,"+FLAGS",'\\DELETED')
-
- def expunge(self):
- list=self.imapf(self.conn.expunge)
-
- def _delete_list(self, filename_list):
+ def _delete_single_mail(self,i):
+ self._imapf(self._conn.store,i,"+FLAGS",'\\DELETED')
+
+ def _expunge(self):
+ list=self._imapf(self._conn.expunge)
+
+ def delete(self, filename_list):
+ assert len(filename_list) > 0
for filename in filename_list:
- list = self.imapf(self.conn.search,None,"(SUBJECT %s)"%filename)
+ list = self._imapf(self._conn.search,None,"(SUBJECT %s)"%filename)
list = list[0].split()
- if len(list) > 0 and list[0] != "":
- self.delete_single_mail(list[0])
- log.Notice("marked %s to be deleted" % filename)
- self.expunge()
- log.Notice("IMAP expunged %s files" % len(filename_list))
+ if len(list)==0 or list[0]=="":raise Exception("no such mail with subject '%s'"%filename)
+ self._delete_single_mail(list[0])
+ log.Notice("marked %s to be deleted" % filename)
+ self._expunge()
+ log.Notice("IMAP expunged %s files" % len(list))
- def _close(self):
- self.conn.select(globals.imap_mailbox)
- self.conn.close()
- self.conn.logout()
+ def close(self):
+ self._conn.select(globals.imap_mailbox)
+ self._conn.close()
+ self._conn.logout()
duplicity.backend.register_backend("imap", ImapBackend);
duplicity.backend.register_backend("imaps", ImapBackend);
+
=== modified file 'duplicity/backends/localbackend.py'
--- duplicity/backends/localbackend.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/localbackend.py 2014-10-15 12:12:04 +0000
@@ -20,11 +20,14 @@
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import os
+import types
+import errno
import duplicity.backend
from duplicity import log
from duplicity import path
-from duplicity.errors import BackendException
+from duplicity import util
+from duplicity.errors import * #@UnusedWildImport
class LocalBackend(duplicity.backend.Backend):
@@ -40,37 +43,90 @@
if not parsed_url.path.startswith('//'):
raise BackendException("Bad file:// path syntax.")
self.remote_pathdir = path.Path(parsed_url.path[2:])
- try:
- os.makedirs(self.remote_pathdir.base)
- except Exception:
- pass
-
- def _move(self, source_path, remote_filename):
- target_path = self.remote_pathdir.append(remote_filename)
- try:
- source_path.rename(target_path)
- return True
- except OSError:
- return False
-
- def _put(self, source_path, remote_filename):
- target_path = self.remote_pathdir.append(remote_filename)
- target_path.writefileobj(source_path.open("rb"))
-
- def _get(self, filename, local_path):
+
+ def handle_error(self, e, op, file1 = None, file2 = None):
+ code = log.ErrorCode.backend_error
+ if hasattr(e, 'errno'):
+ if e.errno == errno.EACCES:
+ code = log.ErrorCode.backend_permission_denied
+ elif e.errno == errno.ENOENT:
+ code = log.ErrorCode.backend_not_found
+ elif e.errno == errno.ENOSPC:
+ code = log.ErrorCode.backend_no_space
+ extra = ' '.join([util.escape(x) for x in [file1, file2] if x])
+ extra = ' '.join([op, extra])
+ if op != 'delete' and op != 'query':
+ log.FatalError(util.uexc(e), code, extra)
+ else:
+ log.Warn(util.uexc(e), code, extra)
+
+ def move(self, source_path, remote_filename = None):
+ self.put(source_path, remote_filename, rename_instead = True)
+
+ def put(self, source_path, remote_filename = None, rename_instead = False):
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+ target_path = self.remote_pathdir.append(remote_filename)
+ log.Info("Writing %s" % target_path.name)
+ """Try renaming first (if allowed to), copying if doesn't work"""
+ if rename_instead:
+ try:
+ source_path.rename(target_path)
+ except OSError:
+ pass
+ except Exception, e:
+ self.handle_error(e, 'put', source_path.name, target_path.name)
+ else:
+ return
+ try:
+ target_path.writefileobj(source_path.open("rb"))
+ except Exception, e:
+ self.handle_error(e, 'put', source_path.name, target_path.name)
+
+ """If we get here, renaming failed previously"""
+ if rename_instead:
+ """We need to simulate its behaviour"""
+ source_path.delete()
+
+ def get(self, filename, local_path):
+ """Get file and put in local_path (Path object)"""
source_path = self.remote_pathdir.append(filename)
- local_path.writefileobj(source_path.open("rb"))
+ try:
+ local_path.writefileobj(source_path.open("rb"))
+ except Exception, e:
+ self.handle_error(e, 'get', source_path.name, local_path.name)
def _list(self):
- return self.remote_pathdir.listdir()
-
- def _delete(self, filename):
- self.remote_pathdir.append(filename).delete()
-
- def _query(self, filename):
- target_file = self.remote_pathdir.append(filename)
- target_file.setdata()
- size = target_file.getsize() if target_file.exists() else -1
- return {'size': size}
+ """List files in that directory"""
+ try:
+ os.makedirs(self.remote_pathdir.base)
+ except Exception:
+ pass
+ try:
+ return self.remote_pathdir.listdir()
+ except Exception, e:
+ self.handle_error(e, 'list', self.remote_pathdir.name)
+
+ def delete(self, filename_list):
+ """Delete all files in filename list"""
+ assert type(filename_list) is not types.StringType
+ for filename in filename_list:
+ try:
+ self.remote_pathdir.append(filename).delete()
+ except Exception, e:
+ self.handle_error(e, 'delete', self.remote_pathdir.append(filename).name)
+
+ def _query_file_info(self, filename):
+ """Query attributes on filename"""
+ try:
+ target_file = self.remote_pathdir.append(filename)
+ if not os.path.exists(target_file.name):
+ return {'size': -1}
+ target_file.setdata()
+ size = target_file.getsize()
+ return {'size': size}
+ except Exception, e:
+ self.handle_error(e, 'query', target_file.name)
+ return {'size': None}
duplicity.backend.register_backend("file", LocalBackend)
=== modified file 'duplicity/backends/megabackend.py'
--- duplicity/backends/megabackend.py 2014-04-26 12:35:04 +0000
+++ duplicity/backends/megabackend.py 2014-10-15 12:12:04 +0000
@@ -22,8 +22,9 @@
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import duplicity.backend
+from duplicity.backend import retry
from duplicity import log
-from duplicity.errors import BackendException
+from duplicity.errors import * #@UnusedWildImport
class MegaBackend(duplicity.backend.Backend):
@@ -62,64 +63,113 @@
self.folder = parent_folder
- def _put(self, source_path, remote_filename):
- try:
- self._delete(remote_filename)
- except Exception:
- pass
- self.client.upload(source_path.get_canonical(), self.folder, dest_filename=remote_filename)
-
- def _get(self, remote_filename, local_path):
- files = self.client.get_files()
- entries = self.__filter_entries(files, self.folder, remote_filename, 'file')
- if len(entries):
- # get first matching remote file
- entry = entries.keys()[0]
- self.client.download((entry, entries[entry]), dest_filename=local_path.name)
- else:
- raise BackendException("Failed to find file '%s' in remote folder '%s'"
- % (remote_filename, self.__get_node_name(self.folder)),
- code=log.ErrorCode.backend_not_found)
-
- def _list(self):
- entries = self.client.get_files_in_node(self.folder)
- return [self.client.get_name_from_file({entry:entries[entry]}) for entry in entries]
-
- def _delete(self, filename):
- files = self.client.get_files()
- entries = self.__filter_entries(files, self.folder, filename, 'file')
- if len(entries):
- self.client.destroy(entries.keys()[0])
- else:
- raise BackendException("Failed to find file '%s' in remote folder '%s'"
- % (filename, self.__get_node_name(self.folder)),
- code=log.ErrorCode.backend_not_found)
+ @retry
+ def put(self, source_path, remote_filename=None, raise_errors=False):
+ """Transfer source_path to remote_filename"""
+ # Default remote file name.
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
+ try:
+ # If remote file already exists in destination folder, remove it.
+ files = self.client.get_files()
+ entries = self.__filter_entries(files, self.folder, remote_filename, 'file')
+
+ for entry in entries:
+ self.client.delete(entry)
+
+ self.client.upload(source_path.get_canonical(), self.folder, dest_filename=remote_filename)
+
+ except Exception, e:
+ self.__handle_error("Failed to upload file '%s' to remote folder '%s': %s"
+ % (source_path.get_canonical(), self.__get_node_name(self.folder), str(e)), raise_errors)
+
+ @retry
+ def get(self, remote_filename, local_path, raise_errors=False):
+ """Get remote filename, saving it to local_path"""
+ try:
+ files = self.client.get_files()
+ entries = self.__filter_entries(files, self.folder, remote_filename, 'file')
+
+ if len(entries):
+ # get first matching remote file
+ entry = entries.keys()[0]
+ self.client.download((entry, entries[entry]), dest_filename=local_path.name)
+ local_path.setdata()
+ return
+ else:
+ self.__handle_error("Failed to find file '%s' in remote folder '%s'"
+ % (remote_filename, self.__get_node_name(self.folder)), raise_errors)
+ except Exception, e:
+ self.__handle_error("Failed to download file '%s' in remote folder '%s': %s"
+ % (remote_filename, self.__get_node_name(self.folder), str(e)), raise_errors)
+
+ @retry
+ def _list(self, raise_errors=False):
+ """List files in folder"""
+ try:
+ entries = self.client.get_files_in_node(self.folder)
+ return [ self.client.get_name_from_file({entry:entries[entry]}) for entry in entries]
+ except Exception, e:
+ self.__handle_error("Failed to fetch list of files in remote folder '%s': %s"
+ % (self.__get_node_name(self.folder), str(e)), raise_errors)
+
+ @retry
+ def delete(self, filename_list, raise_errors=False):
+ """Delete files in filename_list"""
+ files = self.client.get_files()
+ for filename in filename_list:
+ entries = self.__filter_entries(files, self.folder, filename)
+ try:
+ if len(entries) > 0:
+ for entry in entries:
+ if self.client.destroy(entry):
+ self.__handle_error("Failed to remove file '%s' in remote folder '%s'"
+ % (filename, self.__get_node_name(self.folder)), raise_errors)
+ else:
+ log.Warn("Failed to fetch file '%s' in remote folder '%s'"
+ % (filename, self.__get_node_name(self.folder)))
+ except Exception, e:
+ self.__handle_error("Failed to remove file '%s' in remote folder '%s': %s"
+ % (filename, self.__get_node_name(self.folder), str(e)), raise_errors)
def __get_node_name(self, handle):
"""get node name from public handle"""
files = self.client.get_files()
return self.client.get_name_from_file({handle:files[handle]})
+
+ def __handle_error(self, message, raise_errors=True):
+ if raise_errors:
+ raise BackendException(message)
+ else:
+ log.FatalError(message, log.ErrorCode.backend_error)
def __authorize(self, email, password):
- self.client.login(email, password)
+ try:
+ self.client.login(email, password)
+ except Exception, e:
+ self.__handle_error('Error while authenticating client: %s.' % str(e))
def __filter_entries(self, entries, parent_id=None, title=None, type=None):
result = {}
type_map = { 'folder': 1, 'file': 0 }
- for k, v in entries.items():
- try:
- if parent_id != None:
- assert(v['p'] == parent_id)
- if title != None:
- assert(v['a']['n'] == title)
- if type != None:
- assert(v['t'] == type_map[type])
- except AssertionError:
- continue
-
- result.update({k:v})
-
- return result
+ try:
+ for k, v in entries.items():
+ try:
+ if parent_id != None:
+ assert(v['p'] == parent_id)
+ if title != None:
+ assert(v['a']['n'] == title)
+ if type != None:
+ assert(v['t'] == type_map[type])
+ except AssertionError:
+ continue
+
+ result.update({k:v})
+
+ return result
+ except Exception, e:
+ self.__handle_error('Error while fetching remote entries: %s.' % str(e))
duplicity.backend.register_backend('mega', MegaBackend)
=== modified file 'duplicity/backends/rsyncbackend.py'
--- duplicity/backends/rsyncbackend.py 2014-04-26 12:54:37 +0000
+++ duplicity/backends/rsyncbackend.py 2014-10-15 12:12:04 +0000
@@ -23,7 +23,7 @@
import tempfile
import duplicity.backend
-from duplicity.errors import InvalidBackendURL
+from duplicity.errors import * #@UnusedWildImport
from duplicity import globals, tempdir, util
class RsyncBackend(duplicity.backend.Backend):
@@ -58,13 +58,12 @@
if port:
port = " --port=%s" % port
else:
- host_string = host + ":" if host else ""
if parsed_url.path.startswith("//"):
# its an absolute path
- self.url_string = "%s/%s" % (host_string, parsed_url.path.lstrip('/'))
+ self.url_string = "%s:/%s" % (host, parsed_url.path.lstrip('/'))
else:
# its a relative path
- self.url_string = "%s%s" % (host_string, parsed_url.path.lstrip('/'))
+ self.url_string = "%s:%s" % (host, parsed_url.path.lstrip('/'))
if parsed_url.port:
port = " -p %s" % parsed_url.port
# add trailing slash if missing
@@ -106,17 +105,29 @@
raise InvalidBackendURL("Could not determine rsync path: %s"
"" % self.munge_password( url ) )
- def _put(self, source_path, remote_filename):
+ def run_command(self, commandline):
+ result, stdout, stderr = self.subprocess_popen_persist(commandline)
+ return result, stdout
+
+ def put(self, source_path, remote_filename = None):
+ """Use rsync to copy source_dir/filename to remote computer"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
remote_path = os.path.join(self.url_string, remote_filename)
commandline = "%s %s %s" % (self.cmd, source_path.name, remote_path)
- self.subprocess_popen(commandline)
+ self.run_command(commandline)
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
+ """Use rsync to get a remote file"""
remote_path = os.path.join (self.url_string, remote_filename)
commandline = "%s %s %s" % (self.cmd, remote_path, local_path.name)
- self.subprocess_popen(commandline)
+ self.run_command(commandline)
+ local_path.setdata()
+ if not local_path.exists():
+ raise BackendException("File %s not found" % local_path.name)
- def _list(self):
+ def list(self):
+ """List files"""
def split (str):
line = str.split ()
if len (line) > 4 and line[4] != '.':
@@ -124,17 +135,20 @@
else:
return None
commandline = "%s %s" % (self.cmd, self.url_string)
- result, stdout, stderr = self.subprocess_popen(commandline)
- return [x for x in map (split, stdout.split('\n')) if x]
+ result, stdout = self.run_command(commandline)
+ return filter(lambda x: x, map (split, stdout.split('\n')))
- def _delete_list(self, filename_list):
+ def delete(self, filename_list):
+ """Delete files."""
delete_list = filename_list
dont_delete_list = []
- for file in self._list ():
+ for file in self.list ():
if file in delete_list:
delete_list.remove (file)
else:
dont_delete_list.append (file)
+ if len (delete_list) > 0:
+ raise BackendException("Files %s not found" % str (delete_list))
dir = tempfile.mkdtemp()
exclude, exclude_name = tempdir.default().mkstemp_file()
@@ -148,7 +162,7 @@
exclude.close()
commandline = ("%s --recursive --delete --exclude-from=%s %s/ %s" %
(self.cmd, exclude_name, dir, self.url_string))
- self.subprocess_popen(commandline)
+ self.run_command(commandline)
for file in to_delete:
util.ignore_missing(os.unlink, file)
os.rmdir (dir)
=== modified file 'duplicity/backends/sshbackend.py'
--- duplicity/backends/sshbackend.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/sshbackend.py 2014-10-15 12:12:04 +0000
@@ -18,7 +18,6 @@
# along with duplicity; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-import duplicity.backend
from duplicity import globals, log
def warn_option(option, optionvar):
@@ -27,15 +26,11 @@
if (globals.ssh_backend and
globals.ssh_backend.lower().strip() == 'pexpect'):
- from ._ssh_pexpect import SSHPExpectBackend as SSHBackend
+ import _ssh_pexpect
else:
# take user by the hand to prevent typo driven bug reports
if globals.ssh_backend.lower().strip() != 'paramiko':
log.Warn(_("Warning: Selected ssh backend '%s' is neither 'paramiko nor 'pexpect'. Will use default paramiko instead.") % globals.ssh_backend)
warn_option("--scp-command", globals.scp_command)
warn_option("--sftp-command", globals.sftp_command)
- from ._ssh_paramiko import SSHParamikoBackend as SSHBackend
-
-duplicity.backend.register_backend("sftp", SSHBackend)
-duplicity.backend.register_backend("scp", SSHBackend)
-duplicity.backend.register_backend("ssh", SSHBackend)
+ import _ssh_paramiko
=== modified file 'duplicity/backends/swiftbackend.py'
--- duplicity/backends/swiftbackend.py 2014-04-29 23:49:01 +0000
+++ duplicity/backends/swiftbackend.py 2014-10-15 12:12:04 +0000
@@ -19,12 +19,14 @@
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import os
+import time
import duplicity.backend
+from duplicity import globals
from duplicity import log
-from duplicity import util
-from duplicity.errors import BackendException
-
+from duplicity.errors import * #@UnusedWildImport
+from duplicity.util import exception_traceback
+from duplicity.backend import retry
class SwiftBackend(duplicity.backend.Backend):
"""
@@ -42,20 +44,20 @@
conn_kwargs = {}
# if the user has already authenticated
- if 'SWIFT_PREAUTHURL' in os.environ and 'SWIFT_PREAUTHTOKEN' in os.environ:
+ if os.environ.has_key('SWIFT_PREAUTHURL') and os.environ.has_key('SWIFT_PREAUTHTOKEN'):
conn_kwargs['preauthurl'] = os.environ['SWIFT_PREAUTHURL']
conn_kwargs['preauthtoken'] = os.environ['SWIFT_PREAUTHTOKEN']
else:
- if 'SWIFT_USERNAME' not in os.environ:
+ if not os.environ.has_key('SWIFT_USERNAME'):
raise BackendException('SWIFT_USERNAME environment variable '
'not set.')
- if 'SWIFT_PASSWORD' not in os.environ:
+ if not os.environ.has_key('SWIFT_PASSWORD'):
raise BackendException('SWIFT_PASSWORD environment variable '
'not set.')
- if 'SWIFT_AUTHURL' not in os.environ:
+ if not os.environ.has_key('SWIFT_AUTHURL'):
raise BackendException('SWIFT_AUTHURL environment variable '
'not set.')
@@ -63,11 +65,11 @@
conn_kwargs['key'] = os.environ['SWIFT_PASSWORD']
conn_kwargs['authurl'] = os.environ['SWIFT_AUTHURL']
- if 'SWIFT_AUTHVERSION' in os.environ:
+ if os.environ.has_key('SWIFT_AUTHVERSION'):
conn_kwargs['auth_version'] = os.environ['SWIFT_AUTHVERSION']
else:
conn_kwargs['auth_version'] = '1'
- if 'SWIFT_TENANTNAME' in os.environ:
+ if os.environ.has_key('SWIFT_TENANTNAME'):
conn_kwargs['tenant_name'] = os.environ['SWIFT_TENANTNAME']
self.container = parsed_url.path.lstrip('/')
@@ -75,35 +77,126 @@
try:
self.conn = Connection(**conn_kwargs)
self.conn.put_container(self.container)
- except Exception as e:
+ except Exception, e:
log.FatalError("Connection failed: %s %s"
- % (e.__class__.__name__, util.uexc(e)),
+ % (e.__class__.__name__, str(e)),
log.ErrorCode.connection_failed)
- def _error_code(self, operation, e):
- if isinstance(e, self.resp_exc):
- if e.http_status == 404:
- return log.ErrorCode.backend_not_found
-
- def _put(self, source_path, remote_filename):
- self.conn.put_object(self.container, remote_filename,
- file(source_path.name))
-
- def _get(self, remote_filename, local_path):
- headers, body = self.conn.get_object(self.container, remote_filename)
- with open(local_path.name, 'wb') as f:
- for chunk in body:
- f.write(chunk)
+ def put(self, source_path, remote_filename = None):
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
+ for n in range(1, globals.num_retries+1):
+ log.Info("Uploading '%s/%s' " % (self.container, remote_filename))
+ try:
+ self.conn.put_object(self.container,
+ remote_filename,
+ file(source_path.name))
+ return
+ except self.resp_exc, error:
+ log.Warn("Upload of '%s' failed (attempt %d): Swift server returned: %s %s"
+ % (remote_filename, n, error.http_status, error.message))
+ except Exception, e:
+ log.Warn("Upload of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up uploading '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error uploading '%s'" % remote_filename)
+
+ def get(self, remote_filename, local_path):
+ for n in range(1, globals.num_retries+1):
+ log.Info("Downloading '%s/%s'" % (self.container, remote_filename))
+ try:
+ headers, body = self.conn.get_object(self.container,
+ remote_filename)
+ f = open(local_path.name, 'w')
+ for chunk in body:
+ f.write(chunk)
+ local_path.setdata()
+ return
+ except self.resp_exc, resperr:
+ log.Warn("Download of '%s' failed (attempt %s): Swift server returned: %s %s"
+ % (remote_filename, n, resperr.http_status, resperr.message))
+ except Exception, e:
+ log.Warn("Download of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up downloading '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error downloading '%s/%s'"
+ % (self.container, remote_filename))
def _list(self):
- headers, objs = self.conn.get_container(self.container)
- return [ o['name'] for o in objs ]
-
- def _delete(self, filename):
- self.conn.delete_object(self.container, filename)
-
- def _query(self, filename):
- sobject = self.conn.head_object(self.container, filename)
- return {'size': int(sobject['content-length'])}
+ for n in range(1, globals.num_retries+1):
+ log.Info("Listing '%s'" % (self.container))
+ try:
+ # Cloud Files will return a max of 10,000 objects. We have
+ # to make multiple requests to get them all.
+ headers, objs = self.conn.get_container(self.container)
+ return [ o['name'] for o in objs ]
+ except self.resp_exc, resperr:
+ log.Warn("Listing of '%s' failed (attempt %s): Swift server returned: %s %s"
+ % (self.container, n, resperr.http_status, resperr.message))
+ except Exception, e:
+ log.Warn("Listing of '%s' failed (attempt %s): %s: %s"
+ % (self.container, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up listing of '%s' after %s attempts"
+ % (self.container, globals.num_retries))
+ raise BackendException("Error listing '%s'"
+ % (self.container))
+
+ def delete_one(self, remote_filename):
+ for n in range(1, globals.num_retries+1):
+ log.Info("Deleting '%s/%s'" % (self.container, remote_filename))
+ try:
+ self.conn.delete_object(self.container, remote_filename)
+ return
+ except self.resp_exc, resperr:
+ if n > 1 and resperr.http_status == 404:
+ # We failed on a timeout, but delete succeeded on the server
+ log.Warn("Delete of '%s' missing after retry - must have succeded earlier" % remote_filename )
+ return
+ log.Warn("Delete of '%s' failed (attempt %s): Swift server returned: %s %s"
+ % (remote_filename, n, resperr.http_status, resperr.message))
+ except Exception, e:
+ log.Warn("Delete of '%s' failed (attempt %s): %s: %s"
+ % (remote_filename, n, e.__class__.__name__, str(e)))
+ log.Debug("Backtrace of previous error: %s"
+ % exception_traceback())
+ time.sleep(30)
+ log.Warn("Giving up deleting '%s' after %s attempts"
+ % (remote_filename, globals.num_retries))
+ raise BackendException("Error deleting '%s/%s'"
+ % (self.container, remote_filename))
+
+ def delete(self, filename_list):
+ for file in filename_list:
+ self.delete_one(file)
+ log.Debug("Deleted '%s/%s'" % (self.container, file))
+
+ @retry
+ def _query_file_info(self, filename, raise_errors=False):
+ try:
+ sobject = self.conn.head_object(self.container, filename)
+ return {'size': long(sobject['content-length'])}
+ except self.resp_exc:
+ return {'size': -1}
+ except Exception, e:
+ log.Warn("Error querying '%s/%s': %s"
+ "" % (self.container,
+ filename,
+ str(e)))
+ if raise_errors:
+ raise e
+ else:
+ return {'size': None}
duplicity.backend.register_backend("swift", SwiftBackend)
=== removed file 'duplicity/backends/sxbackend.py'
--- duplicity/backends/sxbackend.py 2014-08-17 15:57:59 +0000
+++ duplicity/backends/sxbackend.py 1970-01-01 00:00:00 +0000
@@ -1,51 +0,0 @@
-# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
-#
-# Copyright 2014 Andrea Grandi <a.grandi@xxxxxxxxx>
-#
-# This file is part of duplicity.
-#
-# Duplicity is free software; you can redistribute it and/or modify it
-# under the terms of the GNU General Public License as published by the
-# Free Software Foundation; either version 2 of the License, or (at your
-# option) any later version.
-#
-# Duplicity is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with duplicity; if not, write to the Free Software Foundation,
-# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-import os.path
-import duplicity.backend
-
-class SXBackend(duplicity.backend.Backend):
- """Connect to remote store using Skylable Protocol"""
- def __init__(self, parsed_url):
- duplicity.backend.Backend.__init__(self, parsed_url)
- self.url_string = parsed_url.url_string
-
- def _put(self, source_path, remote_filename):
- remote_path = os.path.join(self.url_string, remote_filename)
- commandline = "sxcp {0} {1}".format(source_path.name, remote_path)
- self.subprocess_popen(commandline)
-
- def _get(self, remote_filename, local_path):
- remote_path = os.path.join(self.url_string, remote_filename)
- commandline = "sxcp {0} {1}".format(remote_path, local_path.name)
- self.subprocess_popen(commandline)
-
- def _list(self):
- # Do a long listing to avoid connection reset
- commandline = "sxls {0}".format(self.url_string)
- _, l, _ = self.subprocess_popen(commandline)
- # Look for our files as the last element of a long list line
- return [x[x.rindex('/')+1:].split()[-1] for x in l.split('\n') if x and not x.startswith("total ")]
-
- def _delete(self, filename):
- commandline = "sxrm {0}/{1}".format(self.url_string, filename)
- self.subprocess_popen(commandline)
-
-duplicity.backend.register_backend("sx", SXBackend)
=== modified file 'duplicity/backends/tahoebackend.py'
--- duplicity/backends/tahoebackend.py 2014-04-22 15:33:00 +0000
+++ duplicity/backends/tahoebackend.py 2014-10-15 12:12:04 +0000
@@ -20,8 +20,9 @@
import duplicity.backend
from duplicity import log
-from duplicity.errors import BackendException
+from duplicity.errors import * #@UnusedWildImport
+from commands import getstatusoutput
class TAHOEBackend(duplicity.backend.Backend):
"""
@@ -35,8 +36,10 @@
self.alias = url[0]
- if len(url) > 1:
+ if len(url) > 2:
self.directory = "/".join(url[1:])
+ elif len(url) == 2:
+ self.directory = url[1]
else:
self.directory = ""
@@ -56,20 +59,28 @@
def run(self, *args):
cmd = " ".join(args)
- _, output, _ = self.subprocess_popen(cmd)
- return output
-
- def _put(self, source_path, remote_filename):
+ log.Debug("tahoe execute: %s" % cmd)
+ (status, output) = getstatusoutput(cmd)
+
+ if status != 0:
+ raise BackendException("Error running %s" % cmd)
+ else:
+ return output
+
+ def put(self, source_path, remote_filename=None):
self.run("tahoe", "cp", source_path.name, self.get_remote_path(remote_filename))
- def _get(self, remote_filename, local_path):
+ def get(self, remote_filename, local_path):
self.run("tahoe", "cp", self.get_remote_path(remote_filename), local_path.name)
+ local_path.setdata()
def _list(self):
- output = self.run("tahoe", "ls", self.get_remote_path())
- return output.split('\n') if output else []
+ log.Debug("tahoe: List")
+ return self.run("tahoe", "ls", self.get_remote_path()).split('\n')
- def _delete(self, filename):
- self.run("tahoe", "rm", self.get_remote_path(filename))
+ def delete(self, filename_list):
+ log.Debug("tahoe: delete(%s)" % filename_list)
+ for filename in filename_list:
+ self.run("tahoe", "rm", self.get_remote_path(filename))
duplicity.backend.register_backend("tahoe", TAHOEBackend)
=== modified file 'duplicity/backends/webdavbackend.py'
--- duplicity/backends/webdavbackend.py 2014-06-28 14:43:36 +0000
+++ duplicity/backends/webdavbackend.py 2014-10-15 12:12:04 +0000
@@ -26,14 +26,14 @@
import re
import urllib
import urllib2
-import urlparse
import xml.dom.minidom
import duplicity.backend
from duplicity import globals
from duplicity import log
-from duplicity import util
-from duplicity.errors import BackendException, FatalBackendException
+from duplicity.errors import * #@UnusedWildImport
+from duplicity import urlparse_2_5 as urlparser
+from duplicity.backend import retry_fatal
class CustomMethodRequest(urllib2.Request):
"""
@@ -54,7 +54,7 @@
global socket, ssl
import socket, ssl
except ImportError:
- raise FatalBackendException("Missing socket or ssl libraries.")
+ raise FatalBackendError("Missing socket or ssl libraries.")
httplib.HTTPSConnection.__init__(self, *args, **kwargs)
@@ -71,21 +71,21 @@
break
# still no cacert file, inform user
if not self.cacert_file:
- raise FatalBackendException("""For certificate verification a cacert database file is needed in one of these locations: %s
+ raise FatalBackendError("""For certificate verification a cacert database file is needed in one of these locations: %s
Hints:
Consult the man page, chapter 'SSL Certificate Verification'.
Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""" % ", ".join(cacert_candidates) )
# check if file is accessible (libssl errors are not very detailed)
if not os.access(self.cacert_file, os.R_OK):
- raise FatalBackendException("Cacert database file '%s' is not readable." % cacert_file)
+ raise FatalBackendError("Cacert database file '%s' is not readable." % cacert_file)
def connect(self):
# create new socket
sock = socket.create_connection((self.host, self.port),
self.timeout)
- if self.tunnel_host:
+ if self._tunnel_host:
self.sock = sock
- self.tunnel()
+ self._tunnel()
# wrap the socket in ssl using verification
self.sock = ssl.wrap_socket(sock,
@@ -96,9 +96,9 @@
def request(self, *args, **kwargs):
try:
return httplib.HTTPSConnection.request(self, *args, **kwargs)
- except ssl.SSLError as e:
+ except ssl.SSLError, e:
# encapsulate ssl errors
- raise BackendException("SSL failed: %s" % util.uexc(e),log.ErrorCode.backend_error)
+ raise BackendException("SSL failed: %s" % str(e),log.ErrorCode.backend_error)
class WebDAVBackend(duplicity.backend.Backend):
@@ -126,7 +126,7 @@
self.username = parsed_url.username
self.password = self.get_password()
- self.directory = self.sanitize_path(parsed_url.path)
+ self.directory = self._sanitize_path(parsed_url.path)
log.Info("Using WebDAV protocol %s" % (globals.webdav_proto,))
log.Info("Using WebDAV host %s port %s" % (parsed_url.hostname, parsed_url.port))
@@ -134,34 +134,31 @@
self.conn = None
- def sanitize_path(self,path):
+ def _sanitize_path(self,path):
if path:
foldpath = re.compile('/+')
return foldpath.sub('/', path + '/' )
else:
return '/'
- def getText(self,nodelist):
+ def _getText(self,nodelist):
rc = ""
for node in nodelist:
if node.nodeType == node.TEXT_NODE:
rc = rc + node.data
return rc
- def _retry_cleanup(self):
- self.connect(forced=True)
-
- def connect(self, forced=False):
+ def _connect(self, forced=False):
"""
Connect or re-connect to the server, updates self.conn
# reconnect on errors as a precaution, there are errors e.g.
# "[Errno 32] Broken pipe" or SSl errors that render the connection unusable
"""
- if not forced and self.conn \
+ if self.retry_count<=1 and self.conn \
and self.conn.host == self.parsed_url.hostname: return
- log.Info("WebDAV create connection on '%s'" % (self.parsed_url.hostname))
- self._close()
+ log.Info("WebDAV create connection on '%s' (retry %s) " % (self.parsed_url.hostname,self.retry_count) )
+ if self.conn: self.conn.close()
# http schemes needed for redirect urls from servers
if self.parsed_url.scheme in ['webdav','http']:
self.conn = httplib.HTTPConnection(self.parsed_url.hostname, self.parsed_url.port)
@@ -171,19 +168,17 @@
else:
self.conn = VerifiedHTTPSConnection(self.parsed_url.hostname, self.parsed_url.port)
else:
- raise FatalBackendException("WebDAV Unknown URI scheme: %s" % (self.parsed_url.scheme))
+ raise FatalBackendError("WebDAV Unknown URI scheme: %s" % (self.parsed_url.scheme))
- def _close(self):
- if self.conn:
- self.conn.close()
+ def close(self):
+ self.conn.close()
def request(self, method, path, data=None, redirected=0):
"""
Wraps the connection.request method to retry once if authentication is
required
"""
- self._close() # or we get previous request's data or exception
- self.connect()
+ self._connect()
quoted_path = urllib.quote(path,"/:~")
@@ -202,12 +197,12 @@
if redirect_url:
log.Notice("WebDAV redirect to: %s " % urllib.unquote(redirect_url) )
if redirected > 10:
- raise FatalBackendException("WebDAV redirected 10 times. Giving up.")
+ raise FatalBackendError("WebDAV redirected 10 times. Giving up.")
self.parsed_url = duplicity.backend.ParsedUrl(redirect_url)
- self.directory = self.sanitize_path(self.parsed_url.path)
+ self.directory = self._sanitize_path(self.parsed_url.path)
return self.request(method,self.directory,data,redirected+1)
else:
- raise FatalBackendException("WebDAV missing location header in redirect response.")
+ raise FatalBackendError("WebDAV missing location header in redirect response.")
elif response.status == 401:
response.read()
response.close()
@@ -267,7 +262,10 @@
auth_string = self.digest_auth_handler.get_authorization(dummy_req, self.digest_challenge)
return 'Digest %s' % auth_string
+ @retry_fatal
def _list(self):
+ """List files in directory"""
+ log.Info("Listing directory %s on WebDAV server" % (self.directory,))
response = None
try:
self.headers['Depth'] = "1"
@@ -292,14 +290,14 @@
dom = xml.dom.minidom.parseString(document)
result = []
for href in dom.getElementsByTagName('d:href') + dom.getElementsByTagName('D:href'):
- filename = self.taste_href(href)
+ filename = self.__taste_href(href)
if filename:
result.append(filename)
+ if response: response.close()
return result
- except Exception as e:
+ except Exception, e:
+ if response: response.close()
raise e
- finally:
- if response: response.close()
def makedir(self):
"""Make (nested) directories on the server."""
@@ -311,6 +309,7 @@
for i in range(1,len(dirs)):
d="/".join(dirs[0:i+1])+"/"
+ self.close() # or we get previous request's data or exception
self.headers['Depth'] = "1"
response = self.request("PROPFIND", d)
del self.headers['Depth']
@@ -319,20 +318,22 @@
if response.status == 404:
log.Info("Creating missing directory %s" % d)
+ self.close() # or we get previous request's data or exception
res = self.request("MKCOL", d)
if res.status != 201:
raise BackendException("WebDAV MKCOL %s failed: %s %s" % (d,res.status,res.reason))
+ self.close()
- def taste_href(self, href):
+ def __taste_href(self, href):
"""
Internal helper to taste the given href node and, if
it is a duplicity file, collect it as a result file.
@return: A matching filename, or None if the href did not match.
"""
- raw_filename = self.getText(href.childNodes).strip()
- parsed_url = urlparse.urlparse(urllib.unquote(raw_filename))
+ raw_filename = self._getText(href.childNodes).strip()
+ parsed_url = urlparser.urlparse(urllib.unquote(raw_filename))
filename = parsed_url.path
log.Debug("webdav path decoding and translation: "
"%s -> %s" % (raw_filename, filename))
@@ -361,8 +362,11 @@
else:
return None
- def _get(self, remote_filename, local_path):
+ @retry_fatal
+ def get(self, remote_filename, local_path):
+ """Get remote filename, saving it to local_path"""
url = self.directory + remote_filename
+ log.Info("Retrieving %s from WebDAV server" % (url ,))
response = None
try:
target_file = local_path.open("wb")
@@ -373,19 +377,25 @@
#import hashlib
#log.Info("WebDAV GOT %s bytes with md5=%s" % (len(data),hashlib.md5(data).hexdigest()) )
assert not target_file.close()
+ local_path.setdata()
response.close()
else:
status = response.status
reason = response.reason
response.close()
raise BackendException("Bad status code %s reason %s." % (status,reason))
- except Exception as e:
+ if response: response.close()
+ except Exception, e:
+ if response: response.close()
raise e
- finally:
- if response: response.close()
- def _put(self, source_path, remote_filename):
+ @retry_fatal
+ def put(self, source_path, remote_filename = None):
+ """Transfer source_path to remote_filename"""
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
url = self.directory + remote_filename
+ log.Info("Saving %s on WebDAV server" % (url ,))
response = None
try:
source_file = source_path.open("rb")
@@ -399,28 +409,32 @@
reason = response.reason
response.close()
raise BackendException("Bad status code %s reason %s." % (status,reason))
- except Exception as e:
+ if response: response.close()
+ except Exception, e:
+ if response: response.close()
raise e
- finally:
- if response: response.close()
- def _delete(self, filename):
- url = self.directory + filename
- response = None
- try:
- response = self.request("DELETE", url)
- if response.status in [200, 204]:
- response.read()
- response.close()
- else:
- status = response.status
- reason = response.reason
- response.close()
- raise BackendException("Bad status code %s reason %s." % (status,reason))
- except Exception as e:
- raise e
- finally:
- if response: response.close()
+ @retry_fatal
+ def delete(self, filename_list):
+ """Delete files in filename_list"""
+ for filename in filename_list:
+ url = self.directory + filename
+ log.Info("Deleting %s from WebDAV server" % (url ,))
+ response = None
+ try:
+ response = self.request("DELETE", url)
+ if response.status in [200, 204]:
+ response.read()
+ response.close()
+ else:
+ status = response.status
+ reason = response.reason
+ response.close()
+ raise BackendException("Bad status code %s reason %s." % (status,reason))
+ if response: response.close()
+ except Exception, e:
+ if response: response.close()
+ raise e
duplicity.backend.register_backend("webdav", WebDAVBackend)
duplicity.backend.register_backend("webdavs", WebDAVBackend)
=== renamed file 'duplicity/backends/par2backend.py' => 'duplicity/backends/~par2wrapperbackend.py'
--- duplicity/backends/par2backend.py 2014-05-12 07:09:00 +0000
+++ duplicity/backends/~par2wrapperbackend.py 2014-10-15 12:12:04 +0000
@@ -16,16 +16,15 @@
# along with duplicity; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-from future_builtins import filter
-
import os
import re
from duplicity import backend
-from duplicity.errors import BackendException
+from duplicity.errors import UnsupportedBackendScheme, BackendException
+from duplicity.pexpect import run
from duplicity import log
from duplicity import globals
-class Par2Backend(backend.Backend):
+class Par2WrapperBackend(backend.Backend):
"""This backend wrap around other backends and create Par2 recovery files
before the file and the Par2 files are transfered with the wrapped backend.
@@ -44,15 +43,13 @@
except AttributeError:
self.common_options = "-q -q"
- self.wrapped_backend = backend.get_backend_object(parsed_url.url_string)
-
- for attr in ['_get', '_put', '_list', '_delete', '_delete_list',
- '_query', '_query_list', '_retry_cleanup', '_error_code',
- '_move', '_close']:
- if hasattr(self.wrapped_backend, attr):
- setattr(self, attr, getattr(self, attr[1:]))
-
- def transfer(self, method, source_path, remote_filename):
+ try:
+ url_string = self.parsed_url.url_string.lstrip('par2+')
+ self.wrapped_backend = backend.get_backend(url_string)
+ except:
+ raise UnsupportedBackendScheme(self.parsed_url.url_string)
+
+ def put(self, source_path, remote_filename = None):
"""create Par2 files and transfer the given file and the Par2 files
with the wrapped backend.
@@ -60,37 +57,34 @@
temp-filename later on. So first of all create a tempdir and symlink
the soure_path with remote_filename into this.
"""
- import pexpect
+ if remote_filename is None:
+ remote_filename = source_path.get_filename()
par2temp = source_path.get_temp_in_same_dir()
par2temp.mkdir()
source_symlink = par2temp.append(remote_filename)
- source_target = source_path.get_canonical()
- if not os.path.isabs(source_target):
- source_target = os.path.join(os.getcwd(), source_target)
- os.symlink(source_target, source_symlink.get_canonical())
+ os.symlink(source_path.get_canonical(), source_symlink.get_canonical())
source_symlink.setdata()
log.Info("Create Par2 recovery files")
par2create = 'par2 c -r%d -n1 %s %s' % (self.redundancy, self.common_options, source_symlink.get_canonical())
- out, returncode = pexpect.run(par2create, -1, True)
+ out, returncode = run(par2create, -1, True)
source_symlink.delete()
files_to_transfer = []
if not returncode:
for file in par2temp.listdir():
files_to_transfer.append(par2temp.append(file))
- method(source_path, remote_filename)
+ ret = self.wrapped_backend.put(source_path, remote_filename)
for file in files_to_transfer:
- method(file, file.get_filename())
+ self.wrapped_backend.put(file, file.get_filename())
par2temp.deltree()
-
- def put(self, local, remote):
- self.transfer(self.wrapped_backend._put, local, remote)
-
- def move(self, local, remote):
- self.transfer(self.wrapped_backend._move, local, remote)
+ return ret
+
+ def move(self, source_path, remote_filename = None):
+ self.put(source_path, remote_filename)
+ source_path.delete()
def get(self, remote_filename, local_path):
"""transfer remote_filename and the related .par2 file into
@@ -100,31 +94,29 @@
If "par2 verify" detect an error transfer the Par2-volumes into the
temp-dir and try to repair.
"""
- import pexpect
par2temp = local_path.get_temp_in_same_dir()
par2temp.mkdir()
local_path_temp = par2temp.append(remote_filename)
- self.wrapped_backend._get(remote_filename, local_path_temp)
+ ret = self.wrapped_backend.get(remote_filename, local_path_temp)
try:
par2file = par2temp.append(remote_filename + '.par2')
- self.wrapped_backend._get(par2file.get_filename(), par2file)
+ self.wrapped_backend.get(par2file.get_filename(), par2file)
par2verify = 'par2 v %s %s %s' % (self.common_options, par2file.get_canonical(), local_path_temp.get_canonical())
- out, returncode = pexpect.run(par2verify, -1, True)
+ out, returncode = run(par2verify, -1, True)
if returncode:
log.Warn("File is corrupt. Try to repair %s" % remote_filename)
- par2volumes = filter(re.compile((r'%s\.vol[\d+]*\.par2' % remote_filename).match,
- self.wrapped_backend._list()))
+ par2volumes = self.list(re.compile(r'%s\.vol[\d+]*\.par2' % remote_filename))
for filename in par2volumes:
file = par2temp.append(filename)
- self.wrapped_backend._get(filename, file)
+ self.wrapped_backend.get(filename, file)
par2repair = 'par2 r %s %s %s' % (self.common_options, par2file.get_canonical(), local_path_temp.get_canonical())
- out, returncode = pexpect.run(par2repair, -1, True)
+ out, returncode = run(par2repair, -1, True)
if returncode:
log.Error("Failed to repair %s" % remote_filename)
@@ -133,26 +125,27 @@
except BackendException:
#par2 file not available
pass
- finally:
- local_path_temp.rename(local_path)
- par2temp.deltree()
+ local_path_temp.rename(local_path)
+ par2temp.deltree()
+ return ret
- def delete(self, filename):
- """delete given filename and its .par2 files
+ def list(self, filter = re.compile(r'(?!.*\.par2$)')):
+ """default filter all files that ends with ".par"
+ filter can be a re.compile instance or False for all remote files
"""
- self.wrapped_backend._delete(filename)
-
- remote_list = self.list()
- filename_list = [filename]
- c = re.compile(r'%s(?:\.vol[\d+]*)?\.par2' % filename)
- for remote_filename in remote_list:
- if c.match(remote_filename):
- self.wrapped_backend._delete(remote_filename)
-
- def delete_list(self, filename_list):
+ list = self.wrapped_backend.list()
+ if not filter:
+ return list
+ filtered_list = []
+ for item in list:
+ if filter.match(item):
+ filtered_list.append(item)
+ return filtered_list
+
+ def delete(self, filename_list):
"""delete given filename_list and all .par2 files that belong to them
"""
- remote_list = self.list()
+ remote_list = self.list(False)
for filename in filename_list[:]:
c = re.compile(r'%s(?:\.vol[\d+]*)?\.par2' % filename)
@@ -160,25 +153,46 @@
if c.match(remote_filename):
filename_list.append(remote_filename)
- return self.wrapped_backend._delete_list(filename_list)
-
-
- def list(self):
- return self.wrapped_backend._list()
-
- def retry_cleanup(self):
- self.wrapped_backend._retry_cleanup()
-
- def error_code(self, operation, e):
- return self.wrapped_backend._error_code(operation, e)
-
- def query(self, filename):
- return self.wrapped_backend._query(filename)
-
- def query_list(self, filename_list):
- return self.wrapped_backend._query(filename_list)
+ return self.wrapped_backend.delete(filename_list)
+
+ """just return the output of coresponding wrapped backend
+ for all other functions
+ """
+ def query_info(self, filename_list, raise_errors=True):
+ return self.wrapped_backend.query_info(filename_list, raise_errors)
+
+ def get_password(self):
+ return self.wrapped_backend.get_password()
+
+ def munge_password(self, commandline):
+ return self.wrapped_backend.munge_password(commandline)
+
+ def run_command(self, commandline):
+ return self.wrapped_backend.run_command(commandline)
+ def run_command_persist(self, commandline):
+ return self.wrapped_backend.run_command_persist(commandline)
+
+ def popen(self, commandline):
+ return self.wrapped_backend.popen(commandline)
+ def popen_persist(self, commandline):
+ return self.wrapped_backend.popen_persist(commandline)
+
+ def _subprocess_popen(self, commandline):
+ return self.wrapped_backend._subprocess_popen(commandline)
+
+ def subprocess_popen(self, commandline):
+ return self.wrapped_backend.subprocess_popen(commandline)
+
+ def subprocess_popen_persist(self, commandline):
+ return self.wrapped_backend.subprocess_popen_persist(commandline)
def close(self):
- self.wrapped_backend._close()
-
-backend.register_backend_prefix('par2', Par2Backend)
+ return self.wrapped_backend.close()
+
+"""register this backend with leading "par2+" for all already known backends
+
+files must be sorted in duplicity.backend.import_backends to catch
+all supported backends
+"""
+for item in backend._backends.keys():
+ backend.register_backend('par2+' + item, Par2WrapperBackend)
=== modified file 'duplicity/cached_ops.py'
--- duplicity/cached_ops.py 2014-04-17 20:50:57 +0000
+++ duplicity/cached_ops.py 2014-10-15 12:12:04 +0000
@@ -34,7 +34,7 @@
def __call__(self, *args):
try:
return self.cache[args]
- except (KeyError, TypeError) as e:
+ except (KeyError, TypeError), e:
result = self.f(*args)
if not isinstance(e, TypeError):
# TypeError most likely means that args is not hashable
=== modified file 'duplicity/collections.py'
--- duplicity/collections.py 2014-04-25 23:53:46 +0000
+++ duplicity/collections.py 2014-10-15 12:12:04 +0000
@@ -21,8 +21,6 @@
"""Classes and functions on collections of backup volumes"""
-from future_builtins import filter, map
-
import types
import gettext
@@ -98,7 +96,7 @@
self.set_manifest(filename)
else:
assert pr.volume_number is not None
- assert pr.volume_number not in self.volume_name_dict, \
+ assert not self.volume_name_dict.has_key(pr.volume_number), \
(self.volume_name_dict, filename)
self.volume_name_dict[pr.volume_number] = filename
@@ -149,7 +147,7 @@
try:
self.backend.delete(rfn)
except Exception:
- log.Debug(_("BackupSet.delete: missing %s") % [util.ufn(f) for f in rfn])
+ log.Debug(_("BackupSet.delete: missing %s") % map(util.ufn, rfn))
pass
for lfn in globals.archive_dir.listdir():
pr = file_naming.parse(lfn)
@@ -160,7 +158,7 @@
try:
globals.archive_dir.append(lfn).delete()
except Exception:
- log.Debug(_("BackupSet.delete: missing %s") % [util.ufn(f) for f in lfn])
+ log.Debug(_("BackupSet.delete: missing %s") % map(util.ufn, lfn))
pass
util.release_lockfile()
@@ -224,7 +222,7 @@
# public key w/o secret key
try:
manifest_buffer = self.backend.get_data(self.remote_manifest_name)
- except GPGError as message:
+ except GPGError, message:
#TODO: We check for gpg v1 and v2 messages, should be an error code.
if ("secret key not available" in message.args[0] or
"No secret key" in message.args[0]):
@@ -249,7 +247,8 @@
assert self.info_set
volume_num_list = self.volume_name_dict.keys()
volume_num_list.sort()
- volume_filenames = [self.volume_name_dict[x] for x in volume_num_list]
+ volume_filenames = map(lambda x: self.volume_name_dict[x],
+ volume_num_list)
if self.remote_manifest_name:
# For convenience of implementation for restart support, we treat
# local partial manifests as this set's remote manifest. But
@@ -339,7 +338,7 @@
"""
Return a list of sets in chain earlier or equal to time
"""
- older_incsets = [s for s in self.incset_list if s.end_time <= time]
+ older_incsets = filter(lambda s: s.end_time <= time, self.incset_list)
return [self.fullset] + older_incsets
def get_last(self):
@@ -528,7 +527,7 @@
return sig_dp.filtered_open("rb")
else:
filename_to_fileobj = self.backend.get_fileobj_read
- return [filename_to_fileobj(f) for f in self.get_filenames(time)]
+ return map(filename_to_fileobj, self.get_filenames(time))
def delete(self, keep_full=False):
"""
@@ -799,7 +798,7 @@
missing files.
"""
log.Debug(_("Extracting backup chains from list of files: %s")
- % [util.ufn(f) for f in filename_list])
+ % map(util.ufn, filename_list))
# First put filenames in set form
sets = []
def add_to_sets(filename):
@@ -817,8 +816,7 @@
sets.append(new_set)
else:
log.Debug(_("Ignoring file (rejected by backup set) '%s'") % util.ufn(filename))
- for f in filename_list:
- add_to_sets(f)
+ map(add_to_sets, filename_list)
sets, incomplete_sets = self.get_sorted_sets(sets)
chains, orphaned_sets = [], []
@@ -841,8 +839,7 @@
else:
log.Debug(_("Found orphaned set %s") % (set.get_timestr(),))
orphaned_sets.append(set)
- for s in sets:
- add_to_chains(s)
+ map(add_to_chains, sets)
return (chains, orphaned_sets, incomplete_sets)
def get_sorted_sets(self, set_list):
@@ -858,7 +855,7 @@
else:
time_set_pairs.append((set.end_time, set))
time_set_pairs.sort()
- return ([p[1] for p in time_set_pairs], incomplete_sets)
+ return (map(lambda p: p[1], time_set_pairs), incomplete_sets)
def get_signature_chains(self, local, filelist = None):
"""
@@ -919,7 +916,7 @@
# Build dictionary from end_times to lists of corresponding chains
endtime_chain_dict = {}
for chain in chain_list:
- if chain.end_time in endtime_chain_dict:
+ if endtime_chain_dict.has_key(chain.end_time):
endtime_chain_dict[chain.end_time].append(chain)
else:
endtime_chain_dict[chain.end_time] = [chain]
@@ -954,14 +951,15 @@
if not self.all_backup_chains:
raise CollectionsError("No backup chains found")
- covering_chains = [c for c in self.all_backup_chains
- if c.start_time <= time <= c.end_time]
+ covering_chains = filter(lambda c: c.start_time <= time <= c.end_time,
+ self.all_backup_chains)
if len(covering_chains) > 1:
raise CollectionsError("Two chains cover the given time")
elif len(covering_chains) == 1:
return covering_chains[0]
- old_chains = [c for c in self.all_backup_chains if c.end_time < time]
+ old_chains = filter(lambda c: c.end_time < time,
+ self.all_backup_chains)
if old_chains:
return old_chains[-1]
else:
@@ -978,12 +976,13 @@
if not self.all_sig_chains:
raise CollectionsError("No signature chains found")
- covering_chains = [c for c in self.all_sig_chains
- if c.start_time <= time <= c.end_time]
+ covering_chains = filter(lambda c: c.start_time <= time <= c.end_time,
+ self.all_sig_chains)
if covering_chains:
return covering_chains[-1] # prefer local if multiple sig chains
- old_chains = [c for c in self.all_sig_chains if c.end_time < time]
+ old_chains = filter(lambda c: c.end_time < time,
+ self.all_sig_chains)
if old_chains:
return old_chains[-1]
else:
@@ -1025,9 +1024,9 @@
def sort_sets(self, setlist):
"""Return new list containing same elems of setlist, sorted by time"""
- pairs = [(s.get_time(), s) for s in setlist]
+ pairs = map(lambda s: (s.get_time(), s), setlist)
pairs.sort()
- return [p[1] for p in pairs]
+ return map(lambda p: p[1], pairs)
def get_chains_older_than(self, t):
"""
=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py 2014-09-30 13:22:24 +0000
+++ duplicity/commandline.py 2014-10-15 12:12:04 +0000
@@ -21,8 +21,6 @@
"""Parse command line, check for consistency, and set globals"""
-from future_builtins import filter
-
from copy import copy
import optparse
import os
@@ -112,7 +110,7 @@
def check_time(option, opt, value):
try:
return dup_time.genstrtotime(value)
- except dup_time.TimeException as e:
+ except dup_time.TimeException, e:
raise optparse.OptionValueError(str(e))
def check_verbosity(option, opt, value):
@@ -210,6 +208,13 @@
global select_opts, select_files, full_backup
global list_current, collection_status, cleanup, remove_time, verify
+ def use_gio(*args):
+ try:
+ import duplicity.backends.giobackend
+ backend.force_backend(duplicity.backends.giobackend.GIOBackend)
+ except ImportError:
+ log.FatalError(_("Unable to load gio backend: %s") % str(sys.exc_info()[1]), log.ErrorCode.gio_not_available)
+
def set_log_fd(fd):
if fd < 1:
raise optparse.OptionValueError("log-fd must be greater than zero.")
@@ -358,9 +363,7 @@
# the time specified
parser.add_option("--full-if-older-than", type = "time", dest = "full_force_time", metavar = _("time"))
- parser.add_option("--gio",action = "callback", dest = "use_gio",
- callback = lambda o, s, v, p: (setattr(p.values, o.dest, True),
- old_fn_deprecation(s)))
+ parser.add_option("--gio", action = "callback", callback = use_gio)
parser.add_option("--gpg-options", action = "extend", metavar = _("options"))
@@ -508,7 +511,9 @@
parser.add_option("--s3_multipart_max_timeout", type="int", metavar=_("number"))
# Option to allow the s3/boto backend use the multiprocessing version.
- parser.add_option("--s3-use-multiprocessing", action = "store_true")
+ # By default it is off since it does not work for Python 2.4 or 2.5.
+ if sys.version_info[:2] >= (2, 6):
+ parser.add_option("--s3-use-multiprocessing", action = "store_true")
# Option to allow use of server side encryption in s3
parser.add_option("--s3-use-server-side-encryption", action="store_true", dest="s3_use_sse")
@@ -519,7 +524,7 @@
# sftp command to use (ssh pexpect backend)
parser.add_option("--sftp-command", metavar = _("command"))
- # allow the user to switch cloudfiles backend
+ # sftp command to use (ssh pexpect backend)
parser.add_option("--cf-backend", metavar = _("pyrax|cloudfiles"))
# If set, use short (< 30 char) filenames for all the remote files.
@@ -862,7 +867,6 @@
webdavs://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
gdocs://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
mega://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
- copy://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
dpbx:///%(some_dir)s
""" % dict
=== modified file 'duplicity/compilec.py'
--- duplicity/compilec.py 2014-05-11 11:50:12 +0000
+++ duplicity/compilec.py 2014-10-15 12:12:04 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python
# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# Copyright 2002 Ben Escoto <ben@xxxxxxxxxxx>
=== modified file 'duplicity/diffdir.py'
--- duplicity/diffdir.py 2014-04-25 23:53:46 +0000
+++ duplicity/diffdir.py 2014-10-15 12:12:04 +0000
@@ -27,8 +27,6 @@
the second, the ROPath iterator is put into tar block form.
"""
-from future_builtins import map
-
import cStringIO, types, math
from duplicity import statistics
from duplicity import util
@@ -81,8 +79,8 @@
global stats
stats = statistics.StatsDeltaProcess()
if type(dirsig_fileobj_list) is types.ListType:
- sig_iter = combine_path_iters([sigtar2path_iter(x) for x
- in dirsig_fileobj_list])
+ sig_iter = combine_path_iters(map(sigtar2path_iter,
+ dirsig_fileobj_list))
else:
sig_iter = sigtar2path_iter(dirsig_fileobj_list)
delta_iter = get_delta_iter(path_iter, sig_iter)
@@ -344,7 +342,8 @@
else:
break # assumed triple_list sorted, so can exit now
- triple_list = [x for x in map(get_triple, range(len(path_iter_list))) if x]
+ triple_list = filter(lambda x: x, map(get_triple,
+ range(len(path_iter_list))))
while triple_list:
triple_list.sort()
yield triple_list[0][2]
@@ -376,7 +375,7 @@
"""
Return path iter combining signatures in list of open sig files
"""
- return combine_path_iters([sigtar2path_iter(x) for x in sig_infp_list])
+ return combine_path_iters(map(sigtar2path_iter, sig_infp_list))
class FileWithReadCounter:
@@ -390,7 +389,7 @@
def read(self, length = -1):
try:
buf = self.infile.read(length)
- except IOError as ex:
+ except IOError, ex:
buf = ""
log.Warn(_("Error %s getting delta for %s") % (str(ex), util.ufn(self.infile.name)))
if stats:
@@ -462,7 +461,7 @@
TarBlockIter initializer
"""
self.input_iter = input_iter
- self.offset = 0 # total length of data read
+ self.offset = 0l # total length of data read
self.process_waiting = False # process_continued has more blocks
self.process_next_vol_number = None # next volume number to write in multivol
self.previous_index = None # holds index of last block returned
@@ -565,7 +564,7 @@
Return closing string for tarfile, reset offset
"""
blocks, remainder = divmod(self.offset, tarfile.RECORDSIZE) #@UnusedVariable
- self.offset = 0
+ self.offset = 0l
return '\0' * (tarfile.RECORDSIZE - remainder) # remainder can be 0
def __iter__(self):
@@ -737,5 +736,5 @@
return 512 # set minimum of 512 bytes
else:
# Split file into about 2000 pieces, rounding to 512
- file_blocksize = int((file_len / (2000 * 512)) * 512)
+ file_blocksize = long((file_len / (2000 * 512)) * 512)
return min(file_blocksize, globals.max_blocksize)
=== modified file 'duplicity/dup_temp.py'
--- duplicity/dup_temp.py 2014-04-17 20:53:21 +0000
+++ duplicity/dup_temp.py 2014-10-15 12:12:04 +0000
@@ -179,9 +179,9 @@
tgt = self.dirpath.append(self.remname)
src_iter = SrcIter(src)
if pr.compressed:
- gpg.GzipWriteFile(src_iter, tgt.name, size = sys.maxsize)
+ gpg.GzipWriteFile(src_iter, tgt.name, size = sys.maxint)
elif pr.encrypted:
- gpg.GPGWriteFile(src_iter, tgt.name, globals.gpg_profile, size = sys.maxsize)
+ gpg.GPGWriteFile(src_iter, tgt.name, globals.gpg_profile, size = sys.maxint)
else:
os.system("cp -p \"%s\" \"%s\"" % (src.name, tgt.name))
globals.backend.move(tgt) #@UndefinedVariable
@@ -195,7 +195,7 @@
src_iter = SrcIter(src)
pr = file_naming.parse(self.permname)
if pr.compressed:
- gpg.GzipWriteFile(src_iter, tgt.name, size = sys.maxsize)
+ gpg.GzipWriteFile(src_iter, tgt.name, size = sys.maxint)
os.unlink(src.name)
else:
os.rename(src.name, tgt.name)
=== modified file 'duplicity/dup_threading.py'
--- duplicity/dup_threading.py 2014-04-17 21:13:48 +0000
+++ duplicity/dup_threading.py 2014-10-15 12:12:04 +0000
@@ -192,7 +192,7 @@
if state['error'] is None:
return state['value']
else:
- raise state['error'].with_traceback(state['trace'])
+ raise state['error'], None, state['trace']
finally:
cv.release()
@@ -207,7 +207,7 @@
cv.release()
return (True, waiter)
- except Exception as e:
+ except Exception, e:
cv.acquire()
state['done'] = True
state['error'] = e
=== modified file 'duplicity/dup_time.py'
--- duplicity/dup_time.py 2014-04-25 23:53:46 +0000
+++ duplicity/dup_time.py 2014-10-15 12:12:04 +0000
@@ -21,8 +21,6 @@
"""Provide time related exceptions and functions"""
-from future_builtins import map
-
import time, types, re, calendar
from duplicity import globals
@@ -64,7 +62,7 @@
def setcurtime(time_in_secs = None):
"""Sets the current time in curtime and curtimestr"""
global curtime, curtimestr
- t = time_in_secs or int(time.time())
+ t = time_in_secs or long(time.time())
assert type(t) in (types.LongType, types.IntType)
curtime, curtimestr = t, timetostring(t)
@@ -139,9 +137,9 @@
# even when we're not in the same timezone that wrote the
# string
if len(timestring) == 16:
- return int(utc_in_secs)
+ return long(utc_in_secs)
else:
- return int(utc_in_secs + tzdtoseconds(timestring[19:]))
+ return long(utc_in_secs + tzdtoseconds(timestring[19:]))
except (TypeError, ValueError, AssertionError):
return None
@@ -171,7 +169,7 @@
if seconds == 1:
partlist.append("1 second")
elif not partlist or seconds > 1:
- if isinstance(seconds, (types.LongType, types.IntType)):
+ if isinstance(seconds, int) or isinstance(seconds, long):
partlist.append("%s seconds" % seconds)
else:
partlist.append("%.2f seconds" % seconds)
=== modified file 'duplicity/errors.py'
--- duplicity/errors.py 2014-04-21 19:21:45 +0000
+++ duplicity/errors.py 2014-10-15 12:12:04 +0000
@@ -23,8 +23,6 @@
Error/exception classes that do not fit naturally anywhere else.
"""
-from duplicity import log
-
class DuplicityError(Exception):
pass
@@ -70,11 +68,9 @@
"""
Raised to indicate a backend specific problem.
"""
- def __init__(self, msg, code=log.ErrorCode.backend_error):
- super(BackendException, self).__init__(msg)
- self.code = code
+ pass
-class FatalBackendException(BackendException):
+class FatalBackendError(DuplicityError):
"""
Raised to indicate a backend failed fatally.
"""
=== modified file 'duplicity/file_naming.py'
--- duplicity/file_naming.py 2014-04-17 21:49:37 +0000
+++ duplicity/file_naming.py 2014-10-15 12:12:04 +0000
@@ -158,7 +158,7 @@
"""
Convert string s in base 36 to long int
"""
- total = 0
+ total = 0L
for i in range(len(s)):
total *= 36
digit_ord = ord(s[i])
=== modified file 'duplicity/globals.py'
--- duplicity/globals.py 2014-05-12 07:09:00 +0000
+++ duplicity/globals.py 2014-10-15 12:12:04 +0000
@@ -87,7 +87,7 @@
gpg_options = ''
# Maximum file blocksize
-max_blocksize = 2048
+max_blocksize = 2048L
# If true, filelists and directory statistics will be split on
# nulls instead of newlines.
@@ -287,6 +287,3 @@
# Verbatim par2 other options
par2_options = ""
-
-# Whether to enable gio backend
-use_gio = False
=== modified file 'duplicity/gpg.py'
--- duplicity/gpg.py 2014-04-20 06:06:34 +0000
+++ duplicity/gpg.py 2014-10-15 12:12:04 +0000
@@ -215,7 +215,7 @@
msg += unicode(line.strip(), locale.getpreferredencoding(), 'replace') + u"\n"
msg += u"===== End GnuPG log =====\n"
if not (msg.find(u"invalid packet (ctb=14)") > -1):
- raise GPGError(msg)
+ raise GPGError, msg
else:
return ""
=== modified file 'duplicity/gpginterface.py'
--- duplicity/gpginterface.py 2014-04-17 22:03:10 +0000
+++ duplicity/gpginterface.py 2014-10-15 12:12:04 +0000
@@ -353,14 +353,14 @@
if attach_fhs == None: attach_fhs = {}
for std in _stds:
- if std not in attach_fhs \
+ if not attach_fhs.has_key(std) \
and std not in create_fhs:
attach_fhs.setdefault(std, getattr(sys, std))
handle_passphrase = 0
if self.passphrase != None \
- and 'passphrase' not in attach_fhs \
+ and not attach_fhs.has_key('passphrase') \
and 'passphrase' not in create_fhs:
handle_passphrase = 1
create_fhs.append('passphrase')
@@ -384,18 +384,18 @@
process = Process()
for fh_name in create_fhs + attach_fhs.keys():
- if fh_name not in _fd_modes:
- raise KeyError(
+ if not _fd_modes.has_key(fh_name):
+ raise KeyError, \
"unrecognized filehandle name '%s'; must be one of %s" \
- % (fh_name, _fd_modes.keys()))
+ % (fh_name, _fd_modes.keys())
for fh_name in create_fhs:
# make sure the user doesn't specify a filehandle
# to be created *and* attached
- if fh_name in attach_fhs:
- raise ValueError(
+ if attach_fhs.has_key(fh_name):
+ raise ValueError, \
"cannot have filehandle '%s' in both create_fhs and attach_fhs" \
- % fh_name)
+ % fh_name
pipe = os.pipe()
# fix by drt@xxxxxxxxxxxxx noting
@@ -660,7 +660,7 @@
if self.returned == None:
self.thread.join()
if self.returned != 0:
- raise IOError("GnuPG exited non-zero, with code %d" % (self.returned >> 8))
+ raise IOError, "GnuPG exited non-zero, with code %d" % (self.returned >> 8)
def threaded_waitpid(process):
=== modified file 'duplicity/lazy.py'
--- duplicity/lazy.py 2014-04-18 14:32:30 +0000
+++ duplicity/lazy.py 2014-10-15 12:12:04 +0000
@@ -23,51 +23,46 @@
import os
+from duplicity.static import * #@UnusedWildImport
+
class Iter:
"""Hold static methods for the manipulation of lazy iterators"""
- @staticmethod
def filter(predicate, iterator): #@NoSelf
"""Like filter in a lazy functional programming language"""
for i in iterator:
if predicate(i):
yield i
- @staticmethod
def map(function, iterator): #@NoSelf
"""Like map in a lazy functional programming language"""
for i in iterator:
yield function(i)
- @staticmethod
def foreach(function, iterator): #@NoSelf
"""Run function on each element in iterator"""
for i in iterator:
function(i)
- @staticmethod
def cat(*iters): #@NoSelf
"""Lazily concatenate iterators"""
for iter in iters:
for i in iter:
yield i
- @staticmethod
def cat2(iter_of_iters): #@NoSelf
"""Lazily concatenate iterators, iterated by big iterator"""
for iter in iter_of_iters:
for i in iter:
yield i
- @staticmethod
def empty(iter): #@NoSelf
"""True if iterator has length 0"""
for i in iter: #@UnusedVariable
return None
return 1
- @staticmethod
def equal(iter1, iter2, verbose = None, operator = lambda x, y: x == y): #@NoSelf
"""True if iterator 1 has same elements as iterator 2
@@ -93,7 +88,6 @@
print "End when i2 = %s" % (i2,)
return None
- @staticmethod
def Or(iter): #@NoSelf
"""True if any element in iterator is true. Short circuiting"""
i = None
@@ -102,7 +96,6 @@
return i
return i
- @staticmethod
def And(iter): #@NoSelf
"""True if all elements in iterator are true. Short circuiting"""
i = 1
@@ -111,7 +104,6 @@
return i
return i
- @staticmethod
def len(iter): #@NoSelf
"""Return length of iterator"""
i = 0
@@ -122,7 +114,6 @@
return i
i = i+1
- @staticmethod
def foldr(f, default, iter): #@NoSelf
"""foldr the "fundamental list recursion operator"?"""
try:
@@ -131,7 +122,6 @@
return default
return f(next, Iter.foldr(f, default, iter))
- @staticmethod
def foldl(f, default, iter): #@NoSelf
"""the fundamental list iteration operator.."""
while 1:
@@ -141,7 +131,6 @@
return default
default = f(default, next)
- @staticmethod
def multiplex(iter, num_of_forks, final_func = None, closing_func = None): #@NoSelf
"""Split a single iterater into a number of streams
@@ -202,6 +191,8 @@
return tuple(map(make_iterator, range(num_of_forks)))
+MakeStatic(Iter)
+
class IterMultiplex2:
"""Multiplex an iterator into 2 parts
=== modified file 'duplicity/librsync.py'
--- duplicity/librsync.py 2014-04-17 21:54:04 +0000
+++ duplicity/librsync.py 2014-10-15 12:12:04 +0000
@@ -26,7 +26,7 @@
"""
-from . import _librsync
+import _librsync
import types, array
blocksize = _librsync.RS_JOB_BLOCKSIZE
@@ -90,7 +90,7 @@
self._add_to_inbuf()
try:
self.eof, len_inbuf_read, cycle_out = self.maker.cycle(self.inbuf)
- except _librsync.librsyncError as e:
+ except _librsync.librsyncError, e:
raise librsyncError(str(e))
self.inbuf = self.inbuf[len_inbuf_read:]
self.outbuf.fromstring(cycle_out)
@@ -126,7 +126,7 @@
LikeFile.__init__(self, infile)
try:
self.maker = _librsync.new_sigmaker(blocksize)
- except _librsync.librsyncError as e:
+ except _librsync.librsyncError, e:
raise librsyncError(str(e))
class DeltaFile(LikeFile):
@@ -148,7 +148,7 @@
assert not signature.close()
try:
self.maker = _librsync.new_deltamaker(sig_string)
- except _librsync.librsyncError as e:
+ except _librsync.librsyncError, e:
raise librsyncError(str(e))
@@ -167,7 +167,7 @@
raise TypeError("basis_file must be a (true) file")
try:
self.maker = _librsync.new_patchmaker(basis_file)
- except _librsync.librsyncError as e:
+ except _librsync.librsyncError, e:
raise librsyncError(str(e))
@@ -182,7 +182,7 @@
"""Return new signature instance"""
try:
self.sig_maker = _librsync.new_sigmaker(blocksize)
- except _librsync.librsyncError as e:
+ except _librsync.librsyncError, e:
raise librsyncError(str(e))
self.gotsig = None
self.buffer = ""
@@ -201,7 +201,7 @@
"""Run self.buffer through sig_maker, add to self.sig_string"""
try:
eof, len_buf_read, cycle_out = self.sig_maker.cycle(self.buffer)
- except _librsync.librsyncError as e:
+ except _librsync.librsyncError, e:
raise librsyncError(str(e))
self.buffer = self.buffer[len_buf_read:]
self.sigstring_list.append(cycle_out)
=== modified file 'duplicity/log.py'
--- duplicity/log.py 2014-04-16 20:45:09 +0000
+++ duplicity/log.py 2014-10-15 12:12:04 +0000
@@ -49,6 +49,7 @@
return DupToLoggerLevel(verb)
def LevelName(level):
+ level = LoggerToDupLevel(level)
if level >= 9: return "DEBUG"
elif level >= 5: return "INFO"
elif level >= 3: return "NOTICE"
@@ -58,10 +59,12 @@
def Log(s, verb_level, code=1, extra=None, force_print=False):
"""Write s to stderr if verbosity level low enough"""
global _logger
+ # controlLine is a terrible hack until duplicity depends on Python 2.5
+ # and its logging 'extra' keyword that allows a custom record dictionary.
if extra:
- controlLine = '%d %s' % (code, extra)
+ _logger.controlLine = '%d %s' % (code, extra)
else:
- controlLine = '%d' % (code)
+ _logger.controlLine = '%d' % (code)
if not s:
s = '' # If None is passed, standard logging would render it as 'None'
@@ -76,9 +79,8 @@
if not isinstance(s, unicode):
s = s.decode("utf8", "replace")
- _logger.log(DupToLoggerLevel(verb_level), s,
- extra={'levelName': LevelName(verb_level),
- 'controlLine': controlLine})
+ _logger.log(DupToLoggerLevel(verb_level), s)
+ _logger.controlLine = None
if force_print:
_logger.setLevel(initial_level)
@@ -303,6 +305,22 @@
shutdown()
sys.exit(code)
+class DupLogRecord(logging.LogRecord):
+ """Custom log record that holds a message code"""
+ def __init__(self, controlLine, *args, **kwargs):
+ global _logger
+ logging.LogRecord.__init__(self, *args, **kwargs)
+ self.controlLine = controlLine
+ self.levelName = LevelName(self.levelno)
+
+class DupLogger(logging.Logger):
+ """Custom logger that creates special code-bearing records"""
+ # controlLine is a terrible hack until duplicity depends on Python 2.5
+ # and its logging 'extra' keyword that allows a custom record dictionary.
+ controlLine = None
+ def makeRecord(self, name, lvl, fn, lno, msg, args, exc_info, *argv, **kwargs):
+ return DupLogRecord(self.controlLine, name, lvl, fn, lno, msg, args, exc_info)
+
class OutFilter(logging.Filter):
"""Filter that only allows warning or less important messages"""
def filter(self, record):
@@ -319,6 +337,7 @@
if _logger:
return
+ logging.setLoggerClass(DupLogger)
_logger = logging.getLogger("duplicity")
# Default verbosity allows notices and above
=== modified file 'duplicity/manifest.py'
--- duplicity/manifest.py 2014-04-25 23:20:12 +0000
+++ duplicity/manifest.py 2014-10-15 12:12:04 +0000
@@ -21,8 +21,6 @@
"""Create and edit manifest for session contents"""
-from future_builtins import filter
-
import re
from duplicity import log
=== modified file 'duplicity/patchdir.py'
--- duplicity/patchdir.py 2014-04-29 23:49:01 +0000
+++ duplicity/patchdir.py 2014-10-15 12:12:04 +0000
@@ -19,8 +19,6 @@
# along with duplicity; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-from future_builtins import filter, map
-
import re #@UnusedImport
import types
import os
@@ -505,7 +503,7 @@
if final_ropath.exists():
# otherwise final patch was delete
yield final_ropath
- except Exception as e:
+ except Exception, e:
filename = normalized[-1].get_ropath().get_relative_path()
log.Warn(_("Error '%s' patching %s") %
(util.uexc(e), util.ufn(filename)),
@@ -519,10 +517,11 @@
the restrict_index.
"""
- diff_iters = [difftar2path_iter(x) for x in tarfile_list]
+ diff_iters = map( difftar2path_iter, tarfile_list )
if restrict_index:
# Apply filter before integration
- diff_iters = [filter_path_iter(x, restrict_index) for x in diff_iters]
+ diff_iters = map( lambda i: filter_path_iter( i, restrict_index ),
+ diff_iters )
return integrate_patch_iters( diff_iters )
def Write_ROPaths( base_path, rop_iter ):
=== modified file 'duplicity/path.py'
--- duplicity/path.py 2014-06-28 15:48:11 +0000
+++ duplicity/path.py 2014-10-15 12:12:04 +0000
@@ -26,8 +26,6 @@
"""
-from future_builtins import filter
-
import stat, errno, socket, time, re, gzip
from duplicity import tarfile
@@ -206,10 +204,10 @@
self.mode = tarinfo.mode
self.stat = StatResult()
- """ Set user and group id
+ """ Set user and group id
use numeric id if name lookup fails
OR
- --numeric-owner is set
+ --numeric-owner is set
"""
try:
if globals.numeric_owner:
@@ -507,7 +505,7 @@
"""Refresh stat cache"""
try:
self.stat = os.lstat(self.name)
- except OSError as e:
+ except OSError, e:
err_string = errno.errorcode[e[0]]
if err_string in ["ENOENT", "ENOTDIR", "ELOOP", "ENOTCONN"]:
self.stat, self.type = None, None # file doesn't exist
=== added file 'duplicity/pexpect.py'
--- duplicity/pexpect.py 1970-01-01 00:00:00 +0000
+++ duplicity/pexpect.py 2014-10-15 12:12:04 +0000
@@ -0,0 +1,1845 @@
+"""Pexpect is a Python module for spawning child applications and controlling
+them automatically. Pexpect can be used for automating interactive applications
+such as ssh, ftp, passwd, telnet, etc. It can be used to a automate setup
+scripts for duplicating software package installations on different servers. It
+can be used for automated software testing. Pexpect is in the spirit of Don
+Libes' Expect, but Pexpect is pure Python. Other Expect-like modules for Python
+require TCL and Expect or require C extensions to be compiled. Pexpect does not
+use C, Expect, or TCL extensions. It should work on any platform that supports
+the standard Python pty module. The Pexpect interface focuses on ease of use so
+that simple tasks are easy.
+
+There are two main interfaces to Pexpect -- the function, run() and the class,
+spawn. You can call the run() function to execute a command and return the
+output. This is a handy replacement for os.system().
+
+For example::
+
+ pexpect.run('ls -la')
+
+The more powerful interface is the spawn class. You can use this to spawn an
+external child command and then interact with the child by sending lines and
+expecting responses.
+
+For example::
+
+ child = pexpect.spawn('scp foo myname@xxxxxxxxxxxxxxxx:.')
+ child.expect ('Password:')
+ child.sendline (mypassword)
+
+This works even for commands that ask for passwords or other input outside of
+the normal stdio streams.
+
+Credits: Noah Spurrier, Richard Holden, Marco Molteni, Kimberley Burchett,
+Robert Stone, Hartmut Goebel, Chad Schroeder, Erick Tryzelaar, Dave Kirby, Ids
+vander Molen, George Todd, Noel Taylor, Nicolas D. Cesar, Alexander Gattin,
+Geoffrey Marshall, Francisco Lourenco, Glen Mabey, Karthik Gurusamy, Fernando
+Perez, Corey Minyard, Jon Cohen, Guillaume Chazarain, Andrew Ryan, Nick
+Craig-Wood, Andrew Stone, Jorgen Grahn (Let me know if I forgot anyone.)
+
+Free, open source, and all that good stuff.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Pexpect Copyright (c) 2008 Noah Spurrier
+http://pexpect.sourceforge.net/
+
+$Id: pexpect.py,v 1.1 2009/01/06 22:11:37 loafman Exp $
+"""
+
+try:
+ import os, sys, time
+ import select
+ import string
+ import re
+ import struct
+ import resource
+ import types
+ import pty
+ import tty
+ import termios
+ import fcntl
+ import errno
+ import traceback
+ import signal
+except ImportError, e:
+ raise ImportError (str(e) + """
+
+A critical module was not found. Probably this operating system does not
+support it. Pexpect is intended for UNIX-like operating systems.""")
+
+__version__ = '2.3'
+__revision__ = '$Revision: 1.1 $'
+__all__ = ['ExceptionPexpect', 'EOF', 'TIMEOUT', 'spawn', 'run', 'which',
+ 'split_command_line', '__version__', '__revision__']
+
+# Exception classes used by this module.
+class ExceptionPexpect(Exception):
+
+ """Base class for all exceptions raised by this module.
+ """
+
+ def __init__(self, value):
+
+ self.value = value
+
+ def __str__(self):
+
+ return str(self.value)
+
+ def get_trace(self):
+
+ """This returns an abbreviated stack trace with lines that only concern
+ the caller. In other words, the stack trace inside the Pexpect module
+ is not included. """
+
+ tblist = traceback.extract_tb(sys.exc_info()[2])
+ #tblist = filter(self.__filter_not_pexpect, tblist)
+ tblist = [item for item in tblist if self.__filter_not_pexpect(item)]
+ tblist = traceback.format_list(tblist)
+ return ''.join(tblist)
+
+ def __filter_not_pexpect(self, trace_list_item):
+
+ """This returns True if list item 0 the string 'pexpect.py' in it. """
+
+ if trace_list_item[0].find('pexpect.py') == -1:
+ return True
+ else:
+ return False
+
+class EOF(ExceptionPexpect):
+
+ """Raised when EOF is read from a child. This usually means the child has exited."""
+
+class TIMEOUT(ExceptionPexpect):
+
+ """Raised when a read time exceeds the timeout. """
+
+##class TIMEOUT_PATTERN(TIMEOUT):
+## """Raised when the pattern match time exceeds the timeout.
+## This is different than a read TIMEOUT because the child process may
+## give output, thus never give a TIMEOUT, but the output
+## may never match a pattern.
+## """
+##class MAXBUFFER(ExceptionPexpect):
+## """Raised when a scan buffer fills before matching an expected pattern."""
+
+def run (command, timeout=-1, withexitstatus=False, events=None, extra_args=None, logfile=None, cwd=None, env=None):
+
+ """
+ This function runs the given command; waits for it to finish; then
+ returns all output as a string. STDERR is included in output. If the full
+ path to the command is not given then the path is searched.
+
+ Note that lines are terminated by CR/LF (\\r\\n) combination even on
+ UNIX-like systems because this is the standard for pseudo ttys. If you set
+ 'withexitstatus' to true, then run will return a tuple of (command_output,
+ exitstatus). If 'withexitstatus' is false then this returns just
+ command_output.
+
+ The run() function can often be used instead of creating a spawn instance.
+ For example, the following code uses spawn::
+
+ from pexpect import * #@UnusedWildImport
+ child = spawn('scp foo myname@xxxxxxxxxxxxxxxx:.')
+ child.expect ('(?i)password')
+ child.sendline (mypassword)
+
+ The previous code can be replace with the following::
+
+ from pexpect import * #@UnusedWildImport
+ run ('scp foo myname@xxxxxxxxxxxxxxxx:.', events={'(?i)password': mypassword})
+
+ Examples
+ ========
+
+ Start the apache daemon on the local machine::
+
+ from pexpect import * #@UnusedWildImport
+ run ("/usr/local/apache/bin/apachectl start")
+
+ Check in a file using SVN::
+
+ from pexpect import * #@UnusedWildImport
+ run ("svn ci -m 'automatic commit' my_file.py")
+
+ Run a command and capture exit status::
+
+ from pexpect import * #@UnusedWildImport
+ (command_output, exitstatus) = run ('ls -l /bin', withexitstatus=1)
+
+ Tricky Examples
+ ===============
+
+ The following will run SSH and execute 'ls -l' on the remote machine. The
+ password 'secret' will be sent if the '(?i)password' pattern is ever seen::
+
+ run ("ssh username@xxxxxxxxxxxxxxxxxxx 'ls -l'", events={'(?i)password':'secret\\n'})
+
+ This will start mencoder to rip a video from DVD. This will also display
+ progress ticks every 5 seconds as it runs. For example::
+
+ from pexpect import * #@UnusedWildImport
+ def print_ticks(d):
+ print d['event_count'],
+ run ("mencoder dvd://1 -o video.avi -oac copy -ovc copy", events={TIMEOUT:print_ticks}, timeout=5)
+
+ The 'events' argument should be a dictionary of patterns and responses.
+ Whenever one of the patterns is seen in the command out run() will send the
+ associated response string. Note that you should put newlines in your
+ string if Enter is necessary. The responses may also contain callback
+ functions. Any callback is function that takes a dictionary as an argument.
+ The dictionary contains all the locals from the run() function, so you can
+ access the child spawn object or any other variable defined in run()
+ (event_count, child, and extra_args are the most useful). A callback may
+ return True to stop the current run process otherwise run() continues until
+ the next event. A callback may also return a string which will be sent to
+ the child. 'extra_args' is not used by directly run(). It provides a way to
+ pass data to a callback function through run() through the locals
+ dictionary passed to a callback. """
+
+ if timeout == -1:
+ child = spawn(command, maxread=2000, logfile=logfile, cwd=cwd, env=env)
+ else:
+ child = spawn(command, timeout=timeout, maxread=2000, logfile=logfile, cwd=cwd, env=env)
+ if events is not None:
+ patterns = events.keys()
+ responses = events.values()
+ else:
+ patterns=None # We assume that EOF or TIMEOUT will save us.
+ responses=None
+ child_result_list = []
+ event_count = 0
+ while 1:
+ try:
+ index = child.expect (patterns)
+ if type(child.after) in types.StringTypes:
+ child_result_list.append(child.before + child.after)
+ else: # child.after may have been a TIMEOUT or EOF, so don't cat those.
+ child_result_list.append(child.before)
+ if type(responses[index]) in types.StringTypes:
+ child.send(responses[index])
+ elif type(responses[index]) is types.FunctionType:
+ callback_result = responses[index](locals())
+ sys.stdout.flush()
+ if type(callback_result) in types.StringTypes:
+ child.send(callback_result)
+ elif callback_result:
+ break
+ else:
+ raise TypeError ('The callback must be a string or function type.')
+ event_count = event_count + 1
+ except TIMEOUT, e:
+ child_result_list.append(child.before)
+ break
+ except EOF, e:
+ child_result_list.append(child.before)
+ break
+ child_result = ''.join(child_result_list)
+ if withexitstatus:
+ child.close()
+ return (child_result, child.exitstatus)
+ else:
+ return child_result
+
+class spawn (object):
+
+ """This is the main class interface for Pexpect. Use this class to start
+ and control child applications. """
+
+ def __init__(self, command, args=[], timeout=30, maxread=2000, searchwindowsize=None, logfile=None, cwd=None, env=None):
+
+ """This is the constructor. The command parameter may be a string that
+ includes a command and any arguments to the command. For example::
+
+ child = pexpect.spawn ('/usr/bin/ftp')
+ child = pexpect.spawn ('/usr/bin/ssh user@xxxxxxxxxxx')
+ child = pexpect.spawn ('ls -latr /tmp')
+
+ You may also construct it with a list of arguments like so::
+
+ child = pexpect.spawn ('/usr/bin/ftp', [])
+ child = pexpect.spawn ('/usr/bin/ssh', ['user@xxxxxxxxxxx'])
+ child = pexpect.spawn ('ls', ['-latr', '/tmp'])
+
+ After this the child application will be created and will be ready to
+ talk to. For normal use, see expect() and send() and sendline().
+
+ Remember that Pexpect does NOT interpret shell meta characters such as
+ redirect, pipe, or wild cards (>, |, or *). This is a common mistake.
+ If you want to run a command and pipe it through another command then
+ you must also start a shell. For example::
+
+ child = pexpect.spawn('/bin/bash -c "ls -l | grep LOG > log_list.txt"')
+ child.expect(pexpect.EOF)
+
+ The second form of spawn (where you pass a list of arguments) is useful
+ in situations where you wish to spawn a command and pass it its own
+ argument list. This can make syntax more clear. For example, the
+ following is equivalent to the previous example::
+
+ shell_cmd = 'ls -l | grep LOG > log_list.txt'
+ child = pexpect.spawn('/bin/bash', ['-c', shell_cmd])
+ child.expect(pexpect.EOF)
+
+ The maxread attribute sets the read buffer size. This is maximum number
+ of bytes that Pexpect will try to read from a TTY at one time. Setting
+ the maxread size to 1 will turn off buffering. Setting the maxread
+ value higher may help performance in cases where large amounts of
+ output are read back from the child. This feature is useful in
+ conjunction with searchwindowsize.
+
+ The searchwindowsize attribute sets the how far back in the incomming
+ seach buffer Pexpect will search for pattern matches. Every time
+ Pexpect reads some data from the child it will append the data to the
+ incomming buffer. The default is to search from the beginning of the
+ imcomming buffer each time new data is read from the child. But this is
+ very inefficient if you are running a command that generates a large
+ amount of data where you want to match The searchwindowsize does not
+ effect the size of the incomming data buffer. You will still have
+ access to the full buffer after expect() returns.
+
+ The logfile member turns on or off logging. All input and output will
+ be copied to the given file object. Set logfile to None to stop
+ logging. This is the default. Set logfile to sys.stdout to echo
+ everything to standard output. The logfile is flushed after each write.
+
+ Example log input and output to a file::
+
+ child = pexpect.spawn('some_command')
+ fout = file('mylog.txt','w')
+ child.logfile = fout
+
+ Example log to stdout::
+
+ child = pexpect.spawn('some_command')
+ child.logfile = sys.stdout
+
+ The logfile_read and logfile_send members can be used to separately log
+ the input from the child and output sent to the child. Sometimes you
+ don't want to see everything you write to the child. You only want to
+ log what the child sends back. For example::
+
+ child = pexpect.spawn('some_command')
+ child.logfile_read = sys.stdout
+
+ To separately log output sent to the child use logfile_send::
+
+ self.logfile_send = fout
+
+ The delaybeforesend helps overcome a weird behavior that many users
+ were experiencing. The typical problem was that a user would expect() a
+ "Password:" prompt and then immediately call sendline() to send the
+ password. The user would then see that their password was echoed back
+ to them. Passwords don't normally echo. The problem is caused by the
+ fact that most applications print out the "Password" prompt and then
+ turn off stdin echo, but if you send your password before the
+ application turned off echo, then you get your password echoed.
+ Normally this wouldn't be a problem when interacting with a human at a
+ real keyboard. If you introduce a slight delay just before writing then
+ this seems to clear up the problem. This was such a common problem for
+ many users that I decided that the default pexpect behavior should be
+ to sleep just before writing to the child application. 1/20th of a
+ second (50 ms) seems to be enough to clear up the problem. You can set
+ delaybeforesend to 0 to return to the old behavior. Most Linux machines
+ don't like this to be below 0.03. I don't know why.
+
+ Note that spawn is clever about finding commands on your path.
+ It uses the same logic that "which" uses to find executables.
+
+ If you wish to get the exit status of the child you must call the
+ close() method. The exit or signal status of the child will be stored
+ in self.exitstatus or self.signalstatus. If the child exited normally
+ then exitstatus will store the exit return code and signalstatus will
+ be None. If the child was terminated abnormally with a signal then
+ signalstatus will store the signal value and exitstatus will be None.
+ If you need more detail you can also read the self.status member which
+ stores the status returned by os.waitpid. You can interpret this using
+ os.WIFEXITED/os.WEXITSTATUS or os.WIFSIGNALED/os.TERMSIG. """
+
+ self.STDIN_FILENO = pty.STDIN_FILENO
+ self.STDOUT_FILENO = pty.STDOUT_FILENO
+ self.STDERR_FILENO = pty.STDERR_FILENO
+ self.stdin = sys.stdin
+ self.stdout = sys.stdout
+ self.stderr = sys.stderr
+
+ self.searcher = None
+ self.ignorecase = False
+ self.before = None
+ self.after = None
+ self.match = None
+ self.match_index = None
+ self.terminated = True
+ self.exitstatus = None
+ self.signalstatus = None
+ self.status = None # status returned by os.waitpid
+ self.flag_eof = False
+ self.pid = None
+ self.child_fd = -1 # initially closed
+ self.timeout = timeout
+ self.delimiter = EOF
+ self.logfile = logfile
+ self.logfile_read = None # input from child (read_nonblocking)
+ self.logfile_send = None # output to send (send, sendline)
+ self.maxread = maxread # max bytes to read at one time into buffer
+ self.buffer = '' # This is the read buffer. See maxread.
+ self.searchwindowsize = searchwindowsize # Anything before searchwindowsize point is preserved, but not searched.
+ # Most Linux machines don't like delaybeforesend to be below 0.03 (30 ms).
+ self.delaybeforesend = 0.05 # Sets sleep time used just before sending data to child. Time in seconds.
+ self.delayafterclose = 0.1 # Sets delay in close() method to allow kernel time to update process status. Time in seconds.
+ self.delayafterterminate = 0.1 # Sets delay in terminate() method to allow kernel time to update process status. Time in seconds.
+ self.softspace = False # File-like object.
+ self.name = '<' + repr(self) + '>' # File-like object.
+ self.encoding = None # File-like object.
+ self.closed = True # File-like object.
+ self.cwd = cwd
+ self.env = env
+ self.__irix_hack = (sys.platform.lower().find('irix')>=0) # This flags if we are running on irix
+ # Solaris uses internal __fork_pty(). All others use pty.fork().
+ if (sys.platform.lower().find('solaris')>=0) or (sys.platform.lower().find('sunos5')>=0):
+ self.use_native_pty_fork = False
+ else:
+ self.use_native_pty_fork = True
+
+
+ # allow dummy instances for subclasses that may not use command or args.
+ if command is None:
+ self.command = None
+ self.args = None
+ self.name = '<pexpect factory incomplete>'
+ else:
+ self._spawn (command, args)
+
+ def __del__(self):
+
+ """This makes sure that no system resources are left open. Python only
+ garbage collects Python objects. OS file descriptors are not Python
+ objects, so they must be handled explicitly. If the child file
+ descriptor was opened outside of this class (passed to the constructor)
+ then this does not close it. """
+
+ if not self.closed:
+ # It is possible for __del__ methods to execute during the
+ # teardown of the Python VM itself. Thus self.close() may
+ # trigger an exception because os.close may be None.
+ # -- Fernando Perez
+ try:
+ self.close()
+ except AttributeError:
+ pass
+
+ def __str__(self):
+
+ """This returns a human-readable string that represents the state of
+ the object. """
+
+ s = []
+ s.append(repr(self))
+ s.append('version: ' + __version__ + ' (' + __revision__ + ')')
+ s.append('command: ' + str(self.command))
+ s.append('args: ' + str(self.args))
+ s.append('searcher: ' + str(self.searcher))
+ s.append('buffer (last 100 chars): ' + str(self.buffer)[-100:])
+ s.append('before (last 100 chars): ' + str(self.before)[-100:])
+ s.append('after: ' + str(self.after))
+ s.append('match: ' + str(self.match))
+ s.append('match_index: ' + str(self.match_index))
+ s.append('exitstatus: ' + str(self.exitstatus))
+ s.append('flag_eof: ' + str(self.flag_eof))
+ s.append('pid: ' + str(self.pid))
+ s.append('child_fd: ' + str(self.child_fd))
+ s.append('closed: ' + str(self.closed))
+ s.append('timeout: ' + str(self.timeout))
+ s.append('delimiter: ' + str(self.delimiter))
+ s.append('logfile: ' + str(self.logfile))
+ s.append('logfile_read: ' + str(self.logfile_read))
+ s.append('logfile_send: ' + str(self.logfile_send))
+ s.append('maxread: ' + str(self.maxread))
+ s.append('ignorecase: ' + str(self.ignorecase))
+ s.append('searchwindowsize: ' + str(self.searchwindowsize))
+ s.append('delaybeforesend: ' + str(self.delaybeforesend))
+ s.append('delayafterclose: ' + str(self.delayafterclose))
+ s.append('delayafterterminate: ' + str(self.delayafterterminate))
+ return '\n'.join(s)
+
+ def _spawn(self,command,args=[]):
+
+ """This starts the given command in a child process. This does all the
+ fork/exec type of stuff for a pty. This is called by __init__. If args
+ is empty then command will be parsed (split on spaces) and args will be
+ set to parsed arguments. """
+
+ # The pid and child_fd of this object get set by this method.
+ # Note that it is difficult for this method to fail.
+ # You cannot detect if the child process cannot start.
+ # So the only way you can tell if the child process started
+ # or not is to try to read from the file descriptor. If you get
+ # EOF immediately then it means that the child is already dead.
+ # That may not necessarily be bad because you may haved spawned a child
+ # that performs some task; creates no stdout output; and then dies.
+
+ # If command is an int type then it may represent a file descriptor.
+ if type(command) == type(0):
+ raise ExceptionPexpect ('Command is an int type. If this is a file descriptor then maybe you want to use fdpexpect.fdspawn which takes an existing file descriptor instead of a command string.')
+
+ if type (args) != type([]):
+ raise TypeError ('The argument, args, must be a list.')
+
+ if args == []:
+ self.args = split_command_line(command)
+ self.command = self.args[0]
+ else:
+ self.args = args[:] # work with a copy
+ self.args.insert (0, command)
+ self.command = command
+
+ command_with_path = which(self.command)
+ if command_with_path is None:
+ raise ExceptionPexpect ('The command was not found or was not executable: %s.' % self.command)
+ self.command = command_with_path
+ self.args[0] = self.command
+
+ self.name = '<' + ' '.join (self.args) + '>'
+
+ assert self.pid is None, 'The pid member should be None.'
+ assert self.command is not None, 'The command member should not be None.'
+
+ if self.use_native_pty_fork:
+ try:
+ self.pid, self.child_fd = pty.fork()
+ except OSError, e:
+ raise ExceptionPexpect('Error! pty.fork() failed: ' + str(e))
+ else: # Use internal __fork_pty
+ self.pid, self.child_fd = self.__fork_pty()
+
+ if self.pid == 0: # Child
+ try:
+ self.child_fd = sys.stdout.fileno() # used by setwinsize()
+ self.setwinsize(24, 80)
+ except Exception:
+ # Some platforms do not like setwinsize (Cygwin).
+ # This will cause problem when running applications that
+ # are very picky about window size.
+ # This is a serious limitation, but not a show stopper.
+ pass
+ # Do not allow child to inherit open file descriptors from parent.
+ max_fd = resource.getrlimit(resource.RLIMIT_NOFILE)[0]
+ for i in range (3, max_fd):
+ try:
+ os.close (i)
+ except OSError:
+ pass
+
+ # I don't know why this works, but ignoring SIGHUP fixes a
+ # problem when trying to start a Java daemon with sudo
+ # (specifically, Tomcat).
+ signal.signal(signal.SIGHUP, signal.SIG_IGN)
+
+ if self.cwd is not None:
+ os.chdir(self.cwd)
+ if self.env is None:
+ os.execv(self.command, self.args)
+ else:
+ os.execvpe(self.command, self.args, self.env)
+
+ # Parent
+ self.terminated = False
+ self.closed = False
+
+ def __fork_pty(self):
+
+ """This implements a substitute for the forkpty system call. This
+ should be more portable than the pty.fork() function. Specifically,
+ this should work on Solaris.
+
+ Modified 10.06.05 by Geoff Marshall: Implemented __fork_pty() method to
+ resolve the issue with Python's pty.fork() not supporting Solaris,
+ particularly ssh. Based on patch to posixmodule.c authored by Noah
+ Spurrier::
+
+ http://mail.python.org/pipermail/python-dev/2003-May/035281.html
+
+ """
+
+ parent_fd, child_fd = os.openpty()
+ if parent_fd < 0 or child_fd < 0:
+ raise ExceptionPexpect, "Error! Could not open pty with os.openpty()."
+
+ pid = os.fork()
+ if pid < 0:
+ raise ExceptionPexpect, "Error! Failed os.fork()."
+ elif pid == 0:
+ # Child.
+ os.close(parent_fd)
+ self.__pty_make_controlling_tty(child_fd)
+
+ os.dup2(child_fd, 0)
+ os.dup2(child_fd, 1)
+ os.dup2(child_fd, 2)
+
+ if child_fd > 2:
+ os.close(child_fd)
+ else:
+ # Parent.
+ os.close(child_fd)
+
+ return pid, parent_fd
+
+ def __pty_make_controlling_tty(self, tty_fd):
+
+ """This makes the pseudo-terminal the controlling tty. This should be
+ more portable than the pty.fork() function. Specifically, this should
+ work on Solaris. """
+
+ child_name = os.ttyname(tty_fd)
+
+ # Disconnect from controlling tty if still connected.
+ fd = os.open("/dev/tty", os.O_RDWR | os.O_NOCTTY);
+ if fd >= 0:
+ os.close(fd)
+
+ os.setsid()
+
+ # Verify we are disconnected from controlling tty
+ try:
+ fd = os.open("/dev/tty", os.O_RDWR | os.O_NOCTTY);
+ if fd >= 0:
+ os.close(fd)
+ raise ExceptionPexpect, "Error! We are not disconnected from a controlling tty."
+ except Exception:
+ # Good! We are disconnected from a controlling tty.
+ pass
+
+ # Verify we can open child pty.
+ fd = os.open(child_name, os.O_RDWR);
+ if fd < 0:
+ raise ExceptionPexpect, "Error! Could not open child pty, " + child_name
+ else:
+ os.close(fd)
+
+ # Verify we now have a controlling tty.
+ fd = os.open("/dev/tty", os.O_WRONLY)
+ if fd < 0:
+ raise ExceptionPexpect, "Error! Could not open controlling tty, /dev/tty"
+ else:
+ os.close(fd)
+
+ def fileno (self): # File-like object.
+
+ """This returns the file descriptor of the pty for the child.
+ """
+
+ return self.child_fd
+
+ def close (self, force=True): # File-like object.
+
+ """This closes the connection with the child application. Note that
+ calling close() more than once is valid. This emulates standard Python
+ behavior with files. Set force to True if you want to make sure that
+ the child is terminated (SIGKILL is sent if the child ignores SIGHUP
+ and SIGINT). """
+
+ if not self.closed:
+ self.flush()
+ os.close (self.child_fd)
+ time.sleep(self.delayafterclose) # Give kernel time to update process status.
+ if self.isalive():
+ if not self.terminate(force):
+ raise ExceptionPexpect ('close() could not terminate the child using terminate()')
+ self.child_fd = -1
+ self.closed = True
+ #self.pid = None
+
+ def flush (self): # File-like object.
+
+ """This does nothing. It is here to support the interface for a
+ File-like object. """
+
+ pass
+
+ def isatty (self): # File-like object.
+
+ """This returns True if the file descriptor is open and connected to a
+ tty(-like) device, else False. """
+
+ return os.isatty(self.child_fd)
+
+ def waitnoecho (self, timeout=-1):
+
+ """This waits until the terminal ECHO flag is set False. This returns
+ True if the echo mode is off. This returns False if the ECHO flag was
+ not set False before the timeout. This can be used to detect when the
+ child is waiting for a password. Usually a child application will turn
+ off echo mode when it is waiting for the user to enter a password. For
+ example, instead of expecting the "password:" prompt you can wait for
+ the child to set ECHO off::
+
+ p = pexpect.spawn ('ssh user@xxxxxxxxxxx')
+ p.waitnoecho()
+ p.sendline(mypassword)
+
+ If timeout is None then this method to block forever until ECHO flag is
+ False.
+
+ """
+
+ if timeout == -1:
+ timeout = self.timeout
+ if timeout is not None:
+ end_time = time.time() + timeout
+ while True:
+ if not self.getecho():
+ return True
+ if timeout < 0 and timeout is not None:
+ return False
+ if timeout is not None:
+ timeout = end_time - time.time()
+ time.sleep(0.1)
+
+ def getecho (self):
+
+ """This returns the terminal echo mode. This returns True if echo is
+ on or False if echo is off. Child applications that are expecting you
+ to enter a password often set ECHO False. See waitnoecho(). """
+
+ attr = termios.tcgetattr(self.child_fd)
+ if attr[3] & termios.ECHO:
+ return True
+ return False
+
+ def setecho (self, state):
+
+ """This sets the terminal echo mode on or off. Note that anything the
+ child sent before the echo will be lost, so you should be sure that
+ your input buffer is empty before you call setecho(). For example, the
+ following will work as expected::
+
+ p = pexpect.spawn('cat')
+ p.sendline ('1234') # We will see this twice (once from tty echo and again from cat).
+ p.expect (['1234'])
+ p.expect (['1234'])
+ p.setecho(False) # Turn off tty echo
+ p.sendline ('abcd') # We will set this only once (echoed by cat).
+ p.sendline ('wxyz') # We will set this only once (echoed by cat)
+ p.expect (['abcd'])
+ p.expect (['wxyz'])
+
+ The following WILL NOT WORK because the lines sent before the setecho
+ will be lost::
+
+ p = pexpect.spawn('cat')
+ p.sendline ('1234') # We will see this twice (once from tty echo and again from cat).
+ p.setecho(False) # Turn off tty echo
+ p.sendline ('abcd') # We will set this only once (echoed by cat).
+ p.sendline ('wxyz') # We will set this only once (echoed by cat)
+ p.expect (['1234'])
+ p.expect (['1234'])
+ p.expect (['abcd'])
+ p.expect (['wxyz'])
+ """
+
+ self.child_fd
+ attr = termios.tcgetattr(self.child_fd)
+ if state:
+ attr[3] = attr[3] | termios.ECHO
+ else:
+ attr[3] = attr[3] & ~termios.ECHO
+ # I tried TCSADRAIN and TCSAFLUSH, but these were inconsistent
+ # and blocked on some platforms. TCSADRAIN is probably ideal if it worked.
+ termios.tcsetattr(self.child_fd, termios.TCSANOW, attr)
+
+ def read_nonblocking (self, size = 1, timeout = -1):
+
+ """This reads at most size characters from the child application. It
+ includes a timeout. If the read does not complete within the timeout
+ period then a TIMEOUT exception is raised. If the end of file is read
+ then an EOF exception will be raised. If a log file was set using
+ setlog() then all data will also be written to the log file.
+
+ If timeout is None then the read may block indefinitely. If timeout is -1
+ then the self.timeout value is used. If timeout is 0 then the child is
+ polled and if there was no data immediately ready then this will raise
+ a TIMEOUT exception.
+
+ The timeout refers only to the amount of time to read at least one
+ character. This is not effected by the 'size' parameter, so if you call
+ read_nonblocking(size=100, timeout=30) and only one character is
+ available right away then one character will be returned immediately.
+ It will not wait for 30 seconds for another 99 characters to come in.
+
+ This is a wrapper around os.read(). It uses select.select() to
+ implement the timeout. """
+
+ if self.closed:
+ raise ValueError ('I/O operation on closed file in read_nonblocking().')
+
+ if timeout == -1:
+ timeout = self.timeout
+
+ # Note that some systems such as Solaris do not give an EOF when
+ # the child dies. In fact, you can still try to read
+ # from the child_fd -- it will block forever or until TIMEOUT.
+ # For this case, I test isalive() before doing any reading.
+ # If isalive() is false, then I pretend that this is the same as EOF.
+ if not self.isalive():
+ r,w,e = self.__select([self.child_fd], [], [], 0) # timeout of 0 means "poll" @UnusedVariable
+ if not r:
+ self.flag_eof = True
+ raise EOF ('End Of File (EOF) in read_nonblocking(). Braindead platform.')
+ elif self.__irix_hack:
+ # This is a hack for Irix. It seems that Irix requires a long delay before checking isalive.
+ # This adds a 2 second delay, but only when the child is terminated.
+ r, w, e = self.__select([self.child_fd], [], [], 2) #@UnusedVariable
+ if not r and not self.isalive():
+ self.flag_eof = True
+ raise EOF ('End Of File (EOF) in read_nonblocking(). Pokey platform.')
+
+ r,w,e = self.__select([self.child_fd], [], [], timeout) #@UnusedVariable
+
+ if not r:
+ if not self.isalive():
+ # Some platforms, such as Irix, will claim that their processes are alive;
+ # then timeout on the select; and then finally admit that they are not alive.
+ self.flag_eof = True
+ raise EOF ('End of File (EOF) in read_nonblocking(). Very pokey platform.')
+ else:
+ raise TIMEOUT ('Timeout exceeded in read_nonblocking().')
+
+ if self.child_fd in r:
+ try:
+ s = os.read(self.child_fd, size)
+ except OSError, e: # Linux does this
+ self.flag_eof = True
+ raise EOF ('End Of File (EOF) in read_nonblocking(). Exception style platform.')
+ if s == '': # BSD style
+ self.flag_eof = True
+ raise EOF ('End Of File (EOF) in read_nonblocking(). Empty string style platform.')
+
+ if self.logfile is not None:
+ self.logfile.write (s)
+ self.logfile.flush()
+ if self.logfile_read is not None:
+ self.logfile_read.write (s)
+ self.logfile_read.flush()
+
+ return s
+
+ raise ExceptionPexpect ('Reached an unexpected state in read_nonblocking().')
+
+ def read (self, size = -1): # File-like object.
+
+ """This reads at most "size" bytes from the file (less if the read hits
+ EOF before obtaining size bytes). If the size argument is negative or
+ omitted, read all data until EOF is reached. The bytes are returned as
+ a string object. An empty string is returned when EOF is encountered
+ immediately. """
+
+ if size == 0:
+ return ''
+ if size < 0:
+ self.expect (self.delimiter) # delimiter default is EOF
+ return self.before
+
+ # I could have done this more directly by not using expect(), but
+ # I deliberately decided to couple read() to expect() so that
+ # I would catch any bugs early and ensure consistant behavior.
+ # It's a little less efficient, but there is less for me to
+ # worry about if I have to later modify read() or expect().
+ # Note, it's OK if size==-1 in the regex. That just means it
+ # will never match anything in which case we stop only on EOF.
+ cre = re.compile('.{%d}' % size, re.DOTALL)
+ index = self.expect ([cre, self.delimiter]) # delimiter default is EOF
+ if index == 0:
+ return self.after ### self.before should be ''. Should I assert this?
+ return self.before
+
+ def readline (self, size = -1): # File-like object.
+
+ """This reads and returns one entire line. A trailing newline is kept
+ in the string, but may be absent when a file ends with an incomplete
+ line. Note: This readline() looks for a \\r\\n pair even on UNIX
+ because this is what the pseudo tty device returns. So contrary to what
+ you may expect you will receive the newline as \\r\\n. An empty string
+ is returned when EOF is hit immediately. Currently, the size argument is
+ mostly ignored, so this behavior is not standard for a file-like
+ object. If size is 0 then an empty string is returned. """
+
+ if size == 0:
+ return ''
+ index = self.expect (['\r\n', self.delimiter]) # delimiter default is EOF
+ if index == 0:
+ return self.before + '\r\n'
+ else:
+ return self.before
+
+ def __iter__ (self): # File-like object.
+
+ """This is to support iterators over a file-like object.
+ """
+
+ return self
+
+ def next (self): # File-like object.
+
+ """This is to support iterators over a file-like object.
+ """
+
+ result = self.readline()
+ if result == "":
+ raise StopIteration
+ return result
+
+ def readlines (self, sizehint = -1): # File-like object.
+
+ """This reads until EOF using readline() and returns a list containing
+ the lines thus read. The optional "sizehint" argument is ignored. """
+
+ lines = []
+ while True:
+ line = self.readline()
+ if not line:
+ break
+ lines.append(line)
+ return lines
+
+ def write(self, s): # File-like object.
+
+ """This is similar to send() except that there is no return value.
+ """
+
+ self.send (s)
+
+ def writelines (self, sequence): # File-like object.
+
+ """This calls write() for each element in the sequence. The sequence
+ can be any iterable object producing strings, typically a list of
+ strings. This does not add line separators There is no return value.
+ """
+
+ for s in sequence:
+ self.write (s)
+
+ def send(self, s):
+
+ """This sends a string to the child process. This returns the number of
+ bytes written. If a log file was set then the data is also written to
+ the log. """
+
+ time.sleep(self.delaybeforesend)
+ if self.logfile is not None:
+ self.logfile.write (s)
+ self.logfile.flush()
+ if self.logfile_send is not None:
+ self.logfile_send.write (s)
+ self.logfile_send.flush()
+ c = os.write(self.child_fd, s)
+ return c
+
+ def sendline(self, s=''):
+
+ """This is like send(), but it adds a line feed (os.linesep). This
+ returns the number of bytes written. """
+
+ n = self.send(s)
+ n = n + self.send (os.linesep)
+ return n
+
+ def sendcontrol(self, char):
+
+ """This sends a control character to the child such as Ctrl-C or
+ Ctrl-D. For example, to send a Ctrl-G (ASCII 7)::
+
+ child.sendcontrol('g')
+
+ See also, sendintr() and sendeof().
+ """
+
+ char = char.lower()
+ a = ord(char)
+ if a>=97 and a<=122:
+ a = a - ord('a') + 1
+ return self.send (chr(a))
+ d = {'@':0, '`':0,
+ '[':27, '{':27,
+ '\\':28, '|':28,
+ ']':29, '}': 29,
+ '^':30, '~':30,
+ '_':31,
+ '?':127}
+ if char not in d:
+ return 0
+ return self.send (chr(d[char]))
+
+ def sendeof(self):
+
+ """This sends an EOF to the child. This sends a character which causes
+ the pending parent output buffer to be sent to the waiting child
+ program without waiting for end-of-line. If it is the first character
+ of the line, the read() in the user program returns 0, which signifies
+ end-of-file. This means to work as expected a sendeof() has to be
+ called at the beginning of a line. This method does not send a newline.
+ It is the responsibility of the caller to ensure the eof is sent at the
+ beginning of a line. """
+
+ ### Hmmm... how do I send an EOF?
+ ###C if ((m = write(pty, *buf, p - *buf)) < 0)
+ ###C return (errno == EWOULDBLOCK) ? n : -1;
+ #fd = sys.stdin.fileno()
+ #old = termios.tcgetattr(fd) # remember current state
+ #attr = termios.tcgetattr(fd)
+ #attr[3] = attr[3] | termios.ICANON # ICANON must be set to recognize EOF
+ #try: # use try/finally to ensure state gets restored
+ # termios.tcsetattr(fd, termios.TCSADRAIN, attr)
+ # if hasattr(termios, 'CEOF'):
+ # os.write (self.child_fd, '%c' % termios.CEOF)
+ # else:
+ # # Silly platform does not define CEOF so assume CTRL-D
+ # os.write (self.child_fd, '%c' % 4)
+ #finally: # restore state
+ # termios.tcsetattr(fd, termios.TCSADRAIN, old)
+ if hasattr(termios, 'VEOF'):
+ char = termios.tcgetattr(self.child_fd)[6][termios.VEOF]
+ else:
+ # platform does not define VEOF so assume CTRL-D
+ char = chr(4)
+ self.send(char)
+
+ def sendintr(self):
+
+ """This sends a SIGINT to the child. It does not require
+ the SIGINT to be the first character on a line. """
+
+ if hasattr(termios, 'VINTR'):
+ char = termios.tcgetattr(self.child_fd)[6][termios.VINTR]
+ else:
+ # platform does not define VINTR so assume CTRL-C
+ char = chr(3)
+ self.send (char)
+
+ def eof (self):
+
+ """This returns True if the EOF exception was ever raised.
+ """
+
+ return self.flag_eof
+
+ def terminate(self, force=False):
+
+ """This forces a child process to terminate. It starts nicely with
+ SIGHUP and SIGINT. If "force" is True then moves onto SIGKILL. This
+ returns True if the child was terminated. This returns False if the
+ child could not be terminated. """
+
+ if not self.isalive():
+ return True
+ try:
+ self.kill(signal.SIGHUP)
+ time.sleep(self.delayafterterminate)
+ if not self.isalive():
+ return True
+ self.kill(signal.SIGCONT)
+ time.sleep(self.delayafterterminate)
+ if not self.isalive():
+ return True
+ self.kill(signal.SIGINT)
+ time.sleep(self.delayafterterminate)
+ if not self.isalive():
+ return True
+ if force:
+ self.kill(signal.SIGKILL)
+ time.sleep(self.delayafterterminate)
+ if not self.isalive():
+ return True
+ else:
+ return False
+ return False
+ except OSError, e:
+ # I think there are kernel timing issues that sometimes cause
+ # this to happen. I think isalive() reports True, but the
+ # process is dead to the kernel.
+ # Make one last attempt to see if the kernel is up to date.
+ time.sleep(self.delayafterterminate)
+ if not self.isalive():
+ return True
+ else:
+ return False
+
+ def wait(self):
+
+ """This waits until the child exits. This is a blocking call. This will
+ not read any data from the child, so this will block forever if the
+ child has unread output and has terminated. In other words, the child
+ may have printed output then called exit(); but, technically, the child
+ is still alive until its output is read. """
+
+ if self.isalive():
+ pid, status = os.waitpid(self.pid, 0) #@UnusedVariable
+ else:
+ raise ExceptionPexpect ('Cannot wait for dead child process.')
+ self.exitstatus = os.WEXITSTATUS(status)
+ if os.WIFEXITED (status):
+ self.status = status
+ self.exitstatus = os.WEXITSTATUS(status)
+ self.signalstatus = None
+ self.terminated = True
+ elif os.WIFSIGNALED (status):
+ self.status = status
+ self.exitstatus = None
+ self.signalstatus = os.WTERMSIG(status)
+ self.terminated = True
+ elif os.WIFSTOPPED (status):
+ raise ExceptionPexpect ('Wait was called for a child process that is stopped. This is not supported. Is some other process attempting job control with our child pid?')
+ return self.exitstatus
+
+ def isalive(self):
+
+ """This tests if the child process is running or not. This is
+ non-blocking. If the child was terminated then this will read the
+ exitstatus or signalstatus of the child. This returns True if the child
+ process appears to be running or False if not. It can take literally
+ SECONDS for Solaris to return the right status. """
+
+ if self.terminated:
+ return False
+
+ if self.flag_eof:
+ # This is for Linux, which requires the blocking form of waitpid to get
+ # status of a defunct process. This is super-lame. The flag_eof would have
+ # been set in read_nonblocking(), so this should be safe.
+ waitpid_options = 0
+ else:
+ waitpid_options = os.WNOHANG
+
+ try:
+ pid, status = os.waitpid(self.pid, waitpid_options)
+ except OSError, e: # No child processes
+ if e[0] == errno.ECHILD:
+ raise ExceptionPexpect ('isalive() encountered condition where "terminated" is 0, but there was no child process. Did someone else call waitpid() on our process?')
+ else:
+ raise e
+
+ # I have to do this twice for Solaris. I can't even believe that I figured this out...
+ # If waitpid() returns 0 it means that no child process wishes to
+ # report, and the value of status is undefined.
+ if pid == 0:
+ try:
+ pid, status = os.waitpid(self.pid, waitpid_options) ### os.WNOHANG) # Solaris!
+ except OSError, e: # This should never happen...
+ if e[0] == errno.ECHILD:
+ raise ExceptionPexpect ('isalive() encountered condition that should never happen. There was no child process. Did someone else call waitpid() on our process?')
+ else:
+ raise e
+
+ # If pid is still 0 after two calls to waitpid() then
+ # the process really is alive. This seems to work on all platforms, except
+ # for Irix which seems to require a blocking call on waitpid or select, so I let read_nonblocking
+ # take care of this situation (unfortunately, this requires waiting through the timeout).
+ if pid == 0:
+ return True
+
+ if pid == 0:
+ return True
+
+ if os.WIFEXITED (status):
+ self.status = status
+ self.exitstatus = os.WEXITSTATUS(status)
+ self.signalstatus = None
+ self.terminated = True
+ elif os.WIFSIGNALED (status):
+ self.status = status
+ self.exitstatus = None
+ self.signalstatus = os.WTERMSIG(status)
+ self.terminated = True
+ elif os.WIFSTOPPED (status):
+ raise ExceptionPexpect ('isalive() encountered condition where child process is stopped. This is not supported. Is some other process attempting job control with our child pid?')
+ return False
+
+ def kill(self, sig):
+
+ """This sends the given signal to the child application. In keeping
+ with UNIX tradition it has a misleading name. It does not necessarily
+ kill the child unless you send the right signal. """
+
+ # Same as os.kill, but the pid is given for you.
+ if self.isalive():
+ os.kill(self.pid, sig)
+
+ def compile_pattern_list(self, patterns):
+
+ """This compiles a pattern-string or a list of pattern-strings.
+ Patterns must be a StringType, EOF, TIMEOUT, SRE_Pattern, or a list of
+ those. Patterns may also be None which results in an empty list (you
+ might do this if waiting for an EOF or TIMEOUT condition without
+ expecting any pattern).
+
+ This is used by expect() when calling expect_list(). Thus expect() is
+ nothing more than::
+
+ cpl = self.compile_pattern_list(pl)
+ return self.expect_list(cpl, timeout)
+
+ If you are using expect() within a loop it may be more
+ efficient to compile the patterns first and then call expect_list().
+ This avoid calls in a loop to compile_pattern_list()::
+
+ cpl = self.compile_pattern_list(my_pattern)
+ while some_condition:
+ ...
+ i = self.expect_list(clp, timeout)
+ ...
+ """
+
+ if patterns is None:
+ return []
+ if type(patterns) is not types.ListType:
+ patterns = [patterns]
+
+ compile_flags = re.DOTALL # Allow dot to match \n
+ if self.ignorecase:
+ compile_flags = compile_flags | re.IGNORECASE
+ compiled_pattern_list = []
+ for p in patterns:
+ if type(p) in types.StringTypes:
+ compiled_pattern_list.append(re.compile(p, compile_flags))
+ elif p is EOF:
+ compiled_pattern_list.append(EOF)
+ elif p is TIMEOUT:
+ compiled_pattern_list.append(TIMEOUT)
+ elif type(p) is type(re.compile('')):
+ compiled_pattern_list.append(p)
+ else:
+ raise TypeError ('Argument must be one of StringTypes, EOF, TIMEOUT, SRE_Pattern, or a list of those type. %s' % str(type(p)))
+
+ return compiled_pattern_list
+
+ def expect(self, pattern, timeout = -1, searchwindowsize=None):
+
+ """This seeks through the stream until a pattern is matched. The
+ pattern is overloaded and may take several types. The pattern can be a
+ StringType, EOF, a compiled re, or a list of any of those types.
+ Strings will be compiled to re types. This returns the index into the
+ pattern list. If the pattern was not a list this returns index 0 on a
+ successful match. This may raise exceptions for EOF or TIMEOUT. To
+ avoid the EOF or TIMEOUT exceptions add EOF or TIMEOUT to the pattern
+ list. That will cause expect to match an EOF or TIMEOUT condition
+ instead of raising an exception.
+
+ If you pass a list of patterns and more than one matches, the first match
+ in the stream is chosen. If more than one pattern matches at that point,
+ the leftmost in the pattern list is chosen. For example::
+
+ # the input is 'foobar'
+ index = p.expect (['bar', 'foo', 'foobar'])
+ # returns 1 ('foo') even though 'foobar' is a "better" match
+
+ Please note, however, that buffering can affect this behavior, since
+ input arrives in unpredictable chunks. For example::
+
+ # the input is 'foobar'
+ index = p.expect (['foobar', 'foo'])
+ # returns 0 ('foobar') if all input is available at once,
+ # but returs 1 ('foo') if parts of the final 'bar' arrive late
+
+ After a match is found the instance attributes 'before', 'after' and
+ 'match' will be set. You can see all the data read before the match in
+ 'before'. You can see the data that was matched in 'after'. The
+ re.MatchObject used in the re match will be in 'match'. If an error
+ occurred then 'before' will be set to all the data read so far and
+ 'after' and 'match' will be None.
+
+ If timeout is -1 then timeout will be set to the self.timeout value.
+
+ A list entry may be EOF or TIMEOUT instead of a string. This will
+ catch these exceptions and return the index of the list entry instead
+ of raising the exception. The attribute 'after' will be set to the
+ exception type. The attribute 'match' will be None. This allows you to
+ write code like this::
+
+ index = p.expect (['good', 'bad', pexpect.EOF, pexpect.TIMEOUT])
+ if index == 0:
+ do_something()
+ elif index == 1:
+ do_something_else()
+ elif index == 2:
+ do_some_other_thing()
+ elif index == 3:
+ do_something_completely_different()
+
+ instead of code like this::
+
+ try:
+ index = p.expect (['good', 'bad'])
+ if index == 0:
+ do_something()
+ elif index == 1:
+ do_something_else()
+ except EOF:
+ do_some_other_thing()
+ except TIMEOUT:
+ do_something_completely_different()
+
+ These two forms are equivalent. It all depends on what you want. You
+ can also just expect the EOF if you are waiting for all output of a
+ child to finish. For example::
+
+ p = pexpect.spawn('/bin/ls')
+ p.expect (pexpect.EOF)
+ print p.before
+
+ If you are trying to optimize for speed then see expect_list().
+ """
+
+ compiled_pattern_list = self.compile_pattern_list(pattern)
+ return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
+
+ def expect_list(self, pattern_list, timeout = -1, searchwindowsize = -1):
+
+ """This takes a list of compiled regular expressions and returns the
+ index into the pattern_list that matched the child output. The list may
+ also contain EOF or TIMEOUT (which are not compiled regular
+ expressions). This method is similar to the expect() method except that
+ expect_list() does not recompile the pattern list on every call. This
+ may help if you are trying to optimize for speed, otherwise just use
+ the expect() method. This is called by expect(). If timeout==-1 then
+ the self.timeout value is used. If searchwindowsize==-1 then the
+ self.searchwindowsize value is used. """
+
+ return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
+
+ def expect_exact(self, pattern_list, timeout = -1, searchwindowsize = -1):
+
+ """This is similar to expect(), but uses plain string matching instead
+ of compiled regular expressions in 'pattern_list'. The 'pattern_list'
+ may be a string; a list or other sequence of strings; or TIMEOUT and
+ EOF.
+
+ This call might be faster than expect() for two reasons: string
+ searching is faster than RE matching and it is possible to limit the
+ search to just the end of the input buffer.
+
+ This method is also useful when you don't want to have to worry about
+ escaping regular expression characters that you want to match."""
+
+ if type(pattern_list) in types.StringTypes or pattern_list in (TIMEOUT, EOF):
+ pattern_list = [pattern_list]
+ return self.expect_loop(searcher_string(pattern_list), timeout, searchwindowsize)
+
+ def expect_loop(self, searcher, timeout = -1, searchwindowsize = -1):
+
+ """This is the common loop used inside expect. The 'searcher' should be
+ an instance of searcher_re or searcher_string, which describes how and what
+ to search for in the input.
+
+ See expect() for other arguments, return value and exceptions. """
+
+ self.searcher = searcher
+
+ if timeout == -1:
+ timeout = self.timeout
+ if timeout is not None:
+ end_time = time.time() + timeout
+ if searchwindowsize == -1:
+ searchwindowsize = self.searchwindowsize
+
+ try:
+ incoming = self.buffer
+ freshlen = len(incoming)
+ while True: # Keep reading until exception or return.
+ index = searcher.search(incoming, freshlen, searchwindowsize)
+ if index >= 0:
+ self.buffer = incoming[searcher.end : ]
+ self.before = incoming[ : searcher.start]
+ self.after = incoming[searcher.start : searcher.end]
+ self.match = searcher.match
+ self.match_index = index
+ return self.match_index
+ # No match at this point
+ if timeout < 0 and timeout is not None:
+ raise TIMEOUT ('Timeout exceeded in expect_any().')
+ # Still have time left, so read more data
+ c = self.read_nonblocking (self.maxread, timeout)
+ freshlen = len(c)
+ time.sleep (0.0001)
+ incoming = incoming + c
+ if timeout is not None:
+ timeout = end_time - time.time()
+ except EOF, e:
+ self.buffer = ''
+ self.before = incoming
+ self.after = EOF
+ index = searcher.eof_index
+ if index >= 0:
+ self.match = EOF
+ self.match_index = index
+ return self.match_index
+ else:
+ self.match = None
+ self.match_index = None
+ raise EOF (str(e) + '\n' + str(self))
+ except TIMEOUT, e:
+ self.buffer = incoming
+ self.before = incoming
+ self.after = TIMEOUT
+ index = searcher.timeout_index
+ if index >= 0:
+ self.match = TIMEOUT
+ self.match_index = index
+ return self.match_index
+ else:
+ self.match = None
+ self.match_index = None
+ raise TIMEOUT (str(e) + '\n' + str(self))
+ except Exception:
+ self.before = incoming
+ self.after = None
+ self.match = None
+ self.match_index = None
+ raise
+
+ def getwinsize(self):
+
+ """This returns the terminal window size of the child tty. The return
+ value is a tuple of (rows, cols). """
+
+ TIOCGWINSZ = getattr(termios, 'TIOCGWINSZ', 1074295912L)
+ s = struct.pack('HHHH', 0, 0, 0, 0)
+ x = fcntl.ioctl(self.fileno(), TIOCGWINSZ, s)
+ return struct.unpack('HHHH', x)[0:2]
+
+ def setwinsize(self, r, c):
+
+ """This sets the terminal window size of the child tty. This will cause
+ a SIGWINCH signal to be sent to the child. This does not change the
+ physical window size. It changes the size reported to TTY-aware
+ applications like vi or curses -- applications that respond to the
+ SIGWINCH signal. """
+
+ # Check for buggy platforms. Some Python versions on some platforms
+ # (notably OSF1 Alpha and RedHat 7.1) truncate the value for
+ # termios.TIOCSWINSZ. It is not clear why this happens.
+ # These platforms don't seem to handle the signed int very well;
+ # yet other platforms like OpenBSD have a large negative value for
+ # TIOCSWINSZ and they don't have a truncate problem.
+ # Newer versions of Linux have totally different values for TIOCSWINSZ.
+ # Note that this fix is a hack.
+ TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)
+ if TIOCSWINSZ == 2148037735L: # L is not required in Python >= 2.2.
+ TIOCSWINSZ = -2146929561 # Same bits, but with sign.
+ # Note, assume ws_xpixel and ws_ypixel are zero.
+ s = struct.pack('HHHH', r, c, 0, 0)
+ fcntl.ioctl(self.fileno(), TIOCSWINSZ, s)
+
+ def interact(self, escape_character = chr(29), input_filter = None, output_filter = None):
+
+ """This gives control of the child process to the interactive user (the
+ human at the keyboard). Keystrokes are sent to the child process, and
+ the stdout and stderr output of the child process is printed. This
+ simply echos the child stdout and child stderr to the real stdout and
+ it echos the real stdin to the child stdin. When the user types the
+ escape_character this method will stop. The default for
+ escape_character is ^]. This should not be confused with ASCII 27 --
+ the ESC character. ASCII 29 was chosen for historical merit because
+ this is the character used by 'telnet' as the escape character. The
+ escape_character will not be sent to the child process.
+
+ You may pass in optional input and output filter functions. These
+ functions should take a string and return a string. The output_filter
+ will be passed all the output from the child process. The input_filter
+ will be passed all the keyboard input from the user. The input_filter
+ is run BEFORE the check for the escape_character.
+
+ Note that if you change the window size of the parent the SIGWINCH
+ signal will not be passed through to the child. If you want the child
+ window size to change when the parent's window size changes then do
+ something like the following example::
+
+ import pexpect, struct, fcntl, termios, signal, sys
+ def sigwinch_passthrough (sig, data):
+ s = struct.pack("HHHH", 0, 0, 0, 0)
+ a = struct.unpack('hhhh', fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ , s))
+ global p
+ p.setwinsize(a[0],a[1])
+ p = pexpect.spawn('/bin/bash') # Note this is global and used in sigwinch_passthrough.
+ signal.signal(signal.SIGWINCH, sigwinch_passthrough)
+ p.interact()
+ """
+
+ # Flush the buffer.
+ self.stdout.write (self.buffer)
+ self.stdout.flush()
+ self.buffer = ''
+ mode = tty.tcgetattr(self.STDIN_FILENO)
+ tty.setraw(self.STDIN_FILENO)
+ try:
+ self.__interact_copy(escape_character, input_filter, output_filter)
+ finally:
+ tty.tcsetattr(self.STDIN_FILENO, tty.TCSAFLUSH, mode)
+
+ def __interact_writen(self, fd, data):
+
+ """This is used by the interact() method.
+ """
+
+ while data != '' and self.isalive():
+ n = os.write(fd, data)
+ data = data[n:]
+
+ def __interact_read(self, fd):
+
+ """This is used by the interact() method.
+ """
+
+ return os.read(fd, 1000)
+
+ def __interact_copy(self, escape_character = None, input_filter = None, output_filter = None):
+
+ """This is used by the interact() method.
+ """
+
+ while self.isalive():
+ r,w,e = self.__select([self.child_fd, self.STDIN_FILENO], [], []) #@UnusedVariable
+ if self.child_fd in r:
+ data = self.__interact_read(self.child_fd)
+ if output_filter: data = output_filter(data)
+ if self.logfile is not None:
+ self.logfile.write (data)
+ self.logfile.flush()
+ os.write(self.STDOUT_FILENO, data)
+ if self.STDIN_FILENO in r:
+ data = self.__interact_read(self.STDIN_FILENO)
+ if input_filter: data = input_filter(data)
+ i = data.rfind(escape_character)
+ if i != -1:
+ data = data[:i]
+ self.__interact_writen(self.child_fd, data)
+ break
+ self.__interact_writen(self.child_fd, data)
+
+ def __select (self, iwtd, owtd, ewtd, timeout=None):
+
+ """This is a wrapper around select.select() that ignores signals. If
+ select.select raises a select.error exception and errno is an EINTR
+ error then it is ignored. Mainly this is used to ignore sigwinch
+ (terminal resize). """
+
+ # if select() is interrupted by a signal (errno==EINTR) then
+ # we loop back and enter the select() again.
+ if timeout is not None:
+ end_time = time.time() + timeout
+ while True:
+ try:
+ return select.select (iwtd, owtd, ewtd, timeout)
+ except select.error, e:
+ if e[0] == errno.EINTR:
+ # if we loop back we have to subtract the amount of time we already waited.
+ if timeout is not None:
+ timeout = end_time - time.time()
+ if timeout < 0:
+ return ([],[],[])
+ else: # something else caused the select.error, so this really is an exception
+ raise
+
+##############################################################################
+# The following methods are no longer supported or allowed.
+
+ def setmaxread (self, maxread):
+
+ """This method is no longer supported or allowed. I don't like getters
+ and setters without a good reason. """
+
+ raise ExceptionPexpect ('This method is no longer supported or allowed. Just assign a value to the maxread member variable.')
+
+ def setlog (self, fileobject):
+
+ """This method is no longer supported or allowed.
+ """
+
+ raise ExceptionPexpect ('This method is no longer supported or allowed. Just assign a value to the logfile member variable.')
+
+##############################################################################
+# End of spawn class
+##############################################################################
+
+class searcher_string (object):
+
+ """This is a plain string search helper for the spawn.expect_any() method.
+
+ Attributes:
+
+ eof_index - index of EOF, or -1
+ timeout_index - index of TIMEOUT, or -1
+
+ After a successful match by the search() method the following attributes
+ are available:
+
+ start - index into the buffer, first byte of match
+ end - index into the buffer, first byte after match
+ match - the matching string itself
+ """
+
+ def __init__(self, strings):
+
+ """This creates an instance of searcher_string. This argument 'strings'
+ may be a list; a sequence of strings; or the EOF or TIMEOUT types. """
+
+ self.eof_index = -1
+ self.timeout_index = -1
+ self._strings = []
+ for n, s in zip(range(len(strings)), strings):
+ if s is EOF:
+ self.eof_index = n
+ continue
+ if s is TIMEOUT:
+ self.timeout_index = n
+ continue
+ self._strings.append((n, s))
+
+ def __str__(self):
+
+ """This returns a human-readable string that represents the state of
+ the object."""
+
+ ss = [ (ns[0],' %d: "%s"' % ns) for ns in self._strings ]
+ ss.append((-1,'searcher_string:'))
+ if self.eof_index >= 0:
+ ss.append ((self.eof_index,' %d: EOF' % self.eof_index))
+ if self.timeout_index >= 0:
+ ss.append ((self.timeout_index,' %d: TIMEOUT' % self.timeout_index))
+ ss.sort()
+ ss = zip(*ss)[1]
+ return '\n'.join(ss)
+
+ def search(self, buffer, freshlen, searchwindowsize=None):
+
+ """This searches 'buffer' for the first occurence of one of the search
+ strings. 'freshlen' must indicate the number of bytes at the end of
+ 'buffer' which have not been searched before. It helps to avoid
+ searching the same, possibly big, buffer over and over again.
+
+ See class spawn for the 'searchwindowsize' argument.
+
+ If there is a match this returns the index of that string, and sets
+ 'start', 'end' and 'match'. Otherwise, this returns -1. """
+
+ absurd_match = len(buffer)
+ first_match = absurd_match
+
+ # 'freshlen' helps a lot here. Further optimizations could
+ # possibly include:
+ #
+ # using something like the Boyer-Moore Fast String Searching
+ # Algorithm; pre-compiling the search through a list of
+ # strings into something that can scan the input once to
+ # search for all N strings; realize that if we search for
+ # ['bar', 'baz'] and the input is '...foo' we need not bother
+ # rescanning until we've read three more bytes.
+ #
+ # Sadly, I don't know enough about this interesting topic. /grahn
+
+ for index, s in self._strings:
+ if searchwindowsize is None:
+ # the match, if any, can only be in the fresh data,
+ # or at the very end of the old data
+ offset = -(freshlen+len(s))
+ else:
+ # better obey searchwindowsize
+ offset = -searchwindowsize
+ n = buffer.find(s, offset)
+ if n >= 0 and n < first_match:
+ first_match = n
+ best_index, best_match = index, s
+ if first_match == absurd_match:
+ return -1
+ self.match = best_match
+ self.start = first_match
+ self.end = self.start + len(self.match)
+ return best_index
+
+class searcher_re (object):
+
+ """This is regular expression string search helper for the
+ spawn.expect_any() method.
+
+ Attributes:
+
+ eof_index - index of EOF, or -1
+ timeout_index - index of TIMEOUT, or -1
+
+ After a successful match by the search() method the following attributes
+ are available:
+
+ start - index into the buffer, first byte of match
+ end - index into the buffer, first byte after match
+ match - the re.match object returned by a succesful re.search
+
+ """
+
+ def __init__(self, patterns):
+
+ """This creates an instance that searches for 'patterns' Where
+ 'patterns' may be a list or other sequence of compiled regular
+ expressions, or the EOF or TIMEOUT types."""
+
+ self.eof_index = -1
+ self.timeout_index = -1
+ self._searches = []
+ for n, s in zip(range(len(patterns)), patterns):
+ if s is EOF:
+ self.eof_index = n
+ continue
+ if s is TIMEOUT:
+ self.timeout_index = n
+ continue
+ self._searches.append((n, s))
+
+ def __str__(self):
+
+ """This returns a human-readable string that represents the state of
+ the object."""
+
+ ss = [ (n,' %d: re.compile("%s")' % (n,str(s.pattern))) for n,s in self._searches]
+ ss.append((-1,'searcher_re:'))
+ if self.eof_index >= 0:
+ ss.append ((self.eof_index,' %d: EOF' % self.eof_index))
+ if self.timeout_index >= 0:
+ ss.append ((self.timeout_index,' %d: TIMEOUT' % self.timeout_index))
+ ss.sort()
+ ss = zip(*ss)[1]
+ return '\n'.join(ss)
+
+ def search(self, buffer, freshlen, searchwindowsize=None):
+
+ """This searches 'buffer' for the first occurence of one of the regular
+ expressions. 'freshlen' must indicate the number of bytes at the end of
+ 'buffer' which have not been searched before.
+
+ See class spawn for the 'searchwindowsize' argument.
+
+ If there is a match this returns the index of that string, and sets
+ 'start', 'end' and 'match'. Otherwise, returns -1."""
+
+ absurd_match = len(buffer)
+ first_match = absurd_match
+ # 'freshlen' doesn't help here -- we cannot predict the
+ # length of a match, and the re module provides no help.
+ if searchwindowsize is None:
+ searchstart = 0
+ else:
+ searchstart = max(0, len(buffer)-searchwindowsize)
+ for index, s in self._searches:
+ match = s.search(buffer, searchstart)
+ if match is None:
+ continue
+ n = match.start()
+ if n < first_match:
+ first_match = n
+ the_match = match
+ best_index = index
+ if first_match == absurd_match:
+ return -1
+ self.start = first_match
+ self.match = the_match
+ self.end = self.match.end()
+ return best_index
+
+def which (filename):
+
+ """This takes a given filename; tries to find it in the environment path;
+ then checks if it is executable. This returns the full path to the filename
+ if found and executable. Otherwise this returns None."""
+
+ # Special case where filename already contains a path.
+ if os.path.dirname(filename) != '':
+ if os.access (filename, os.X_OK):
+ return filename
+
+ if not os.environ.has_key('PATH') or os.environ['PATH'] == '':
+ p = os.defpath
+ else:
+ p = os.environ['PATH']
+
+ # Oddly enough this was the one line that made Pexpect
+ # incompatible with Python 1.5.2.
+ #pathlist = p.split (os.pathsep)
+ pathlist = string.split (p, os.pathsep)
+
+ for path in pathlist:
+ f = os.path.join(path, filename)
+ if os.access(f, os.X_OK):
+ return f
+ return None
+
+def split_command_line(command_line):
+
+ """This splits a command line into a list of arguments. It splits arguments
+ on spaces, but handles embedded quotes, doublequotes, and escaped
+ characters. It's impossible to do this with a regular expression, so I
+ wrote a little state machine to parse the command line. """
+
+ arg_list = []
+ arg = ''
+
+ # Constants to name the states we can be in.
+ state_basic = 0
+ state_esc = 1
+ state_singlequote = 2
+ state_doublequote = 3
+ state_whitespace = 4 # The state of consuming whitespace between commands.
+ state = state_basic
+
+ for c in command_line:
+ if state == state_basic or state == state_whitespace:
+ if c == '\\': # Escape the next character
+ state = state_esc
+ elif c == r"'": # Handle single quote
+ state = state_singlequote
+ elif c == r'"': # Handle double quote
+ state = state_doublequote
+ elif c.isspace():
+ # Add arg to arg_list if we aren't in the middle of whitespace.
+ if state == state_whitespace:
+ None # Do nothing.
+ else:
+ arg_list.append(arg)
+ arg = ''
+ state = state_whitespace
+ else:
+ arg = arg + c
+ state = state_basic
+ elif state == state_esc:
+ arg = arg + c
+ state = state_basic
+ elif state == state_singlequote:
+ if c == r"'":
+ state = state_basic
+ else:
+ arg = arg + c
+ elif state == state_doublequote:
+ if c == r'"':
+ state = state_basic
+ else:
+ arg = arg + c
+
+ if arg != '':
+ arg_list.append(arg)
+ return arg_list
+
+# vi:ts=4:sw=4:expandtab:ft=python:
=== modified file 'duplicity/progress.py'
--- duplicity/progress.py 2014-04-20 14:02:34 +0000
+++ duplicity/progress.py 2014-10-15 12:12:04 +0000
@@ -32,9 +32,6 @@
This is a forecast based on gathered evidence.
"""
-from __future__ import absolute_import
-
-import collections as sys_collections
import math
import threading
import time
@@ -44,6 +41,29 @@
import pickle
import os
+def import_non_local(name, custom_name=None):
+ """
+ This function is needed to play a trick... as there exists a local
+ "collections" module, that is named the same as a system module
+ """
+ import imp, sys
+
+ custom_name = custom_name or name
+
+ f, pathname, desc = imp.find_module(name, sys.path[1:])
+ module = imp.load_module(custom_name, f, pathname, desc)
+ f.close()
+
+ return module
+
+"""
+Import non-local module, use a custom name to differentiate it from local
+This name is only used internally for identifying the module. We decide
+the name in the local scope by assigning it to the variable sys_collections.
+"""
+sys_collections = import_non_local('collections','sys_collections')
+
+
tracker = None
progress_thread = None
@@ -103,7 +123,7 @@
-class ProgressTracker():
+class ProgressTracker:
def __init__(self):
self.total_stats = None
@@ -242,7 +262,7 @@
projection = 1.0
if self.progress_estimation > 0:
projection = (1.0 - self.progress_estimation) / self.progress_estimation
- self.time_estimation = int(projection * float(self.elapsed_sum.total_seconds()))
+ self.time_estimation = long(projection * float(self.elapsed_sum.total_seconds()))
# Apply values only when monotonic, so the estimates look more consistent to the human eye
if self.progress_estimation < last_progress_estimation:
@@ -277,7 +297,7 @@
volume and for the current volume
"""
changing = max(bytecount - self.last_bytecount, 0)
- self.total_bytecount += int(changing) # Annotate only changing bytes since last probe
+ self.total_bytecount += long(changing) # Annotate only changing bytes since last probe
self.last_bytecount = bytecount
if changing > 0:
self.stall_last_time = datetime.now()
=== modified file 'duplicity/robust.py'
--- duplicity/robust.py 2014-04-17 20:50:57 +0000
+++ duplicity/robust.py 2014-10-15 12:12:04 +0000
@@ -39,7 +39,7 @@
# RPathException, Rdiff.RdiffException,
# librsync.librsyncError, C.UnknownFileTypeError), exc:
# TracebackArchive.add()
- except (IOError, EnvironmentError, librsync.librsyncError, path.PathException) as exc:
+ except (IOError, EnvironmentError, librsync.librsyncError, path.PathException), exc:
if (not isinstance(exc, EnvironmentError) or
((exc[0] in errno.errorcode)
and errno.errorcode[exc[0]] in
=== modified file 'duplicity/selection.py'
--- duplicity/selection.py 2014-04-25 23:53:46 +0000
+++ duplicity/selection.py 2014-10-15 12:12:04 +0000
@@ -19,8 +19,6 @@
# along with duplicity; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-from future_builtins import filter, map
-
import os #@UnusedImport
import re #@UnusedImport
import stat #@UnusedImport
@@ -237,8 +235,8 @@
filelists[filelists_index], 0, arg))
filelists_index += 1
elif opt == "--exclude-globbing-filelist":
- for sf in self.filelist_globbing_get_sfs(filelists[filelists_index], 0, arg):
- self.add_selection_func(sf)
+ map(self.add_selection_func,
+ self.filelist_globbing_get_sfs(filelists[filelists_index], 0, arg))
filelists_index += 1
elif opt == "--exclude-other-filesystems":
self.add_selection_func(self.other_filesystems_get_sf(0))
@@ -251,14 +249,14 @@
filelists[filelists_index], 1, arg))
filelists_index += 1
elif opt == "--include-globbing-filelist":
- for sf in self.filelist_globbing_get_sfs(filelists[filelists_index], 1, arg):
- self.add_selection_func(sf)
+ map(self.add_selection_func,
+ self.filelist_globbing_get_sfs(filelists[filelists_index], 1, arg))
filelists_index += 1
elif opt == "--include-regexp":
self.add_selection_func(self.regexp_get_sf(arg, 1))
else:
assert 0, "Bad selection option %s" % opt
- except SelectError as e:
+ except SelectError, e:
self.parse_catch_error(e)
assert filelists_index == len(filelists)
self.parse_last_excludes()
@@ -353,7 +351,7 @@
continue # skip blanks
try:
tuple = self.filelist_parse_line(line, include)
- except FilePrefixError as exc:
+ except FilePrefixError, exc:
incr_warnings(exc)
continue
tuple_list.append(tuple)
@@ -628,7 +626,8 @@
raise GlobbingError("Consecutive '/'s found in globbing string "
+ glob_str)
- prefixes = ["/".join(glob_parts[:i+1]) for i in range(len(glob_parts))]
+ prefixes = map(lambda i: "/".join(glob_parts[:i+1]),
+ range(len(glob_parts)))
# we must make exception for root "/", only dir to end in slash
if prefixes[0] == "":
prefixes[0] = "/"
=== added file 'duplicity/static.py'
--- duplicity/static.py 1970-01-01 00:00:00 +0000
+++ duplicity/static.py 2014-10-15 12:12:04 +0000
@@ -0,0 +1,46 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2002 Ben Escoto <ben@xxxxxxxxxxx>
+# Copyright 2007 Kenneth Loafman <kenneth@xxxxxxxxxxx>
+#
+# This file is part of duplicity.
+#
+# Duplicity is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# Duplicity is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+"""MakeStatic and MakeClass
+
+These functions are used to make all the instance methods in a class
+into static or class methods.
+
+"""
+
+class StaticMethodsError(Exception): pass
+
+def MakeStatic(cls):
+ """turn instance methods into static ones
+
+ The methods (that don't begin with _) of any class that
+ subclasses this will be turned into static methods.
+
+ """
+ for name in dir(cls):
+ if name[0] != "_":
+ cls.__dict__[name] = staticmethod(cls.__dict__[name])
+
+def MakeClass(cls):
+ """Turn instance methods into classmethods. Ignore _ like above"""
+ for name in dir(cls):
+ if name[0] != "_":
+ cls.__dict__[name] = classmethod(cls.__dict__[name])
=== modified file 'duplicity/statistics.py'
--- duplicity/statistics.py 2014-04-25 23:53:46 +0000
+++ duplicity/statistics.py 2014-10-15 12:12:04 +0000
@@ -21,8 +21,6 @@
"""Generate and process backup statistics"""
-from future_builtins import map
-
import re, time, os
from duplicity import dup_time
@@ -101,11 +99,12 @@
def get_stats_line(self, index, use_repr = 1):
"""Return one line abbreviated version of full stats string"""
- file_attrs = [str(self.get_stat(a)) for a in self.stat_file_attrs]
+ file_attrs = map(lambda attr: str(self.get_stat(attr)),
+ self.stat_file_attrs)
if not index:
filename = "."
else:
- filename = os.path.join(*index)
+ filename = apply(os.path.join, index)
if use_repr:
# use repr to quote newlines in relative filename, then
# take of leading and trailing quote and quote spaces.
@@ -124,7 +123,7 @@
for attr, val_string in zip(self.stat_file_attrs,
lineparts[-len(self.stat_file_attrs):]):
try:
- val = int(val_string)
+ val = long(val_string)
except ValueError:
try:
val = float(val_string)
@@ -231,7 +230,7 @@
error(line)
try:
try:
- val1 = int(value_string)
+ val1 = long(value_string)
except ValueError:
val1 = None
val2 = float(value_string)
=== modified file 'duplicity/tarfile.py'
--- duplicity/tarfile.py 2014-04-16 20:45:09 +0000
+++ duplicity/tarfile.py 2014-10-15 12:12:04 +0000
@@ -1,35 +1,2594 @@
-# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
-#
-# Copyright 2013 Michael Terry <mike@xxxxxxxxxxx>
-#
-# This file is part of duplicity.
-#
-# Duplicity is free software; you can redistribute it and/or modify it
-# under the terms of the GNU General Public License as published by the
-# Free Software Foundation; either version 2 of the License, or (at your
-# option) any later version.
-#
-# Duplicity is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with duplicity; if not, write to the Free Software Foundation,
-# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-"""Like system tarfile but with caching."""
-
-from __future__ import absolute_import
-
-import tarfile
-
-# Grab all symbols in tarfile, to try to reproduce its API exactly.
-# from <> import * wouldn't get everything we want, since tarfile defines
-# __all__. So we do it ourselves.
-for sym in dir(tarfile):
- globals()[sym] = getattr(tarfile, sym)
-
-# Now make sure that we cache the grp/pwd ops
+#! /usr/bin/python2.7
+# -*- coding: iso-8859-1 -*-
+#-------------------------------------------------------------------
+# tarfile.py
+#-------------------------------------------------------------------
+# Copyright (C) 2002 Lars Gust�l <lars@xxxxxxxxxxxx>
+# All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation
+# files (the "Software"), to deal in the Software without
+# restriction, including without limitation the rights to use,
+# copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following
+# conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+"""Read from and write to tar format archives.
+"""
+
+__version__ = "$Revision: 85213 $"
+# $Source$
+
+version = "0.9.0"
+__author__ = "Lars Gust�l (lars@xxxxxxxxxxxx)"
+__date__ = "$Date: 2010-10-04 10:37:53 -0500 (Mon, 04 Oct 2010) $"
+__cvsid__ = "$Id: tarfile.py 85213 2010-10-04 15:37:53Z lars.gustaebel $"
+__credits__ = "Gustavo Niemeyer, Niels Gust�l, Richard Townsend."
+
+#---------
+# Imports
+#---------
+import sys
+import os
+import shutil
+import stat
+import errno
+import time
+import struct
+import copy
+import re
+import operator
+
from duplicity import cached_ops
grp = pwd = cached_ops
+
+# from tarfile import *
+__all__ = ["TarFile", "TarInfo", "is_tarfile", "TarError"]
+
+#---------------------------------------------------------
+# tar constants
+#---------------------------------------------------------
+NUL = "\0" # the null character
+BLOCKSIZE = 512 # length of processing blocks
+RECORDSIZE = BLOCKSIZE * 20 # length of records
+GNU_MAGIC = "ustar \0" # magic gnu tar string
+POSIX_MAGIC = "ustar\x0000" # magic posix tar string
+
+LENGTH_NAME = 100 # maximum length of a filename
+LENGTH_LINK = 100 # maximum length of a linkname
+LENGTH_PREFIX = 155 # maximum length of the prefix field
+
+REGTYPE = "0" # regular file
+AREGTYPE = "\0" # regular file
+LNKTYPE = "1" # link (inside tarfile)
+SYMTYPE = "2" # symbolic link
+CHRTYPE = "3" # character special device
+BLKTYPE = "4" # block special device
+DIRTYPE = "5" # directory
+FIFOTYPE = "6" # fifo special device
+CONTTYPE = "7" # contiguous file
+
+GNUTYPE_LONGNAME = "L" # GNU tar longname
+GNUTYPE_LONGLINK = "K" # GNU tar longlink
+GNUTYPE_SPARSE = "S" # GNU tar sparse file
+
+XHDTYPE = "x" # POSIX.1-2001 extended header
+XGLTYPE = "g" # POSIX.1-2001 global header
+SOLARIS_XHDTYPE = "X" # Solaris extended header
+
+USTAR_FORMAT = 0 # POSIX.1-1988 (ustar) format
+GNU_FORMAT = 1 # GNU tar format
+PAX_FORMAT = 2 # POSIX.1-2001 (pax) format
+DEFAULT_FORMAT = GNU_FORMAT
+
+#---------------------------------------------------------
+# tarfile constants
+#---------------------------------------------------------
+# File types that tarfile supports:
+SUPPORTED_TYPES = (REGTYPE, AREGTYPE, LNKTYPE,
+ SYMTYPE, DIRTYPE, FIFOTYPE,
+ CONTTYPE, CHRTYPE, BLKTYPE,
+ GNUTYPE_LONGNAME, GNUTYPE_LONGLINK,
+ GNUTYPE_SPARSE)
+
+# File types that will be treated as a regular file.
+REGULAR_TYPES = (REGTYPE, AREGTYPE,
+ CONTTYPE, GNUTYPE_SPARSE)
+
+# File types that are part of the GNU tar format.
+GNU_TYPES = (GNUTYPE_LONGNAME, GNUTYPE_LONGLINK,
+ GNUTYPE_SPARSE)
+
+# Fields from a pax header that override a TarInfo attribute.
+PAX_FIELDS = ("path", "linkpath", "size", "mtime",
+ "uid", "gid", "uname", "gname")
+
+# Fields in a pax header that are numbers, all other fields
+# are treated as strings.
+PAX_NUMBER_FIELDS = {
+ "atime": float,
+ "ctime": float,
+ "mtime": float,
+ "uid": int,
+ "gid": int,
+ "size": int
+}
+
+#---------------------------------------------------------
+# Bits used in the mode field, values in octal.
+#---------------------------------------------------------
+S_IFLNK = 0120000 # symbolic link
+S_IFREG = 0100000 # regular file
+S_IFBLK = 0060000 # block device
+S_IFDIR = 0040000 # directory
+S_IFCHR = 0020000 # character device
+S_IFIFO = 0010000 # fifo
+
+TSUID = 04000 # set UID on execution
+TSGID = 02000 # set GID on execution
+TSVTX = 01000 # reserved
+
+TUREAD = 0400 # read by owner
+TUWRITE = 0200 # write by owner
+TUEXEC = 0100 # execute/search by owner
+TGREAD = 0040 # read by group
+TGWRITE = 0020 # write by group
+TGEXEC = 0010 # execute/search by group
+TOREAD = 0004 # read by other
+TOWRITE = 0002 # write by other
+TOEXEC = 0001 # execute/search by other
+
+#---------------------------------------------------------
+# initialization
+#---------------------------------------------------------
+ENCODING = sys.getfilesystemencoding()
+if ENCODING is None:
+ ENCODING = sys.getdefaultencoding()
+
+#---------------------------------------------------------
+# Some useful functions
+#---------------------------------------------------------
+
+def stn(s, length):
+ """Convert a python string to a null-terminated string buffer.
+ """
+ return s[:length] + (length - len(s)) * NUL
+
+def nts(s):
+ """Convert a null-terminated string field to a python string.
+ """
+ # Use the string up to the first null char.
+ p = s.find("\0")
+ if p == -1:
+ return s
+ return s[:p]
+
+def nti(s):
+ """Convert a number field to a python number.
+ """
+ # There are two possible encodings for a number field, see
+ # itn() below.
+ if s[0] != chr(0200):
+ try:
+ n = int(nts(s) or "0", 8)
+ except ValueError:
+ raise InvalidHeaderError("invalid header")
+ else:
+ n = 0L
+ for i in xrange(len(s) - 1):
+ n <<= 8
+ n += ord(s[i + 1])
+ return n
+
+def itn(n, digits=8, format=DEFAULT_FORMAT):
+ """Convert a python number to a number field.
+ """
+ # POSIX 1003.1-1988 requires numbers to be encoded as a string of
+ # octal digits followed by a null-byte, this allows values up to
+ # (8**(digits-1))-1. GNU tar allows storing numbers greater than
+ # that if necessary. A leading 0200 byte indicates this particular
+ # encoding, the following digits-1 bytes are a big-endian
+ # representation. This allows values up to (256**(digits-1))-1.
+ if 0 <= n < 8 ** (digits - 1):
+ s = "%0*o" % (digits - 1, n) + NUL
+ else:
+ if format != GNU_FORMAT or n >= 256 ** (digits - 1):
+ raise ValueError("overflow in number field")
+
+ if n < 0:
+ # XXX We mimic GNU tar's behaviour with negative numbers,
+ # this could raise OverflowError.
+ n = struct.unpack("L", struct.pack("l", n))[0]
+
+ s = ""
+ for i in xrange(digits - 1):
+ s = chr(n & 0377) + s
+ n >>= 8
+ s = chr(0200) + s
+ return s
+
+def uts(s, encoding, errors):
+ """Convert a unicode object to a string.
+ """
+ if errors == "utf-8":
+ # An extra error handler similar to the -o invalid=UTF-8 option
+ # in POSIX.1-2001. Replace untranslatable characters with their
+ # UTF-8 representation.
+ try:
+ return s.encode(encoding, "strict")
+ except UnicodeEncodeError:
+ x = []
+ for c in s:
+ try:
+ x.append(c.encode(encoding, "strict"))
+ except UnicodeEncodeError:
+ x.append(c.encode("utf8"))
+ return "".join(x)
+ else:
+ return s.encode(encoding, errors)
+
+def calc_chksums(buf):
+ """Calculate the checksum for a member's header by summing up all
+ characters except for the chksum field which is treated as if
+ it was filled with spaces. According to the GNU tar sources,
+ some tars (Sun and NeXT) calculate chksum with signed char,
+ which will be different if there are chars in the buffer with
+ the high bit set. So we calculate two checksums, unsigned and
+ signed.
+ """
+ unsigned_chksum = 256 + sum(struct.unpack("148B", buf[:148]) + struct.unpack("356B", buf[156:512]))
+ signed_chksum = 256 + sum(struct.unpack("148b", buf[:148]) + struct.unpack("356b", buf[156:512]))
+ return unsigned_chksum, signed_chksum
+
+def copyfileobj(src, dst, length=None):
+ """Copy length bytes from fileobj src to fileobj dst.
+ If length is None, copy the entire content.
+ """
+ if length == 0:
+ return
+ if length is None:
+ shutil.copyfileobj(src, dst)
+ return
+
+ BUFSIZE = 16 * 1024
+ blocks, remainder = divmod(length, BUFSIZE)
+ for b in xrange(blocks):
+ buf = src.read(BUFSIZE)
+ if len(buf) < BUFSIZE:
+ raise IOError("end of file reached")
+ dst.write(buf)
+
+ if remainder != 0:
+ buf = src.read(remainder)
+ if len(buf) < remainder:
+ raise IOError("end of file reached")
+ dst.write(buf)
+ return
+
+filemode_table = (
+ ((S_IFLNK, "l"),
+ (S_IFREG, "-"),
+ (S_IFBLK, "b"),
+ (S_IFDIR, "d"),
+ (S_IFCHR, "c"),
+ (S_IFIFO, "p")),
+
+ ((TUREAD, "r"),),
+ ((TUWRITE, "w"),),
+ ((TUEXEC|TSUID, "s"),
+ (TSUID, "S"),
+ (TUEXEC, "x")),
+
+ ((TGREAD, "r"),),
+ ((TGWRITE, "w"),),
+ ((TGEXEC|TSGID, "s"),
+ (TSGID, "S"),
+ (TGEXEC, "x")),
+
+ ((TOREAD, "r"),),
+ ((TOWRITE, "w"),),
+ ((TOEXEC|TSVTX, "t"),
+ (TSVTX, "T"),
+ (TOEXEC, "x"))
+)
+
+def filemode(mode):
+ """Convert a file's mode to a string of the form
+ -rwxrwxrwx.
+ Used by TarFile.list()
+ """
+ perm = []
+ for table in filemode_table:
+ for bit, char in table:
+ if mode & bit == bit:
+ perm.append(char)
+ break
+ else:
+ perm.append("-")
+ return "".join(perm)
+
+class TarError(Exception):
+ """Base exception."""
+ pass
+class ExtractError(TarError):
+ """General exception for extract errors."""
+ pass
+class ReadError(TarError):
+ """Exception for unreadble tar archives."""
+ pass
+class CompressionError(TarError):
+ """Exception for unavailable compression methods."""
+ pass
+class StreamError(TarError):
+ """Exception for unsupported operations on stream-like TarFiles."""
+ pass
+class HeaderError(TarError):
+ """Base exception for header errors."""
+ pass
+class EmptyHeaderError(HeaderError):
+ """Exception for empty headers."""
+ pass
+class TruncatedHeaderError(HeaderError):
+ """Exception for truncated headers."""
+ pass
+class EOFHeaderError(HeaderError):
+ """Exception for end of file headers."""
+ pass
+class InvalidHeaderError(HeaderError):
+ """Exception for invalid headers."""
+ pass
+class SubsequentHeaderError(HeaderError):
+ """Exception for missing and invalid extended headers."""
+ pass
+
+#---------------------------
+# internal stream interface
+#---------------------------
+class _LowLevelFile:
+ """Low-level file object. Supports reading and writing.
+ It is used instead of a regular file object for streaming
+ access.
+ """
+
+ def __init__(self, name, mode):
+ mode = {
+ "r": os.O_RDONLY,
+ "w": os.O_WRONLY | os.O_CREAT | os.O_TRUNC,
+ }[mode]
+ if hasattr(os, "O_BINARY"):
+ mode |= os.O_BINARY
+ self.fd = os.open(name, mode, 0666)
+
+ def close(self):
+ os.close(self.fd)
+
+ def read(self, size):
+ return os.read(self.fd, size)
+
+ def write(self, s):
+ os.write(self.fd, s)
+
+class _Stream:
+ """Class that serves as an adapter between TarFile and
+ a stream-like object. The stream-like object only
+ needs to have a read() or write() method and is accessed
+ blockwise. Use of gzip or bzip2 compression is possible.
+ A stream-like object could be for example: sys.stdin,
+ sys.stdout, a socket, a tape device etc.
+
+ _Stream is intended to be used only internally.
+ """
+
+ def __init__(self, name, mode, comptype, fileobj, bufsize):
+ """Construct a _Stream object.
+ """
+ self._extfileobj = True
+ if fileobj is None:
+ fileobj = _LowLevelFile(name, mode)
+ self._extfileobj = False
+
+ if comptype == '*':
+ # Enable transparent compression detection for the
+ # stream interface
+ fileobj = _StreamProxy(fileobj)
+ comptype = fileobj.getcomptype()
+
+ self.name = name or ""
+ self.mode = mode
+ self.comptype = comptype
+ self.fileobj = fileobj
+ self.bufsize = bufsize
+ self.buf = ""
+ self.pos = 0L
+ self.closed = False
+
+ if comptype == "gz":
+ try:
+ import zlib
+ except ImportError:
+ raise CompressionError("zlib module is not available")
+ self.zlib = zlib
+ self.crc = zlib.crc32("") & 0xffffffffL
+ if mode == "r":
+ self._init_read_gz()
+ else:
+ self._init_write_gz()
+
+ if comptype == "bz2":
+ try:
+ import bz2
+ except ImportError:
+ raise CompressionError("bz2 module is not available")
+ if mode == "r":
+ self.dbuf = ""
+ self.cmp = bz2.BZ2Decompressor()
+ else:
+ self.cmp = bz2.BZ2Compressor()
+
+ def __del__(self):
+ if hasattr(self, "closed") and not self.closed:
+ self.close()
+
+ def _init_write_gz(self):
+ """Initialize for writing with gzip compression.
+ """
+ self.cmp = self.zlib.compressobj(9, self.zlib.DEFLATED,
+ -self.zlib.MAX_WBITS,
+ self.zlib.DEF_MEM_LEVEL,
+ 0)
+ timestamp = struct.pack("<L", long(time.time()))
+ self.__write("\037\213\010\010%s\002\377" % timestamp)
+ if self.name.endswith(".gz"):
+ self.name = self.name[:-3]
+ self.__write(self.name + NUL)
+
+ def write(self, s):
+ """Write string s to the stream.
+ """
+ if self.comptype == "gz":
+ self.crc = self.zlib.crc32(s, self.crc) & 0xffffffffL
+ self.pos += len(s)
+ if self.comptype != "tar":
+ s = self.cmp.compress(s)
+ self.__write(s)
+
+ def __write(self, s):
+ """Write string s to the stream if a whole new block
+ is ready to be written.
+ """
+ self.buf += s
+ while len(self.buf) > self.bufsize:
+ self.fileobj.write(self.buf[:self.bufsize])
+ self.buf = self.buf[self.bufsize:]
+
+ def close(self):
+ """Close the _Stream object. No operation should be
+ done on it afterwards.
+ """
+ if self.closed:
+ return
+
+ if self.mode == "w" and self.comptype != "tar":
+ self.buf += self.cmp.flush()
+
+ if self.mode == "w" and self.buf:
+ self.fileobj.write(self.buf)
+ self.buf = ""
+ if self.comptype == "gz":
+ # The native zlib crc is an unsigned 32-bit integer, but
+ # the Python wrapper implicitly casts that to a signed C
+ # long. So, on a 32-bit box self.crc may "look negative",
+ # while the same crc on a 64-bit box may "look positive".
+ # To avoid irksome warnings from the `struct` module, force
+ # it to look positive on all boxes.
+ self.fileobj.write(struct.pack("<L", self.crc & 0xffffffffL))
+ self.fileobj.write(struct.pack("<L", self.pos & 0xffffFFFFL))
+
+ if not self._extfileobj:
+ self.fileobj.close()
+
+ self.closed = True
+
+ def _init_read_gz(self):
+ """Initialize for reading a gzip compressed fileobj.
+ """
+ self.cmp = self.zlib.decompressobj(-self.zlib.MAX_WBITS)
+ self.dbuf = ""
+
+ # taken from gzip.GzipFile with some alterations
+ if self.__read(2) != "\037\213":
+ raise ReadError("not a gzip file")
+ if self.__read(1) != "\010":
+ raise CompressionError("unsupported compression method")
+
+ flag = ord(self.__read(1))
+ self.__read(6)
+
+ if flag & 4:
+ xlen = ord(self.__read(1)) + 256 * ord(self.__read(1))
+ self.read(xlen)
+ if flag & 8:
+ while True:
+ s = self.__read(1)
+ if not s or s == NUL:
+ break
+ if flag & 16:
+ while True:
+ s = self.__read(1)
+ if not s or s == NUL:
+ break
+ if flag & 2:
+ self.__read(2)
+
+ def tell(self):
+ """Return the stream's file pointer position.
+ """
+ return self.pos
+
+ def seek(self, pos=0):
+ """Set the stream's file pointer to pos. Negative seeking
+ is forbidden.
+ """
+ if pos - self.pos >= 0:
+ blocks, remainder = divmod(pos - self.pos, self.bufsize)
+ for i in xrange(blocks):
+ self.read(self.bufsize)
+ self.read(remainder)
+ else:
+ raise StreamError("seeking backwards is not allowed")
+ return self.pos
+
+ def read(self, size=None):
+ """Return the next size number of bytes from the stream.
+ If size is not defined, return all bytes of the stream
+ up to EOF.
+ """
+ if size is None:
+ t = []
+ while True:
+ buf = self._read(self.bufsize)
+ if not buf:
+ break
+ t.append(buf)
+ buf = "".join(t)
+ else:
+ buf = self._read(size)
+ self.pos += len(buf)
+ return buf
+
+ def _read(self, size):
+ """Return size bytes from the stream.
+ """
+ if self.comptype == "tar":
+ return self.__read(size)
+
+ c = len(self.dbuf)
+ t = [self.dbuf]
+ while c < size:
+ buf = self.__read(self.bufsize)
+ if not buf:
+ break
+ try:
+ buf = self.cmp.decompress(buf)
+ except IOError:
+ raise ReadError("invalid compressed data")
+ t.append(buf)
+ c += len(buf)
+ t = "".join(t)
+ self.dbuf = t[size:]
+ return t[:size]
+
+ def __read(self, size):
+ """Return size bytes from stream. If internal buffer is empty,
+ read another block from the stream.
+ """
+ c = len(self.buf)
+ t = [self.buf]
+ while c < size:
+ buf = self.fileobj.read(self.bufsize)
+ if not buf:
+ break
+ t.append(buf)
+ c += len(buf)
+ t = "".join(t)
+ self.buf = t[size:]
+ return t[:size]
+# class _Stream
+
+class _StreamProxy(object):
+ """Small proxy class that enables transparent compression
+ detection for the Stream interface (mode 'r|*').
+ """
+
+ def __init__(self, fileobj):
+ self.fileobj = fileobj
+ self.buf = self.fileobj.read(BLOCKSIZE)
+
+ def read(self, size):
+ self.read = self.fileobj.read
+ return self.buf
+
+ def getcomptype(self):
+ if self.buf.startswith("\037\213\010"):
+ return "gz"
+ if self.buf.startswith("BZh91"):
+ return "bz2"
+ return "tar"
+
+ def close(self):
+ self.fileobj.close()
+# class StreamProxy
+
+class _BZ2Proxy(object):
+ """Small proxy class that enables external file object
+ support for "r:bz2" and "w:bz2" modes. This is actually
+ a workaround for a limitation in bz2 module's BZ2File
+ class which (unlike gzip.GzipFile) has no support for
+ a file object argument.
+ """
+
+ blocksize = 16 * 1024
+
+ def __init__(self, fileobj, mode):
+ self.fileobj = fileobj
+ self.mode = mode
+ self.name = getattr(self.fileobj, "name", None)
+ self.init()
+
+ def init(self):
+ import bz2
+ self.pos = 0
+ if self.mode == "r":
+ self.bz2obj = bz2.BZ2Decompressor()
+ self.fileobj.seek(0)
+ self.buf = ""
+ else:
+ self.bz2obj = bz2.BZ2Compressor()
+
+ def read(self, size):
+ b = [self.buf]
+ x = len(self.buf)
+ while x < size:
+ raw = self.fileobj.read(self.blocksize)
+ if not raw:
+ break
+ data = self.bz2obj.decompress(raw)
+ b.append(data)
+ x += len(data)
+ self.buf = "".join(b)
+
+ buf = self.buf[:size]
+ self.buf = self.buf[size:]
+ self.pos += len(buf)
+ return buf
+
+ def seek(self, pos):
+ if pos < self.pos:
+ self.init()
+ self.read(pos - self.pos)
+
+ def tell(self):
+ return self.pos
+
+ def write(self, data):
+ self.pos += len(data)
+ raw = self.bz2obj.compress(data)
+ self.fileobj.write(raw)
+
+ def close(self):
+ if self.mode == "w":
+ raw = self.bz2obj.flush()
+ self.fileobj.write(raw)
+# class _BZ2Proxy
+
+#------------------------
+# Extraction file object
+#------------------------
+class _FileInFile(object):
+ """A thin wrapper around an existing file object that
+ provides a part of its data as an individual file
+ object.
+ """
+
+ def __init__(self, fileobj, offset, size, sparse=None):
+ self.fileobj = fileobj
+ self.offset = offset
+ self.size = size
+ self.sparse = sparse
+ self.position = 0
+
+ def tell(self):
+ """Return the current file position.
+ """
+ return self.position
+
+ def seek(self, position):
+ """Seek to a position in the file.
+ """
+ self.position = position
+
+ def read(self, size=None):
+ """Read data from the file.
+ """
+ if size is None:
+ size = self.size - self.position
+ else:
+ size = min(size, self.size - self.position)
+
+ if self.sparse is None:
+ return self.readnormal(size)
+ else:
+ return self.readsparse(size)
+
+ def readnormal(self, size):
+ """Read operation for regular files.
+ """
+ self.fileobj.seek(self.offset + self.position)
+ self.position += size
+ return self.fileobj.read(size)
+
+ def readsparse(self, size):
+ """Read operation for sparse files.
+ """
+ data = []
+ while size > 0:
+ buf = self.readsparsesection(size)
+ if not buf:
+ break
+ size -= len(buf)
+ data.append(buf)
+ return "".join(data)
+
+ def readsparsesection(self, size):
+ """Read a single section of a sparse file.
+ """
+ section = self.sparse.find(self.position)
+
+ if section is None:
+ return ""
+
+ size = min(size, section.offset + section.size - self.position)
+
+ if isinstance(section, _data):
+ realpos = section.realpos + self.position - section.offset
+ self.fileobj.seek(self.offset + realpos)
+ self.position += size
+ return self.fileobj.read(size)
+ else:
+ self.position += size
+ return NUL * size
+#class _FileInFile
+
+
+class ExFileObject(object):
+ """File-like object for reading an archive member.
+ Is returned by TarFile.extractfile().
+ """
+ blocksize = 1024
+
+ def __init__(self, tarfile, tarinfo):
+ self.fileobj = _FileInFile(tarfile.fileobj,
+ tarinfo.offset_data,
+ tarinfo.size,
+ getattr(tarinfo, "sparse", None))
+ self.name = tarinfo.name
+ self.mode = "r"
+ self.closed = False
+ self.size = tarinfo.size
+
+ self.position = 0
+ self.buffer = ""
+
+ def read(self, size=None):
+ """Read at most size bytes from the file. If size is not
+ present or None, read all data until EOF is reached.
+ """
+ if self.closed:
+ raise ValueError("I/O operation on closed file")
+
+ buf = ""
+ if self.buffer:
+ if size is None:
+ buf = self.buffer
+ self.buffer = ""
+ else:
+ buf = self.buffer[:size]
+ self.buffer = self.buffer[size:]
+
+ if size is None:
+ buf += self.fileobj.read()
+ else:
+ buf += self.fileobj.read(size - len(buf))
+
+ self.position += len(buf)
+ return buf
+
+ def readline(self, size=-1):
+ """Read one entire line from the file. If size is present
+ and non-negative, return a string with at most that
+ size, which may be an incomplete line.
+ """
+ if self.closed:
+ raise ValueError("I/O operation on closed file")
+
+ if "\n" in self.buffer:
+ pos = self.buffer.find("\n") + 1
+ else:
+ buffers = [self.buffer]
+ while True:
+ buf = self.fileobj.read(self.blocksize)
+ buffers.append(buf)
+ if not buf or "\n" in buf:
+ self.buffer = "".join(buffers)
+ pos = self.buffer.find("\n") + 1
+ if pos == 0:
+ # no newline found.
+ pos = len(self.buffer)
+ break
+
+ if size != -1:
+ pos = min(size, pos)
+
+ buf = self.buffer[:pos]
+ self.buffer = self.buffer[pos:]
+ self.position += len(buf)
+ return buf
+
+ def readlines(self):
+ """Return a list with all remaining lines.
+ """
+ result = []
+ while True:
+ line = self.readline()
+ if not line: break
+ result.append(line)
+ return result
+
+ def tell(self):
+ """Return the current file position.
+ """
+ if self.closed:
+ raise ValueError("I/O operation on closed file")
+
+ return self.position
+
+ def seek(self, pos, whence=0):
+ """Seek to a position in the file.
+ """
+ if self.closed:
+ raise ValueError("I/O operation on closed file")
+
+ if whence == 0:
+ self.position = min(max(pos, 0), self.size)
+ elif whence == 1:
+ if pos < 0:
+ self.position = max(self.position + pos, 0)
+ else:
+ self.position = min(self.position + pos, self.size)
+ elif whence == 2:
+ self.position = max(min(self.size + pos, self.size), 0)
+ else:
+ raise ValueError("Invalid argument")
+
+ self.buffer = ""
+ self.fileobj.seek(self.position)
+
+ def close(self):
+ """Close the file object.
+ """
+ self.closed = True
+
+ def __iter__(self):
+ """Get an iterator over the file's lines.
+ """
+ while True:
+ line = self.readline()
+ if not line:
+ break
+ yield line
+#class ExFileObject
+
+#------------------
+# Exported Classes
+#------------------
+class TarInfo(object):
+ """Informational class which holds the details about an
+ archive member given by a tar header block.
+ TarInfo objects are returned by TarFile.getmember(),
+ TarFile.getmembers() and TarFile.gettarinfo() and are
+ usually created internally.
+ """
+
+ def __init__(self, name=""):
+ """Construct a TarInfo object. name is the optional name
+ of the member.
+ """
+ self.name = name # member name
+ self.mode = 0644 # file permissions
+ self.uid = 0 # user id
+ self.gid = 0 # group id
+ self.size = 0 # file size
+ self.mtime = 0 # modification time
+ self.chksum = 0 # header checksum
+ self.type = REGTYPE # member type
+ self.linkname = "" # link name
+ self.uname = "" # user name
+ self.gname = "" # group name
+ self.devmajor = 0 # device major number
+ self.devminor = 0 # device minor number
+
+ self.offset = 0 # the tar header starts here
+ self.offset_data = 0 # the file's data starts here
+
+ self.pax_headers = {} # pax header information
+
+ # In pax headers the "name" and "linkname" field are called
+ # "path" and "linkpath".
+ def _getpath(self):
+ return self.name
+ def _setpath(self, name):
+ self.name = name
+ path = property(_getpath, _setpath)
+
+ def _getlinkpath(self):
+ return self.linkname
+ def _setlinkpath(self, linkname):
+ self.linkname = linkname
+ linkpath = property(_getlinkpath, _setlinkpath)
+
+ def __repr__(self):
+ return "<%s %r at %#x>" % (self.__class__.__name__,self.name,id(self))
+
+ def get_info(self, encoding, errors):
+ """Return the TarInfo's attributes as a dictionary.
+ """
+ info = {
+ "name": self.name,
+ "mode": self.mode & 07777,
+ "uid": self.uid,
+ "gid": self.gid,
+ "size": self.size,
+ "mtime": self.mtime,
+ "chksum": self.chksum,
+ "type": self.type,
+ "linkname": self.linkname,
+ "uname": self.uname,
+ "gname": self.gname,
+ "devmajor": self.devmajor,
+ "devminor": self.devminor
+ }
+
+ if info["type"] == DIRTYPE and not info["name"].endswith("/"):
+ info["name"] += "/"
+
+ for key in ("name", "linkname", "uname", "gname"):
+ if type(info[key]) is unicode:
+ info[key] = info[key].encode(encoding, errors)
+
+ return info
+
+ def tobuf(self, format=DEFAULT_FORMAT, encoding=ENCODING, errors="strict"):
+ """Return a tar header as a string of 512 byte blocks.
+ """
+ info = self.get_info(encoding, errors)
+
+ if format == USTAR_FORMAT:
+ return self.create_ustar_header(info)
+ elif format == GNU_FORMAT:
+ return self.create_gnu_header(info)
+ elif format == PAX_FORMAT:
+ return self.create_pax_header(info, encoding, errors)
+ else:
+ raise ValueError("invalid format")
+
+ def create_ustar_header(self, info):
+ """Return the object as a ustar header block.
+ """
+ info["magic"] = POSIX_MAGIC
+
+ if len(info["linkname"]) > LENGTH_LINK:
+ raise ValueError("linkname is too long")
+
+ if len(info["name"]) > LENGTH_NAME:
+ info["prefix"], info["name"] = self._posix_split_name(info["name"])
+
+ return self._create_header(info, USTAR_FORMAT)
+
+ def create_gnu_header(self, info):
+ """Return the object as a GNU header block sequence.
+ """
+ info["magic"] = GNU_MAGIC
+
+ buf = ""
+ if len(info["linkname"]) > LENGTH_LINK:
+ buf += self._create_gnu_long_header(info["linkname"], GNUTYPE_LONGLINK)
+
+ if len(info["name"]) > LENGTH_NAME:
+ buf += self._create_gnu_long_header(info["name"], GNUTYPE_LONGNAME)
+
+ return buf + self._create_header(info, GNU_FORMAT)
+
+ def create_pax_header(self, info, encoding, errors):
+ """Return the object as a ustar header block. If it cannot be
+ represented this way, prepend a pax extended header sequence
+ with supplement information.
+ """
+ info["magic"] = POSIX_MAGIC
+ pax_headers = self.pax_headers.copy()
+
+ # Test string fields for values that exceed the field length or cannot
+ # be represented in ASCII encoding.
+ for name, hname, length in (
+ ("name", "path", LENGTH_NAME), ("linkname", "linkpath", LENGTH_LINK),
+ ("uname", "uname", 32), ("gname", "gname", 32)):
+
+ if hname in pax_headers:
+ # The pax header has priority.
+ continue
+
+ val = info[name].decode(encoding, errors)
+
+ # Try to encode the string as ASCII.
+ try:
+ val.encode("ascii")
+ except UnicodeEncodeError:
+ pax_headers[hname] = val
+ continue
+
+ if len(info[name]) > length:
+ pax_headers[hname] = val
+
+ # Test number fields for values that exceed the field limit or values
+ # that like to be stored as float.
+ for name, digits in (("uid", 8), ("gid", 8), ("size", 12), ("mtime", 12)):
+ if name in pax_headers:
+ # The pax header has priority. Avoid overflow.
+ info[name] = 0
+ continue
+
+ val = info[name]
+ if not 0 <= val < 8 ** (digits - 1) or isinstance(val, float):
+ pax_headers[name] = unicode(val)
+ info[name] = 0
+
+ # Create a pax extended header if necessary.
+ if pax_headers:
+ buf = self._create_pax_generic_header(pax_headers)
+ else:
+ buf = ""
+
+ return buf + self._create_header(info, USTAR_FORMAT)
+
+ @classmethod
+ def create_pax_global_header(cls, pax_headers):
+ """Return the object as a pax global header block sequence.
+ """
+ return cls._create_pax_generic_header(pax_headers, type=XGLTYPE)
+
+ def _posix_split_name(self, name):
+ """Split a name longer than 100 chars into a prefix
+ and a name part.
+ """
+ prefix = name[:LENGTH_PREFIX + 1]
+ while prefix and prefix[-1] != "/":
+ prefix = prefix[:-1]
+
+ name = name[len(prefix):]
+ prefix = prefix[:-1]
+
+ if not prefix or len(name) > LENGTH_NAME:
+ raise ValueError("name is too long")
+ return prefix, name
+
+ @staticmethod
+ def _create_header(info, format):
+ """Return a header block. info is a dictionary with file
+ information, format must be one of the *_FORMAT constants.
+ """
+ parts = [
+ stn(info.get("name", ""), 100),
+ itn(info.get("mode", 0) & 07777, 8, format),
+ itn(info.get("uid", 0), 8, format),
+ itn(info.get("gid", 0), 8, format),
+ itn(info.get("size", 0), 12, format),
+ itn(info.get("mtime", 0), 12, format),
+ " ", # checksum field
+ info.get("type", REGTYPE),
+ stn(info.get("linkname", ""), 100),
+ stn(info.get("magic", POSIX_MAGIC), 8),
+ stn(info.get("uname", ""), 32),
+ stn(info.get("gname", ""), 32),
+ itn(info.get("devmajor", 0), 8, format),
+ itn(info.get("devminor", 0), 8, format),
+ stn(info.get("prefix", ""), 155)
+ ]
+
+ buf = struct.pack("%ds" % BLOCKSIZE, "".join(parts))
+ chksum = calc_chksums(buf[-BLOCKSIZE:])[0]
+ buf = buf[:-364] + "%06o\0" % chksum + buf[-357:]
+ return buf
+
+ @staticmethod
+ def _create_payload(payload):
+ """Return the string payload filled with zero bytes
+ up to the next 512 byte border.
+ """
+ blocks, remainder = divmod(len(payload), BLOCKSIZE)
+ if remainder > 0:
+ payload += (BLOCKSIZE - remainder) * NUL
+ return payload
+
+ @classmethod
+ def _create_gnu_long_header(cls, name, type):
+ """Return a GNUTYPE_LONGNAME or GNUTYPE_LONGLINK sequence
+ for name.
+ """
+ name += NUL
+
+ info = {}
+ info["name"] = "././@LongLink"
+ info["type"] = type
+ info["size"] = len(name)
+ info["magic"] = GNU_MAGIC
+
+ # create extended header + name blocks.
+ return cls._create_header(info, USTAR_FORMAT) + \
+ cls._create_payload(name)
+
+ @classmethod
+ def _create_pax_generic_header(cls, pax_headers, type=XHDTYPE):
+ """Return a POSIX.1-2001 extended or global header sequence
+ that contains a list of keyword, value pairs. The values
+ must be unicode objects.
+ """
+ records = []
+ for keyword, value in pax_headers.iteritems():
+ keyword = keyword.encode("utf8")
+ value = value.encode("utf8")
+ l = len(keyword) + len(value) + 3 # ' ' + '=' + '\n'
+ n = p = 0
+ while True:
+ n = l + len(str(p))
+ if n == p:
+ break
+ p = n
+ records.append("%d %s=%s\n" % (p, keyword, value))
+ records = "".join(records)
+
+ # We use a hardcoded "././@PaxHeader" name like star does
+ # instead of the one that POSIX recommends.
+ info = {}
+ info["name"] = "././@PaxHeader"
+ info["type"] = type
+ info["size"] = len(records)
+ info["magic"] = POSIX_MAGIC
+
+ # Create pax header + record blocks.
+ return cls._create_header(info, USTAR_FORMAT) + \
+ cls._create_payload(records)
+
+ @classmethod
+ def frombuf(cls, buf):
+ """Construct a TarInfo object from a 512 byte string buffer.
+ """
+ if len(buf) == 0:
+ raise EmptyHeaderError("empty header")
+ if len(buf) != BLOCKSIZE:
+ raise TruncatedHeaderError("truncated header")
+ if buf.count(NUL) == BLOCKSIZE:
+ raise EOFHeaderError("end of file header")
+
+ chksum = nti(buf[148:156])
+ if chksum not in calc_chksums(buf):
+ raise InvalidHeaderError("bad checksum")
+
+ obj = cls()
+ obj.buf = buf
+ obj.name = nts(buf[0:100])
+ obj.mode = nti(buf[100:108])
+ obj.uid = nti(buf[108:116])
+ obj.gid = nti(buf[116:124])
+ obj.size = nti(buf[124:136])
+ obj.mtime = nti(buf[136:148])
+ obj.chksum = chksum
+ obj.type = buf[156:157]
+ obj.linkname = nts(buf[157:257])
+ obj.uname = nts(buf[265:297])
+ obj.gname = nts(buf[297:329])
+ obj.devmajor = nti(buf[329:337])
+ obj.devminor = nti(buf[337:345])
+ prefix = nts(buf[345:500])
+
+ # Old V7 tar format represents a directory as a regular
+ # file with a trailing slash.
+ if obj.type == AREGTYPE and obj.name.endswith("/"):
+ obj.type = DIRTYPE
+
+ # Remove redundant slashes from directories.
+ if obj.isdir():
+ obj.name = obj.name.rstrip("/")
+
+ # Reconstruct a ustar longname.
+ if prefix and obj.type not in GNU_TYPES:
+ obj.name = prefix + "/" + obj.name
+ return obj
+
+ @classmethod
+ def fromtarfile(cls, tarfile):
+ """Return the next TarInfo object from TarFile object
+ tarfile.
+ """
+ buf = tarfile.fileobj.read(BLOCKSIZE)
+ obj = cls.frombuf(buf)
+ obj.offset = tarfile.fileobj.tell() - BLOCKSIZE
+ return obj._proc_member(tarfile)
+
+ #--------------------------------------------------------------------------
+ # The following are methods that are called depending on the type of a
+ # member. The entry point is _proc_member() which can be overridden in a
+ # subclass to add custom _proc_*() methods. A _proc_*() method MUST
+ # implement the following
+ # operations:
+ # 1. Set self.offset_data to the position where the data blocks begin,
+ # if there is data that follows.
+ # 2. Set tarfile.offset to the position where the next member's header will
+ # begin.
+ # 3. Return self or another valid TarInfo object.
+ def _proc_member(self, tarfile):
+ """Choose the right processing method depending on
+ the type and call it.
+ """
+ if self.type in (GNUTYPE_LONGNAME, GNUTYPE_LONGLINK):
+ return self._proc_gnulong(tarfile)
+ elif self.type == GNUTYPE_SPARSE:
+ return self._proc_sparse(tarfile)
+ elif self.type in (XHDTYPE, XGLTYPE, SOLARIS_XHDTYPE):
+ return self._proc_pax(tarfile)
+ else:
+ return self._proc_builtin(tarfile)
+
+ def _proc_builtin(self, tarfile):
+ """Process a builtin type or an unknown type which
+ will be treated as a regular file.
+ """
+ self.offset_data = tarfile.fileobj.tell()
+ offset = self.offset_data
+ if self.isreg() or self.type not in SUPPORTED_TYPES:
+ # Skip the following data blocks.
+ offset += self._block(self.size)
+ tarfile.offset = offset
+
+ # Patch the TarInfo object with saved global
+ # header information.
+ self._apply_pax_info(tarfile.pax_headers, tarfile.encoding, tarfile.errors)
+
+ return self
+
+ def _proc_gnulong(self, tarfile):
+ """Process the blocks that hold a GNU longname
+ or longlink member.
+ """
+ buf = tarfile.fileobj.read(self._block(self.size))
+
+ # Fetch the next header and process it.
+ try:
+ next = self.fromtarfile(tarfile)
+ except HeaderError:
+ raise SubsequentHeaderError("missing or bad subsequent header")
+
+ # Patch the TarInfo object from the next header with
+ # the longname information.
+ next.offset = self.offset
+ if self.type == GNUTYPE_LONGNAME:
+ next.name = nts(buf)
+ elif self.type == GNUTYPE_LONGLINK:
+ next.linkname = nts(buf)
+
+ return next
+
+ def _proc_sparse(self, tarfile):
+ """Process a GNU sparse header plus extra headers.
+ """
+ buf = self.buf
+ sp = _ringbuffer()
+ pos = 386
+ lastpos = 0L
+ realpos = 0L
+ # There are 4 possible sparse structs in the
+ # first header.
+ for i in xrange(4):
+ try:
+ offset = nti(buf[pos:pos + 12])
+ numbytes = nti(buf[pos + 12:pos + 24])
+ except ValueError:
+ break
+ if offset > lastpos:
+ sp.append(_hole(lastpos, offset - lastpos))
+ sp.append(_data(offset, numbytes, realpos))
+ realpos += numbytes
+ lastpos = offset + numbytes
+ pos += 24
+
+ isextended = ord(buf[482])
+ origsize = nti(buf[483:495])
+
+ # If the isextended flag is given,
+ # there are extra headers to process.
+ while isextended == 1:
+ buf = tarfile.fileobj.read(BLOCKSIZE)
+ pos = 0
+ for i in xrange(21):
+ try:
+ offset = nti(buf[pos:pos + 12])
+ numbytes = nti(buf[pos + 12:pos + 24])
+ except ValueError:
+ break
+ if offset > lastpos:
+ sp.append(_hole(lastpos, offset - lastpos))
+ sp.append(_data(offset, numbytes, realpos))
+ realpos += numbytes
+ lastpos = offset + numbytes
+ pos += 24
+ isextended = ord(buf[504])
+
+ if lastpos < origsize:
+ sp.append(_hole(lastpos, origsize - lastpos))
+
+ self.sparse = sp
+
+ self.offset_data = tarfile.fileobj.tell()
+ tarfile.offset = self.offset_data + self._block(self.size)
+ self.size = origsize
+
+ return self
+
+ def _proc_pax(self, tarfile):
+ """Process an extended or global header as described in
+ POSIX.1-2001.
+ """
+ # Read the header information.
+ buf = tarfile.fileobj.read(self._block(self.size))
+
+ # A pax header stores supplemental information for either
+ # the following file (extended) or all following files
+ # (global).
+ if self.type == XGLTYPE:
+ pax_headers = tarfile.pax_headers
+ else:
+ pax_headers = tarfile.pax_headers.copy()
+
+ # Parse pax header information. A record looks like that:
+ # "%d %s=%s\n" % (length, keyword, value). length is the size
+ # of the complete record including the length field itself and
+ # the newline. keyword and value are both UTF-8 encoded strings.
+ regex = re.compile(r"(\d+) ([^=]+)=", re.U)
+ pos = 0
+ while True:
+ match = regex.match(buf, pos)
+ if not match:
+ break
+
+ length, keyword = match.groups()
+ length = int(length)
+ value = buf[match.end(2) + 1:match.start(1) + length - 1]
+
+ keyword = keyword.decode("utf8")
+ value = value.decode("utf8")
+
+ pax_headers[keyword] = value
+ pos += length
+
+ # Fetch the next header.
+ try:
+ next = self.fromtarfile(tarfile)
+ except HeaderError:
+ raise SubsequentHeaderError("missing or bad subsequent header")
+
+ if self.type in (XHDTYPE, SOLARIS_XHDTYPE):
+ # Patch the TarInfo object with the extended header info.
+ next._apply_pax_info(pax_headers, tarfile.encoding, tarfile.errors)
+ next.offset = self.offset
+
+ if "size" in pax_headers:
+ # If the extended header replaces the size field,
+ # we need to recalculate the offset where the next
+ # header starts.
+ offset = next.offset_data
+ if next.isreg() or next.type not in SUPPORTED_TYPES:
+ offset += next._block(next.size)
+ tarfile.offset = offset
+
+ return next
+
+ def _apply_pax_info(self, pax_headers, encoding, errors):
+ """Replace fields with supplemental information from a previous
+ pax extended or global header.
+ """
+ for keyword, value in pax_headers.iteritems():
+ if keyword not in PAX_FIELDS:
+ continue
+
+ if keyword == "path":
+ value = value.rstrip("/")
+
+ if keyword in PAX_NUMBER_FIELDS:
+ try:
+ value = PAX_NUMBER_FIELDS[keyword](value)
+ except ValueError:
+ value = 0
+ else:
+ value = uts(value, encoding, errors)
+
+ setattr(self, keyword, value)
+
+ self.pax_headers = pax_headers.copy()
+
+ def _block(self, count):
+ """Round up a byte count by BLOCKSIZE and return it,
+ e.g. _block(834) => 1024.
+ """
+ blocks, remainder = divmod(count, BLOCKSIZE)
+ if remainder:
+ blocks += 1
+ return blocks * BLOCKSIZE
+
+ def isreg(self):
+ return self.type in REGULAR_TYPES
+ def isfile(self):
+ return self.isreg()
+ def isdir(self):
+ return self.type == DIRTYPE
+ def issym(self):
+ return self.type == SYMTYPE
+ def islnk(self):
+ return self.type == LNKTYPE
+ def ischr(self):
+ return self.type == CHRTYPE
+ def isblk(self):
+ return self.type == BLKTYPE
+ def isfifo(self):
+ return self.type == FIFOTYPE
+ def issparse(self):
+ return self.type == GNUTYPE_SPARSE
+ def isdev(self):
+ return self.type in (CHRTYPE, BLKTYPE, FIFOTYPE)
+# class TarInfo
+
+class TarFile(object):
+ """The TarFile Class provides an interface to tar archives.
+ """
+
+ debug = 0 # May be set from 0 (no msgs) to 3 (all msgs)
+
+ dereference = False # If true, add content of linked file to the
+ # tar file, else the link.
+
+ ignore_zeros = False # If true, skips empty or invalid blocks and
+ # continues processing.
+
+ errorlevel = 1 # If 0, fatal errors only appear in debug
+ # messages (if debug >= 0). If > 0, errors
+ # are passed to the caller as exceptions.
+
+ format = DEFAULT_FORMAT # The format to use when creating an archive.
+
+ encoding = ENCODING # Encoding for 8-bit character strings.
+
+ errors = None # Error handler for unicode conversion.
+
+ tarinfo = TarInfo # The default TarInfo class to use.
+
+ fileobject = ExFileObject # The default ExFileObject class to use.
+
+ def __init__(self, name=None, mode="r", fileobj=None, format=None,
+ tarinfo=None, dereference=None, ignore_zeros=None, encoding=None,
+ errors=None, pax_headers=None, debug=None, errorlevel=None):
+ """Open an (uncompressed) tar archive `name'. `mode' is either 'r' to
+ read from an existing archive, 'a' to append data to an existing
+ file or 'w' to create a new file overwriting an existing one. `mode'
+ defaults to 'r'.
+ If `fileobj' is given, it is used for reading or writing data. If it
+ can be determined, `mode' is overridden by `fileobj's mode.
+ `fileobj' is not closed, when TarFile is closed.
+ """
+ if len(mode) > 1 or mode not in "raw":
+ raise ValueError("mode must be 'r', 'a' or 'w'")
+ self.mode = mode
+ self._mode = {"r": "rb", "a": "r+b", "w": "wb"}[mode]
+
+ if not fileobj:
+ if self.mode == "a" and not os.path.exists(name):
+ # Create nonexistent files in append mode.
+ self.mode = "w"
+ self._mode = "wb"
+ fileobj = bltn_open(name, self._mode)
+ self._extfileobj = False
+ else:
+ if name is None and hasattr(fileobj, "name"):
+ name = fileobj.name
+ if hasattr(fileobj, "mode"):
+ self._mode = fileobj.mode
+ self._extfileobj = True
+ if name:
+ self.name = os.path.abspath(name)
+ else:
+ self.name = None
+ self.fileobj = fileobj
+
+ # Init attributes.
+ if format is not None:
+ self.format = format
+ if tarinfo is not None:
+ self.tarinfo = tarinfo
+ if dereference is not None:
+ self.dereference = dereference
+ if ignore_zeros is not None:
+ self.ignore_zeros = ignore_zeros
+ if encoding is not None:
+ self.encoding = encoding
+
+ if errors is not None:
+ self.errors = errors
+ elif mode == "r":
+ self.errors = "utf-8"
+ else:
+ self.errors = "strict"
+
+ if pax_headers is not None and self.format == PAX_FORMAT:
+ self.pax_headers = pax_headers
+ else:
+ self.pax_headers = {}
+
+ if debug is not None:
+ self.debug = debug
+ if errorlevel is not None:
+ self.errorlevel = errorlevel
+
+ # Init datastructures.
+ self.closed = False
+ self.members = [] # list of members as TarInfo objects
+ self._loaded = False # flag if all members have been read
+ self.offset = self.fileobj.tell()
+ # current position in the archive file
+ self.inodes = {} # dictionary caching the inodes of
+ # archive members already added
+
+ try:
+ if self.mode == "r":
+ self.firstmember = None
+ self.firstmember = self.next()
+
+ if self.mode == "a":
+ # Move to the end of the archive,
+ # before the first empty block.
+ while True:
+ self.fileobj.seek(self.offset)
+ try:
+ tarinfo = self.tarinfo.fromtarfile(self)
+ self.members.append(tarinfo)
+ except EOFHeaderError:
+ self.fileobj.seek(self.offset)
+ break
+ except HeaderError, e:
+ raise ReadError(str(e))
+
+ if self.mode in "aw":
+ self._loaded = True
+
+ if self.pax_headers:
+ buf = self.tarinfo.create_pax_global_header(self.pax_headers.copy())
+ self.fileobj.write(buf)
+ self.offset += len(buf)
+ except:
+ if not self._extfileobj:
+ self.fileobj.close()
+ self.closed = True
+ raise
+
+ def _getposix(self):
+ return self.format == USTAR_FORMAT
+ def _setposix(self, value):
+ import warnings
+ warnings.warn("use the format attribute instead", DeprecationWarning,
+ 2)
+ if value:
+ self.format = USTAR_FORMAT
+ else:
+ self.format = GNU_FORMAT
+ posix = property(_getposix, _setposix)
+
+ #--------------------------------------------------------------------------
+ # Below are the classmethods which act as alternate constructors to the
+ # TarFile class. The open() method is the only one that is needed for
+ # public use; it is the "super"-constructor and is able to select an
+ # adequate "sub"-constructor for a particular compression using the mapping
+ # from OPEN_METH.
+ #
+ # This concept allows one to subclass TarFile without losing the comfort of
+ # the super-constructor. A sub-constructor is registered and made available
+ # by adding it to the mapping in OPEN_METH.
+
+ @classmethod
+ def open(cls, name=None, mode="r", fileobj=None, bufsize=RECORDSIZE, **kwargs):
+ """Open a tar archive for reading, writing or appending. Return
+ an appropriate TarFile class.
+
+ mode:
+ 'r' or 'r:*' open for reading with transparent compression
+ 'r:' open for reading exclusively uncompressed
+ 'r:gz' open for reading with gzip compression
+ 'r:bz2' open for reading with bzip2 compression
+ 'a' or 'a:' open for appending, creating the file if necessary
+ 'w' or 'w:' open for writing without compression
+ 'w:gz' open for writing with gzip compression
+ 'w:bz2' open for writing with bzip2 compression
+
+ 'r|*' open a stream of tar blocks with transparent compression
+ 'r|' open an uncompressed stream of tar blocks for reading
+ 'r|gz' open a gzip compressed stream of tar blocks
+ 'r|bz2' open a bzip2 compressed stream of tar blocks
+ 'w|' open an uncompressed stream for writing
+ 'w|gz' open a gzip compressed stream for writing
+ 'w|bz2' open a bzip2 compressed stream for writing
+ """
+
+ if not name and not fileobj:
+ raise ValueError("nothing to open")
+
+ if mode in ("r", "r:*"):
+ # Find out which *open() is appropriate for opening the file.
+ for comptype in cls.OPEN_METH:
+ func = getattr(cls, cls.OPEN_METH[comptype])
+ if fileobj is not None:
+ saved_pos = fileobj.tell()
+ try:
+ return func(name, "r", fileobj, **kwargs)
+ except (ReadError, CompressionError), e:
+ if fileobj is not None:
+ fileobj.seek(saved_pos)
+ continue
+ raise ReadError("file could not be opened successfully")
+
+ elif ":" in mode:
+ filemode, comptype = mode.split(":", 1)
+ filemode = filemode or "r"
+ comptype = comptype or "tar"
+
+ # Select the *open() function according to
+ # given compression.
+ if comptype in cls.OPEN_METH:
+ func = getattr(cls, cls.OPEN_METH[comptype])
+ else:
+ raise CompressionError("unknown compression type %r" % comptype)
+ return func(name, filemode, fileobj, **kwargs)
+
+ elif "|" in mode:
+ filemode, comptype = mode.split("|", 1)
+ filemode = filemode or "r"
+ comptype = comptype or "tar"
+
+ if filemode not in "rw":
+ raise ValueError("mode must be 'r' or 'w'")
+
+ t = cls(name, filemode,
+ _Stream(name, filemode, comptype, fileobj, bufsize),
+ **kwargs)
+ t._extfileobj = False
+ return t
+
+ elif mode in "aw":
+ return cls.taropen(name, mode, fileobj, **kwargs)
+
+ raise ValueError("undiscernible mode")
+
+ @classmethod
+ def taropen(cls, name, mode="r", fileobj=None, **kwargs):
+ """Open uncompressed tar archive name for reading or writing.
+ """
+ if len(mode) > 1 or mode not in "raw":
+ raise ValueError("mode must be 'r', 'a' or 'w'")
+ return cls(name, mode, fileobj, **kwargs)
+
+ @classmethod
+ def gzopen(cls, name, mode="r", fileobj=None, compresslevel=9, **kwargs):
+ """Open gzip compressed tar archive name for reading or writing.
+ Appending is not allowed.
+ """
+ if len(mode) > 1 or mode not in "rw":
+ raise ValueError("mode must be 'r' or 'w'")
+
+ try:
+ import gzip
+ gzip.GzipFile
+ except (ImportError, AttributeError):
+ raise CompressionError("gzip module is not available")
+
+ if fileobj is None:
+ fileobj = bltn_open(name, mode + "b")
+
+ try:
+ t = cls.taropen(name, mode,
+ gzip.GzipFile(name, mode, compresslevel, fileobj),
+ **kwargs)
+ except IOError:
+ raise ReadError("not a gzip file")
+ t._extfileobj = False
+ return t
+
+ @classmethod
+ def bz2open(cls, name, mode="r", fileobj=None, compresslevel=9, **kwargs):
+ """Open bzip2 compressed tar archive name for reading or writing.
+ Appending is not allowed.
+ """
+ if len(mode) > 1 or mode not in "rw":
+ raise ValueError("mode must be 'r' or 'w'.")
+
+ try:
+ import bz2
+ except ImportError:
+ raise CompressionError("bz2 module is not available")
+
+ if fileobj is not None:
+ fileobj = _BZ2Proxy(fileobj, mode)
+ else:
+ fileobj = bz2.BZ2File(name, mode, compresslevel=compresslevel)
+
+ try:
+ t = cls.taropen(name, mode, fileobj, **kwargs)
+ except (IOError, EOFError):
+ raise ReadError("not a bzip2 file")
+ t._extfileobj = False
+ return t
+
+ # All *open() methods are registered here.
+ OPEN_METH = {
+ "tar": "taropen", # uncompressed tar
+ "gz": "gzopen", # gzip compressed tar
+ "bz2": "bz2open" # bzip2 compressed tar
+ }
+
+ #--------------------------------------------------------------------------
+ # The public methods which TarFile provides:
+
+ def close(self):
+ """Close the TarFile. In write-mode, two finishing zero blocks are
+ appended to the archive.
+ """
+ if self.closed:
+ return
+
+ if self.mode in "aw":
+ self.fileobj.write(NUL * (BLOCKSIZE * 2))
+ self.offset += (BLOCKSIZE * 2)
+ # fill up the end with zero-blocks
+ # (like option -b20 for tar does)
+ blocks, remainder = divmod(self.offset, RECORDSIZE)
+ if remainder > 0:
+ self.fileobj.write(NUL * (RECORDSIZE - remainder))
+
+ if not self._extfileobj:
+ self.fileobj.close()
+ self.closed = True
+
+ def getmember(self, name):
+ """Return a TarInfo object for member `name'. If `name' can not be
+ found in the archive, KeyError is raised. If a member occurs more
+ than once in the archive, its last occurrence is assumed to be the
+ most up-to-date version.
+ """
+ tarinfo = self._getmember(name)
+ if tarinfo is None:
+ raise KeyError("filename %r not found" % name)
+ return tarinfo
+
+ def getmembers(self):
+ """Return the members of the archive as a list of TarInfo objects. The
+ list has the same order as the members in the archive.
+ """
+ self._check()
+ if not self._loaded: # if we want to obtain a list of
+ self._load() # all members, we first have to
+ # scan the whole archive.
+ return self.members
+
+ def getnames(self):
+ """Return the members of the archive as a list of their names. It has
+ the same order as the list returned by getmembers().
+ """
+ return [tarinfo.name for tarinfo in self.getmembers()]
+
+ def gettarinfo(self, name=None, arcname=None, fileobj=None):
+ """Create a TarInfo object for either the file `name' or the file
+ object `fileobj' (using os.fstat on its file descriptor). You can
+ modify some of the TarInfo's attributes before you add it using
+ addfile(). If given, `arcname' specifies an alternative name for the
+ file in the archive.
+ """
+ self._check("aw")
+
+ # When fileobj is given, replace name by
+ # fileobj's real name.
+ if fileobj is not None:
+ name = fileobj.name
+
+ # Building the name of the member in the archive.
+ # Backward slashes are converted to forward slashes,
+ # Absolute paths are turned to relative paths.
+ if arcname is None:
+ arcname = name
+ drv, arcname = os.path.splitdrive(arcname)
+ arcname = arcname.replace(os.sep, "/")
+ arcname = arcname.lstrip("/")
+
+ # Now, fill the TarInfo object with
+ # information specific for the file.
+ tarinfo = self.tarinfo()
+ tarinfo.tarfile = self
+
+ # Use os.stat or os.lstat, depending on platform
+ # and if symlinks shall be resolved.
+ if fileobj is None:
+ if hasattr(os, "lstat") and not self.dereference:
+ statres = os.lstat(name)
+ else:
+ statres = os.stat(name)
+ else:
+ statres = os.fstat(fileobj.fileno())
+ linkname = ""
+
+ stmd = statres.st_mode
+ if stat.S_ISREG(stmd):
+ inode = (statres.st_ino, statres.st_dev)
+ if not self.dereference and statres.st_nlink > 1 and \
+ inode in self.inodes and arcname != self.inodes[inode]:
+ # Is it a hardlink to an already
+ # archived file?
+ type = LNKTYPE
+ linkname = self.inodes[inode]
+ else:
+ # The inode is added only if its valid.
+ # For win32 it is always 0.
+ type = REGTYPE
+ if inode[0]:
+ self.inodes[inode] = arcname
+ elif stat.S_ISDIR(stmd):
+ type = DIRTYPE
+ elif stat.S_ISFIFO(stmd):
+ type = FIFOTYPE
+ elif stat.S_ISLNK(stmd):
+ type = SYMTYPE
+ linkname = os.readlink(name)
+ elif stat.S_ISCHR(stmd):
+ type = CHRTYPE
+ elif stat.S_ISBLK(stmd):
+ type = BLKTYPE
+ else:
+ return None
+
+ # Fill the TarInfo object with all
+ # information we can get.
+ tarinfo.name = arcname
+ tarinfo.mode = stmd
+ tarinfo.uid = statres.st_uid
+ tarinfo.gid = statres.st_gid
+ if type == REGTYPE:
+ tarinfo.size = statres.st_size
+ else:
+ tarinfo.size = 0L
+ tarinfo.mtime = statres.st_mtime
+ tarinfo.type = type
+ tarinfo.linkname = linkname
+ if pwd:
+ try:
+ tarinfo.uname = pwd.getpwuid(tarinfo.uid)[0]
+ except KeyError:
+ pass
+ if grp:
+ try:
+ tarinfo.gname = grp.getgrgid(tarinfo.gid)[0]
+ except KeyError:
+ pass
+
+ if type in (CHRTYPE, BLKTYPE):
+ if hasattr(os, "major") and hasattr(os, "minor"):
+ tarinfo.devmajor = os.major(statres.st_rdev)
+ tarinfo.devminor = os.minor(statres.st_rdev)
+ return tarinfo
+
+ def list(self, verbose=True):
+ """Print a table of contents to sys.stdout. If `verbose' is False, only
+ the names of the members are printed. If it is True, an `ls -l'-like
+ output is produced.
+ """
+ self._check()
+
+ for tarinfo in self:
+ if verbose:
+ print filemode(tarinfo.mode),
+ print "%s/%s" % (tarinfo.uname or tarinfo.uid,
+ tarinfo.gname or tarinfo.gid),
+ if tarinfo.ischr() or tarinfo.isblk():
+ print "%10s" % ("%d,%d" \
+ % (tarinfo.devmajor, tarinfo.devminor)),
+ else:
+ print "%10d" % tarinfo.size,
+ print "%d-%02d-%02d %02d:%02d:%02d" \
+ % time.localtime(tarinfo.mtime)[:6],
+
+ if tarinfo.isdir():
+ print tarinfo.name + "/",
+ else:
+ print tarinfo.name,
+
+ if verbose:
+ if tarinfo.issym():
+ print "->", tarinfo.linkname,
+ if tarinfo.islnk():
+ print "link to", tarinfo.linkname,
+ print
+
+ def add(self, name, arcname=None, recursive=True, exclude=None, filter=None):
+ """Add the file `name' to the archive. `name' may be any type of file
+ (directory, fifo, symbolic link, etc.). If given, `arcname'
+ specifies an alternative name for the file in the archive.
+ Directories are added recursively by default. This can be avoided by
+ setting `recursive' to False. `exclude' is a function that should
+ return True for each filename to be excluded. `filter' is a function
+ that expects a TarInfo object argument and returns the changed
+ TarInfo object, if it returns None the TarInfo object will be
+ excluded from the archive.
+ """
+ self._check("aw")
+
+ if arcname is None:
+ arcname = name
+
+ # Exclude pathnames.
+ if exclude is not None:
+ import warnings
+ warnings.warn("use the filter argument instead",
+ DeprecationWarning, 2)
+ if exclude(name):
+ self._dbg(2, "tarfile: Excluded %r" % name)
+ return
+
+ # Skip if somebody tries to archive the archive...
+ if self.name is not None and os.path.abspath(name) == self.name:
+ self._dbg(2, "tarfile: Skipped %r" % name)
+ return
+
+ self._dbg(1, name)
+
+ # Create a TarInfo object from the file.
+ tarinfo = self.gettarinfo(name, arcname)
+
+ if tarinfo is None:
+ self._dbg(1, "tarfile: Unsupported type %r" % name)
+ return
+
+ # Change or exclude the TarInfo object.
+ if filter is not None:
+ tarinfo = filter(tarinfo)
+ if tarinfo is None:
+ self._dbg(2, "tarfile: Excluded %r" % name)
+ return
+
+ # Append the tar header and data to the archive.
+ if tarinfo.isreg():
+ f = bltn_open(name, "rb")
+ self.addfile(tarinfo, f)
+ f.close()
+
+ elif tarinfo.isdir():
+ self.addfile(tarinfo)
+ if recursive:
+ for f in os.listdir(name):
+ self.add(os.path.join(name, f), os.path.join(arcname, f),
+ recursive, exclude, filter)
+
+ else:
+ self.addfile(tarinfo)
+
+ def addfile(self, tarinfo, fileobj=None):
+ """Add the TarInfo object `tarinfo' to the archive. If `fileobj' is
+ given, tarinfo.size bytes are read from it and added to the archive.
+ You can create TarInfo objects using gettarinfo().
+ On Windows platforms, `fileobj' should always be opened with mode
+ 'rb' to avoid irritation about the file size.
+ """
+ self._check("aw")
+
+ tarinfo = copy.copy(tarinfo)
+
+ buf = tarinfo.tobuf(self.format, self.encoding, self.errors)
+ self.fileobj.write(buf)
+ self.offset += len(buf)
+
+ # If there's data to follow, append it.
+ if fileobj is not None:
+ copyfileobj(fileobj, self.fileobj, tarinfo.size)
+ blocks, remainder = divmod(tarinfo.size, BLOCKSIZE)
+ if remainder > 0:
+ self.fileobj.write(NUL * (BLOCKSIZE - remainder))
+ blocks += 1
+ self.offset += blocks * BLOCKSIZE
+
+ self.members.append(tarinfo)
+
+ def extractall(self, path=".", members=None):
+ """Extract all members from the archive to the current working
+ directory and set owner, modification time and permissions on
+ directories afterwards. `path' specifies a different directory
+ to extract to. `members' is optional and must be a subset of the
+ list returned by getmembers().
+ """
+ directories = []
+
+ if members is None:
+ members = self
+
+ for tarinfo in members:
+ if tarinfo.isdir():
+ # Extract directories with a safe mode.
+ directories.append(tarinfo)
+ tarinfo = copy.copy(tarinfo)
+ tarinfo.mode = 0700
+ self.extract(tarinfo, path)
+
+ # Reverse sort directories.
+ directories.sort(key=operator.attrgetter('name'))
+ directories.reverse()
+
+ # Set correct owner, mtime and filemode on directories.
+ for tarinfo in directories:
+ dirpath = os.path.join(path, tarinfo.name)
+ try:
+ self.chown(tarinfo, dirpath)
+ self.utime(tarinfo, dirpath)
+ self.chmod(tarinfo, dirpath)
+ except ExtractError, e:
+ if self.errorlevel > 1:
+ raise
+ else:
+ self._dbg(1, "tarfile: %s" % e)
+
+ def extract(self, member, path=""):
+ """Extract a member from the archive to the current working directory,
+ using its full name. Its file information is extracted as accurately
+ as possible. `member' may be a filename or a TarInfo object. You can
+ specify a different directory using `path'.
+ """
+ self._check("r")
+
+ if isinstance(member, basestring):
+ tarinfo = self.getmember(member)
+ else:
+ tarinfo = member
+
+ # Prepare the link target for makelink().
+ if tarinfo.islnk():
+ tarinfo._link_target = os.path.join(path, tarinfo.linkname)
+
+ try:
+ self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
+ except EnvironmentError, e:
+ if self.errorlevel > 0:
+ raise
+ else:
+ if e.filename is None:
+ self._dbg(1, "tarfile: %s" % e.strerror)
+ else:
+ self._dbg(1, "tarfile: %s %r" % (e.strerror, e.filename))
+ except ExtractError, e:
+ if self.errorlevel > 1:
+ raise
+ else:
+ self._dbg(1, "tarfile: %s" % e)
+
+ def extractfile(self, member):
+ """Extract a member from the archive as a file object. `member' may be
+ a filename or a TarInfo object. If `member' is a regular file, a
+ file-like object is returned. If `member' is a link, a file-like
+ object is constructed from the link's target. If `member' is none of
+ the above, None is returned.
+ The file-like object is read-only and provides the following
+ methods: read(), readline(), readlines(), seek() and tell()
+ """
+ self._check("r")
+
+ if isinstance(member, basestring):
+ tarinfo = self.getmember(member)
+ else:
+ tarinfo = member
+
+ if tarinfo.isreg():
+ return self.fileobject(self, tarinfo)
+
+ elif tarinfo.type not in SUPPORTED_TYPES:
+ # If a member's type is unknown, it is treated as a
+ # regular file.
+ return self.fileobject(self, tarinfo)
+
+ elif tarinfo.islnk() or tarinfo.issym():
+ if isinstance(self.fileobj, _Stream):
+ # A small but ugly workaround for the case that someone tries
+ # to extract a (sym)link as a file-object from a non-seekable
+ # stream of tar blocks.
+ raise StreamError("cannot extract (sym)link as file object")
+ else:
+ # A (sym)link's file object is its target's file object.
+ return self.extractfile(self._find_link_target(tarinfo))
+ else:
+ # If there's no data associated with the member (directory, chrdev,
+ # blkdev, etc.), return None instead of a file object.
+ return None
+
+ def _extract_member(self, tarinfo, targetpath):
+ """Extract the TarInfo object tarinfo to a physical
+ file called targetpath.
+ """
+ # Fetch the TarInfo object for the given name
+ # and build the destination pathname, replacing
+ # forward slashes to platform specific separators.
+ targetpath = targetpath.rstrip("/")
+ targetpath = targetpath.replace("/", os.sep)
+
+ # Create all upper directories.
+ upperdirs = os.path.dirname(targetpath)
+ if upperdirs and not os.path.exists(upperdirs):
+ # Create directories that are not part of the archive with
+ # default permissions.
+ os.makedirs(upperdirs)
+
+ if tarinfo.islnk() or tarinfo.issym():
+ self._dbg(1, "%s -> %s" % (tarinfo.name, tarinfo.linkname))
+ else:
+ self._dbg(1, tarinfo.name)
+
+ if tarinfo.isreg():
+ self.makefile(tarinfo, targetpath)
+ elif tarinfo.isdir():
+ self.makedir(tarinfo, targetpath)
+ elif tarinfo.isfifo():
+ self.makefifo(tarinfo, targetpath)
+ elif tarinfo.ischr() or tarinfo.isblk():
+ self.makedev(tarinfo, targetpath)
+ elif tarinfo.islnk() or tarinfo.issym():
+ self.makelink(tarinfo, targetpath)
+ elif tarinfo.type not in SUPPORTED_TYPES:
+ self.makeunknown(tarinfo, targetpath)
+ else:
+ self.makefile(tarinfo, targetpath)
+
+ self.chown(tarinfo, targetpath)
+ if not tarinfo.issym():
+ self.chmod(tarinfo, targetpath)
+ self.utime(tarinfo, targetpath)
+
+ #--------------------------------------------------------------------------
+ # Below are the different file methods. They are called via
+ # _extract_member() when extract() is called. They can be replaced in a
+ # subclass to implement other functionality.
+
+ def makedir(self, tarinfo, targetpath):
+ """Make a directory called targetpath.
+ """
+ try:
+ # Use a safe mode for the directory, the real mode is set
+ # later in _extract_member().
+ os.mkdir(targetpath, 0700)
+ except EnvironmentError, e:
+ if e.errno != errno.EEXIST:
+ raise
+
+ def makefile(self, tarinfo, targetpath):
+ """Make a file called targetpath.
+ """
+ source = self.extractfile(tarinfo)
+ target = bltn_open(targetpath, "wb")
+ copyfileobj(source, target)
+ source.close()
+ target.close()
+
+ def makeunknown(self, tarinfo, targetpath):
+ """Make a file from a TarInfo object with an unknown type
+ at targetpath.
+ """
+ self.makefile(tarinfo, targetpath)
+ self._dbg(1, "tarfile: Unknown file type %r, " \
+ "extracted as regular file." % tarinfo.type)
+
+ def makefifo(self, tarinfo, targetpath):
+ """Make a fifo called targetpath.
+ """
+ if hasattr(os, "mkfifo"):
+ os.mkfifo(targetpath)
+ else:
+ raise ExtractError("fifo not supported by system")
+
+ def makedev(self, tarinfo, targetpath):
+ """Make a character or block device called targetpath.
+ """
+ if not hasattr(os, "mknod") or not hasattr(os, "makedev"):
+ raise ExtractError("special devices not supported by system")
+
+ mode = tarinfo.mode
+ if tarinfo.isblk():
+ mode |= stat.S_IFBLK
+ else:
+ mode |= stat.S_IFCHR
+
+ os.mknod(targetpath, mode,
+ os.makedev(tarinfo.devmajor, tarinfo.devminor))
+
+ def makelink(self, tarinfo, targetpath):
+ """Make a (symbolic) link called targetpath. If it cannot be created
+ (platform limitation), we try to make a copy of the referenced file
+ instead of a link.
+ """
+ if hasattr(os, "symlink") and hasattr(os, "link"):
+ # For systems that support symbolic and hard links.
+ if tarinfo.issym():
+ os.symlink(tarinfo.linkname, targetpath)
+ else:
+ # See extract().
+ if os.path.exists(tarinfo._link_target):
+ os.link(tarinfo._link_target, targetpath)
+ else:
+ self._extract_member(self._find_link_target(tarinfo), targetpath)
+ else:
+ try:
+ self._extract_member(self._find_link_target(tarinfo), targetpath)
+ except KeyError:
+ raise ExtractError("unable to resolve link inside archive")
+
+ def chown(self, tarinfo, targetpath):
+ """Set owner of targetpath according to tarinfo.
+ """
+ if pwd and hasattr(os, "geteuid") and os.geteuid() == 0:
+ # We have to be root to do so.
+ try:
+ g = grp.getgrnam(tarinfo.gname)[2]
+ except KeyError:
+ try:
+ g = grp.getgrgid(tarinfo.gid)[2]
+ except KeyError:
+ g = os.getgid()
+ try:
+ u = pwd.getpwnam(tarinfo.uname)[2]
+ except KeyError:
+ try:
+ u = pwd.getpwuid(tarinfo.uid)[2]
+ except KeyError:
+ u = os.getuid()
+ try:
+ if tarinfo.issym() and hasattr(os, "lchown"):
+ os.lchown(targetpath, u, g)
+ else:
+ if sys.platform != "os2emx":
+ os.chown(targetpath, u, g)
+ except EnvironmentError, e:
+ raise ExtractError("could not change owner to %d:%d" % (u, g))
+
+ def chmod(self, tarinfo, targetpath):
+ """Set file permissions of targetpath according to tarinfo.
+ """
+ if hasattr(os, 'chmod'):
+ try:
+ os.chmod(targetpath, tarinfo.mode)
+ except EnvironmentError, e:
+ raise ExtractError("could not change mode")
+
+ def utime(self, tarinfo, targetpath):
+ """Set modification time of targetpath according to tarinfo.
+ """
+ if not hasattr(os, 'utime'):
+ return
+ try:
+ os.utime(targetpath, (tarinfo.mtime, tarinfo.mtime))
+ except EnvironmentError, e:
+ raise ExtractError("could not change modification time")
+
+ #--------------------------------------------------------------------------
+ def next(self):
+ """Return the next member of the archive as a TarInfo object, when
+ TarFile is opened for reading. Return None if there is no more
+ available.
+ """
+ self._check("ra")
+ if self.firstmember is not None:
+ m = self.firstmember
+ self.firstmember = None
+ return m
+
+ # Read the next block.
+ self.fileobj.seek(self.offset)
+ tarinfo = None
+ while True:
+ try:
+ tarinfo = self.tarinfo.fromtarfile(self)
+ except EOFHeaderError, e:
+ if self.ignore_zeros:
+ self._dbg(2, "0x%X: %s" % (self.offset, e))
+ self.offset += BLOCKSIZE
+ continue
+ except InvalidHeaderError, e:
+ if self.ignore_zeros:
+ self._dbg(2, "0x%X: %s" % (self.offset, e))
+ self.offset += BLOCKSIZE
+ continue
+ elif self.offset == 0:
+ raise ReadError(str(e))
+ except EmptyHeaderError:
+ if self.offset == 0:
+ raise ReadError("empty file")
+ except TruncatedHeaderError, e:
+ if self.offset == 0:
+ raise ReadError(str(e))
+ except SubsequentHeaderError, e:
+ raise ReadError(str(e))
+ break
+
+ if tarinfo is not None:
+ self.members.append(tarinfo)
+ else:
+ self._loaded = True
+
+ return tarinfo
+
+ #--------------------------------------------------------------------------
+ # Little helper methods:
+
+ def _getmember(self, name, tarinfo=None, normalize=False):
+ """Find an archive member by name from bottom to top.
+ If tarinfo is given, it is used as the starting point.
+ """
+ # Ensure that all members have been loaded.
+ members = self.getmembers()
+
+ # Limit the member search list up to tarinfo.
+ if tarinfo is not None:
+ members = members[:members.index(tarinfo)]
+
+ if normalize:
+ name = os.path.normpath(name)
+
+ for member in reversed(members):
+ if normalize:
+ member_name = os.path.normpath(member.name)
+ else:
+ member_name = member.name
+
+ if name == member_name:
+ return member
+
+ def _load(self):
+ """Read through the entire archive file and look for readable
+ members.
+ """
+ while True:
+ tarinfo = self.next()
+ if tarinfo is None:
+ break
+ self._loaded = True
+
+ def _check(self, mode=None):
+ """Check if TarFile is still open, and if the operation's mode
+ corresponds to TarFile's mode.
+ """
+ if self.closed:
+ raise IOError("%s is closed" % self.__class__.__name__)
+ if mode is not None and self.mode not in mode:
+ raise IOError("bad operation for mode %r" % self.mode)
+
+ def _find_link_target(self, tarinfo):
+ """Find the target member of a symlink or hardlink member in the
+ archive.
+ """
+ if tarinfo.issym():
+ # Always search the entire archive.
+ linkname = os.path.dirname(tarinfo.name) + "/" + tarinfo.linkname
+ limit = None
+ else:
+ # Search the archive before the link, because a hard link is
+ # just a reference to an already archived file.
+ linkname = tarinfo.linkname
+ limit = tarinfo
+
+ member = self._getmember(linkname, tarinfo=limit, normalize=True)
+ if member is None:
+ raise KeyError("linkname %r not found" % linkname)
+ return member
+
+ def __iter__(self):
+ """Provide an iterator object.
+ """
+ if self._loaded:
+ return iter(self.members)
+ else:
+ return TarIter(self)
+
+ def _dbg(self, level, msg):
+ """Write debugging output to sys.stderr.
+ """
+ if level <= self.debug:
+ print >> sys.stderr, msg
+
+ def __enter__(self):
+ self._check()
+ return self
+
+ def __exit__(self, type, value, traceback):
+ if type is None:
+ self.close()
+ else:
+ # An exception occurred. We must not call close() because
+ # it would try to write end-of-archive blocks and padding.
+ if not self._extfileobj:
+ self.fileobj.close()
+ self.closed = True
+# class TarFile
+
+class TarIter:
+ """Iterator Class.
+
+ for tarinfo in TarFile(...):
+ suite...
+ """
+
+ def __init__(self, tarfile):
+ """Construct a TarIter object.
+ """
+ self.tarfile = tarfile
+ self.index = 0
+ def __iter__(self):
+ """Return iterator object.
+ """
+ return self
+ def next(self):
+ """Return the next item using TarFile's next() method.
+ When all members have been read, set TarFile as _loaded.
+ """
+ # Fix for SF #1100429: Under rare circumstances it can
+ # happen that getmembers() is called during iteration,
+ # which will cause TarIter to stop prematurely.
+ if not self.tarfile._loaded:
+ tarinfo = self.tarfile.next()
+ if not tarinfo:
+ self.tarfile._loaded = True
+ raise StopIteration
+ else:
+ try:
+ tarinfo = self.tarfile.members[self.index]
+ except IndexError:
+ raise StopIteration
+ self.index += 1
+ return tarinfo
+
+# Helper classes for sparse file support
+class _section:
+ """Base class for _data and _hole.
+ """
+ def __init__(self, offset, size):
+ self.offset = offset
+ self.size = size
+ def __contains__(self, offset):
+ return self.offset <= offset < self.offset + self.size
+
+class _data(_section):
+ """Represent a data section in a sparse file.
+ """
+ def __init__(self, offset, size, realpos):
+ _section.__init__(self, offset, size)
+ self.realpos = realpos
+
+class _hole(_section):
+ """Represent a hole section in a sparse file.
+ """
+ pass
+
+class _ringbuffer(list):
+ """Ringbuffer class which increases performance
+ over a regular list.
+ """
+ def __init__(self):
+ self.idx = 0
+ def find(self, offset):
+ idx = self.idx
+ while True:
+ item = self[idx]
+ if offset in item:
+ break
+ idx += 1
+ if idx == len(self):
+ idx = 0
+ if idx == self.idx:
+ # End of File
+ return None
+ self.idx = idx
+ return item
+
+#---------------------------------------------
+# zipfile compatible TarFile class
+#---------------------------------------------
+TAR_PLAIN = 0 # zipfile.ZIP_STORED
+TAR_GZIPPED = 8 # zipfile.ZIP_DEFLATED
+class TarFileCompat:
+ """TarFile class compatible with standard module zipfile's
+ ZipFile class.
+ """
+ def __init__(self, file, mode="r", compression=TAR_PLAIN):
+ from warnings import warnpy3k
+ warnpy3k("the TarFileCompat class has been removed in Python 3.0",
+ stacklevel=2)
+ if compression == TAR_PLAIN:
+ self.tarfile = TarFile.taropen(file, mode)
+ elif compression == TAR_GZIPPED:
+ self.tarfile = TarFile.gzopen(file, mode)
+ else:
+ raise ValueError("unknown compression constant")
+ if mode[0:1] == "r":
+ members = self.tarfile.getmembers()
+ for m in members:
+ m.filename = m.name
+ m.file_size = m.size
+ m.date_time = time.gmtime(m.mtime)[:6]
+ def namelist(self):
+ return map(lambda m: m.name, self.infolist())
+ def infolist(self):
+ return filter(lambda m: m.type in REGULAR_TYPES,
+ self.tarfile.getmembers())
+ def printdir(self):
+ self.tarfile.list()
+ def testzip(self):
+ return
+ def getinfo(self, name):
+ return self.tarfile.getmember(name)
+ def read(self, name):
+ return self.tarfile.extractfile(self.tarfile.getmember(name)).read()
+ def write(self, filename, arcname=None, compress_type=None):
+ self.tarfile.add(filename, arcname)
+ def writestr(self, zinfo, bytes):
+ try:
+ from cStringIO import StringIO
+ except ImportError:
+ from StringIO import StringIO
+ import calendar
+ tinfo = TarInfo(zinfo.filename)
+ tinfo.size = len(bytes)
+ tinfo.mtime = calendar.timegm(zinfo.date_time)
+ self.tarfile.addfile(tinfo, StringIO(bytes))
+ def close(self):
+ self.tarfile.close()
+#class TarFileCompat
+
+#--------------------
+# exported functions
+#--------------------
+def is_tarfile(name):
+ """Return True if name points to a tar archive that we
+ are able to handle, else return False.
+ """
+ try:
+ t = open(name)
+ t.close()
+ return True
+ except TarError:
+ return False
+
+bltn_open = open
+open = TarFile.open
=== modified file 'duplicity/tempdir.py'
--- duplicity/tempdir.py 2014-04-20 05:58:47 +0000
+++ duplicity/tempdir.py 2014-10-15 12:12:04 +0000
@@ -213,7 +213,7 @@
"""
self.__lock.acquire()
try:
- if fname in self.__pending:
+ if self.__pending.has_key(fname):
log.Debug(_("Forgetting temporary file %s") % util.ufn(fname))
del(self.__pending[fname])
else:
=== added file 'duplicity/urlparse_2_5.py'
--- duplicity/urlparse_2_5.py 1970-01-01 00:00:00 +0000
+++ duplicity/urlparse_2_5.py 2014-10-15 12:12:04 +0000
@@ -0,0 +1,385 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+
+"""Parse (absolute and relative) URLs.
+
+See RFC 1808: "Relative Uniform Resource Locators", by R. Fielding,
+UC Irvine, June 1995.
+"""
+
+__all__ = ["urlparse", "urlunparse", "urljoin", "urldefrag",
+ "urlsplit", "urlunsplit"]
+
+# A classification of schemes ('' means apply by default)
+uses_relative = ['ftp', 'ftps', 'http', 'gopher', 'nntp',
+ 'wais', 'file', 'https', 'shttp', 'mms',
+ 'prospero', 'rtsp', 'rtspu', '', 'sftp', 'imap', 'imaps']
+uses_netloc = ['ftp', 'ftps', 'http', 'gopher', 'nntp', 'telnet',
+ 'wais', 'file', 'mms', 'https', 'shttp',
+ 'snews', 'prospero', 'rtsp', 'rtspu', 'rsync', '',
+ 'svn', 'svn+ssh', 'sftp', 'imap', 'imaps']
+non_hierarchical = ['gopher', 'hdl', 'mailto', 'news',
+ 'telnet', 'wais', 'snews', 'sip', 'sips', 'imap', 'imaps']
+uses_params = ['ftp', 'ftps', 'hdl', 'prospero', 'http',
+ 'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips',
+ 'mms', '', 'sftp', 'imap', 'imaps']
+uses_query = ['http', 'wais', 'https', 'shttp', 'mms',
+ 'gopher', 'rtsp', 'rtspu', 'sip', 'sips', 'imap', 'imaps', '']
+uses_fragment = ['ftp', 'ftps', 'hdl', 'http', 'gopher', 'news',
+ 'nntp', 'wais', 'https', 'shttp', 'snews',
+ 'file', 'prospero', '']
+
+# Characters valid in scheme names
+scheme_chars = ('abcdefghijklmnopqrstuvwxyz'
+ 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+ '0123456789'
+ '+-.')
+
+MAX_CACHE_SIZE = 20
+_parse_cache = {}
+
+def clear_cache():
+ """Clear the parse cache."""
+ global _parse_cache
+ _parse_cache = {}
+
+import string
+def _rsplit(str, delim, numsplit):
+ parts = string.split(str, delim)
+ if len(parts) <= numsplit + 1:
+ return parts
+ else:
+ left = string.join(parts[0:-numsplit], delim)
+ right = string.join(parts[len(parts)-numsplit:], delim)
+ return [left, right]
+
+class BaseResult(tuple):
+ """Base class for the parsed result objects.
+
+ This provides the attributes shared by the two derived result
+ objects as read-only properties. The derived classes are
+ responsible for checking the right number of arguments were
+ supplied to the constructor.
+
+ """
+
+ __slots__ = ()
+
+ # Attributes that access the basic components of the URL:
+
+ def get_scheme(self):
+ return self[0]
+ scheme = property(get_scheme)
+
+ def get_netloc(self):
+ return self[1]
+ netloc = property(get_netloc)
+
+ def get_path(self):
+ return self[2]
+ path = property(get_path)
+
+ def get_query(self):
+ return self[-2]
+ query = property(get_query)
+
+ def get_fragment(self):
+ return self[-1]
+ fragment = property(get_fragment)
+
+ # Additional attributes that provide access to parsed-out portions
+ # of the netloc:
+
+ def get_username(self):
+ netloc = self.netloc
+ if "@" in netloc:
+ userinfo = _rsplit(netloc, "@", 1)[0]
+ if ":" in userinfo:
+ userinfo = userinfo.split(":", 1)[0]
+ return userinfo
+ return None
+ username = property(get_username)
+
+ def get_password(self):
+ netloc = self.netloc
+ if "@" in netloc:
+ userinfo = _rsplit(netloc, "@", 1)[0]
+ if ":" in userinfo:
+ return userinfo.split(":", 1)[1]
+ return None
+ password = property(get_password)
+
+ def get_hostname(self):
+ netloc = self.netloc.split('@')[-1]
+ if '[' in netloc and ']' in netloc:
+ return netloc.split(']')[0][1:].lower()
+ elif ':' in netloc:
+ return netloc.split(':')[0].lower()
+ elif netloc == '':
+ return None
+ else:
+ return netloc.lower()
+ hostname = property(get_hostname)
+
+ def get_port(self):
+ netloc = self.netloc.split('@')[-1].split(']')[-1]
+ if ":" in netloc:
+ port = netloc.split(":", 1)[1]
+ return int(port, 10)
+ return None
+ port = property(get_port)
+
+
+class SplitResult(BaseResult):
+
+ __slots__ = ()
+
+ def __new__(cls, scheme, netloc, path, query, fragment):
+ return BaseResult.__new__(
+ cls, (scheme, netloc, path, query, fragment))
+
+ def geturl(self):
+ return urlunsplit(self)
+
+
+class ParseResult(BaseResult):
+
+ __slots__ = ()
+
+ def __new__(cls, scheme, netloc, path, params, query, fragment):
+ return BaseResult.__new__(
+ cls, (scheme, netloc, path, params, query, fragment))
+
+ def get_params(self):
+ return self[3]
+ params = property(get_params)
+
+ def geturl(self):
+ return urlunparse(self)
+
+
+def urlparse(url, scheme='', allow_fragments=True):
+ """Parse a URL into 6 components:
+ <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
+ Return a 6-tuple: (scheme, netloc, path, params, query, fragment).
+ Note that we don't break the components up in smaller bits
+ (e.g. netloc is a single string) and we don't expand % escapes."""
+ tuple = urlsplit(url, scheme, allow_fragments)
+ scheme, netloc, url, query, fragment = tuple
+ if scheme in uses_params and ';' in url:
+ url, params = _splitparams(url)
+ else:
+ params = ''
+ return ParseResult(scheme, netloc, url, params, query, fragment)
+
+def _splitparams(url):
+ if '/' in url:
+ i = url.find(';', url.rfind('/'))
+ if i < 0:
+ return url, ''
+ else:
+ i = url.find(';')
+ return url[:i], url[i+1:]
+
+def _splitnetloc(url, start=0):
+ for c in '/?#': # the order is important!
+ delim = url.find(c, start)
+ if delim >= 0:
+ break
+ else:
+ delim = len(url)
+ return url[start:delim], url[delim:]
+
+def urlsplit(url, scheme='', allow_fragments=True):
+ """Parse a URL into 5 components:
+ <scheme>://<netloc>/<path>?<query>#<fragment>
+ Return a 5-tuple: (scheme, netloc, path, query, fragment).
+ Note that we don't break the components up in smaller bits
+ (e.g. netloc is a single string) and we don't expand % escapes."""
+ allow_fragments = bool(allow_fragments)
+ key = url, scheme, allow_fragments
+ cached = _parse_cache.get(key, None)
+ if cached:
+ return cached
+ if len(_parse_cache) >= MAX_CACHE_SIZE: # avoid runaway growth
+ clear_cache()
+ netloc = query = fragment = ''
+ i = url.find(':')
+ if i > 0:
+ if url[:i] == 'http': # optimize the common case
+ scheme = url[:i].lower()
+ url = url[i+1:]
+ if url[:2] == '//':
+ netloc, url = _splitnetloc(url, 2)
+ if allow_fragments and '#' in url:
+ url, fragment = url.split('#', 1)
+ if '?' in url:
+ url, query = url.split('?', 1)
+ v = SplitResult(scheme, netloc, url, query, fragment)
+ _parse_cache[key] = v
+ return v
+ for c in url[:i]:
+ if c not in scheme_chars:
+ break
+ else:
+ scheme, url = url[:i].lower(), url[i+1:]
+ if scheme in uses_netloc and url[:2] == '//':
+ netloc, url = _splitnetloc(url, 2)
+ if allow_fragments and scheme in uses_fragment and '#' in url:
+ url, fragment = url.split('#', 1)
+ if scheme in uses_query and '?' in url:
+ url, query = url.split('?', 1)
+ v = SplitResult(scheme, netloc, url, query, fragment)
+ _parse_cache[key] = v
+ return v
+
+def urlunparse((scheme, netloc, url, params, query, fragment)):
+ """Put a parsed URL back together again. This may result in a
+ slightly different, but equivalent URL, if the URL that was parsed
+ originally had redundant delimiters, e.g. a ? with an empty query
+ (the draft states that these are equivalent)."""
+ if params:
+ url = "%s;%s" % (url, params)
+ return urlunsplit((scheme, netloc, url, query, fragment))
+
+def urlunsplit((scheme, netloc, url, query, fragment)):
+ if netloc or (scheme and scheme in uses_netloc and url[:2] != '//'):
+ if url and url[:1] != '/': url = '/' + url
+ url = '//' + (netloc or '') + url
+ if scheme:
+ url = scheme + ':' + url
+ if query:
+ url = url + '?' + query
+ if fragment:
+ url = url + '#' + fragment
+ return url
+
+def urljoin(base, url, allow_fragments=True):
+ """Join a base URL and a possibly relative URL to form an absolute
+ interpretation of the latter."""
+ if not base:
+ return url
+ if not url:
+ return base
+ bscheme, bnetloc, bpath, bparams, bquery, bfragment = urlparse(base, '', allow_fragments) #@UnusedVariable
+ scheme, netloc, path, params, query, fragment = urlparse(url, bscheme, allow_fragments)
+ if scheme != bscheme or scheme not in uses_relative:
+ return url
+ if scheme in uses_netloc:
+ if netloc:
+ return urlunparse((scheme, netloc, path,
+ params, query, fragment))
+ netloc = bnetloc
+ if path[:1] == '/':
+ return urlunparse((scheme, netloc, path,
+ params, query, fragment))
+ if not (path or params or query):
+ return urlunparse((scheme, netloc, bpath,
+ bparams, bquery, fragment))
+ segments = bpath.split('/')[:-1] + path.split('/')
+ # XXX The stuff below is bogus in various ways...
+ if segments[-1] == '.':
+ segments[-1] = ''
+ while '.' in segments:
+ segments.remove('.')
+ while 1:
+ i = 1
+ n = len(segments) - 1
+ while i < n:
+ if (segments[i] == '..'
+ and segments[i-1] not in ('', '..')):
+ del segments[i-1:i+1]
+ break
+ i = i+1
+ else:
+ break
+ if segments == ['', '..']:
+ segments[-1] = ''
+ elif len(segments) >= 2 and segments[-1] == '..':
+ segments[-2:] = ['']
+ return urlunparse((scheme, netloc, '/'.join(segments),
+ params, query, fragment))
+
+def urldefrag(url):
+ """Removes any existing fragment from URL.
+
+ Returns a tuple of the defragmented URL and the fragment. If
+ the URL contained no fragments, the second element is the
+ empty string.
+ """
+ if '#' in url:
+ s, n, p, a, q, frag = urlparse(url)
+ defrag = urlunparse((s, n, p, a, q, ''))
+ return defrag, frag
+ else:
+ return url, ''
+
+
+test_input = """
+ http://a/b/c/d
+
+ g:h = <URL:g:h>
+ http:g = <URL:http://a/b/c/g>
+ http: = <URL:http://a/b/c/d>
+ g = <URL:http://a/b/c/g>
+ ./g = <URL:http://a/b/c/g>
+ g/ = <URL:http://a/b/c/g/>
+ /g = <URL:http://a/g>
+ //g = <URL:http://g>
+ ?y = <URL:http://a/b/c/d?y>
+ g?y = <URL:http://a/b/c/g?y>
+ g?y/./x = <URL:http://a/b/c/g?y/./x>
+ . = <URL:http://a/b/c/>
+ ./ = <URL:http://a/b/c/>
+ .. = <URL:http://a/b/>
+ ../ = <URL:http://a/b/>
+ ../g = <URL:http://a/b/g>
+ ../.. = <URL:http://a/>
+ ../../g = <URL:http://a/g>
+ ../../../g = <URL:http://a/../g>
+ ./../g = <URL:http://a/b/g>
+ ./g/. = <URL:http://a/b/c/g/>
+ /./g = <URL:http://a/./g>
+ g/./h = <URL:http://a/b/c/g/h>
+ g/../h = <URL:http://a/b/c/h>
+ http:g = <URL:http://a/b/c/g>
+ http: = <URL:http://a/b/c/d>
+ http:?y = <URL:http://a/b/c/d?y>
+ http:g?y = <URL:http://a/b/c/g?y>
+ http:g?y/./x = <URL:http://a/b/c/g?y/./x>
+"""
+
+def test():
+ import sys
+ base = ''
+ if sys.argv[1:]:
+ fn = sys.argv[1]
+ if fn == '-':
+ fp = sys.stdin
+ else:
+ fp = open(fn)
+ else:
+ try:
+ from cStringIO import StringIO
+ except ImportError:
+ from StringIO import StringIO
+ fp = StringIO(test_input)
+ while 1:
+ line = fp.readline()
+ if not line: break
+ words = line.split()
+ if not words:
+ continue
+ url = words[0]
+ parts = urlparse(url)
+ print '%-10s : %s' % (url, parts)
+ abs = urljoin(base, url)
+ if not base:
+ base = abs
+ wrapped = '<URL:%s>' % abs
+ print '%-10s = %s' % (url, wrapped)
+ if len(words) == 3 and words[1] == '=':
+ if wrapped != words[2]:
+ print 'EXPECTED', words[2], '!!!!!!!!!!'
+
+if __name__ == '__main__':
+ test()
=== modified file 'duplicity/util.py'
--- duplicity/util.py 2014-04-29 23:49:01 +0000
+++ duplicity/util.py 2014-10-15 12:12:04 +0000
@@ -23,8 +23,6 @@
Miscellaneous utilities.
"""
-from future_builtins import map
-
import errno
import os
import sys
@@ -88,7 +86,7 @@
"""
try:
return fn()
- except Exception as e:
+ except Exception, e:
if globals.ignore_errors:
log.Warn(_("IGNORED_ERROR: Warning: ignoring error as requested: %s: %s")
% (e.__class__.__name__, uexc(e)))
@@ -139,7 +137,7 @@
"""
try:
fn(filename)
- except OSError as ex:
+ except OSError, ex:
if ex.errno == errno.ENOENT:
pass
else:
=== modified file 'po/POTFILES.in'
--- po/POTFILES.in 2014-04-20 13:15:18 +0000
+++ po/POTFILES.in 2014-10-15 12:12:04 +0000
@@ -7,6 +7,7 @@
duplicity/selection.py
duplicity/globals.py
duplicity/commandline.py
+duplicity/urlparse_2_5.py
duplicity/dup_temp.py
duplicity/backend.py
duplicity/asyncscheduler.py
@@ -15,6 +16,7 @@
duplicity/collections.py
duplicity/log.py
duplicity/robust.py
+duplicity/static.py
duplicity/diffdir.py
duplicity/lazy.py
duplicity/backends/_cf_pyrax.py
@@ -49,6 +51,7 @@
duplicity/filechunkio.py
duplicity/dup_threading.py
duplicity/path.py
+duplicity/pexpect.py
duplicity/gpginterface.py
duplicity/dup_time.py
duplicity/gpg.py
=== modified file 'po/duplicity.pot'
--- po/duplicity.pot 2014-09-30 13:22:24 +0000
+++ po/duplicity.pot 2014-10-15 12:12:04 +0000
@@ -8,7 +8,11 @@
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: Kenneth Loafman <kenneth@xxxxxxxxxxx>\n"
+<<<<<<< TREE
"POT-Creation-Date: 2014-09-30 07:16-0500\n"
+=======
+"POT-Creation-Date: 2014-09-30 07:27-0500\n"
+>>>>>>> MERGE-SOURCE
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@xxxxxx>\n"
@@ -69,243 +73,243 @@
"Continuing restart on file %s."
msgstr ""
-#: ../bin/duplicity:297
+#: ../bin/duplicity:299
#, python-format
msgid "File %s was corrupted during upload."
msgstr ""
-#: ../bin/duplicity:331
+#: ../bin/duplicity:333
msgid ""
"Restarting backup, but current encryption settings do not match original "
"settings"
msgstr ""
-#: ../bin/duplicity:354
+#: ../bin/duplicity:356
#, python-format
msgid "Restarting after volume %s, file %s, block %s"
msgstr ""
-#: ../bin/duplicity:421
+#: ../bin/duplicity:423
#, python-format
msgid "Processed volume %d"
msgstr ""
-#: ../bin/duplicity:570
+#: ../bin/duplicity:572
msgid ""
"Fatal Error: Unable to start incremental backup. Old signatures not found "
"and incremental specified"
msgstr ""
-#: ../bin/duplicity:574
+#: ../bin/duplicity:576
msgid "No signatures found, switching to full backup."
msgstr ""
-#: ../bin/duplicity:588
+#: ../bin/duplicity:590
msgid "Backup Statistics"
msgstr ""
-#: ../bin/duplicity:693
+#: ../bin/duplicity:695
#, python-format
msgid "%s not found in archive, no files restored."
msgstr ""
-#: ../bin/duplicity:697
+#: ../bin/duplicity:699
msgid "No files found in archive - nothing restored."
msgstr ""
-#: ../bin/duplicity:730
+#: ../bin/duplicity:732
#, python-format
msgid "Processed volume %d of %d"
msgstr ""
-#: ../bin/duplicity:764
+#: ../bin/duplicity:766
#, python-format
msgid "Invalid data - %s hash mismatch for file:"
msgstr ""
-#: ../bin/duplicity:766
+#: ../bin/duplicity:768
#, python-format
msgid "Calculated hash: %s"
msgstr ""
-#: ../bin/duplicity:767
+#: ../bin/duplicity:769
#, python-format
msgid "Manifest hash: %s"
msgstr ""
-#: ../bin/duplicity:805
+#: ../bin/duplicity:807
#, python-format
msgid "Volume was signed by key %s, not %s"
msgstr ""
-#: ../bin/duplicity:835
+#: ../bin/duplicity:837
#, python-format
msgid "Verify complete: %s, %s."
msgstr ""
-#: ../bin/duplicity:836
+#: ../bin/duplicity:838
#, python-format
msgid "%d file compared"
msgid_plural "%d files compared"
msgstr[0] ""
msgstr[1] ""
-#: ../bin/duplicity:838
+#: ../bin/duplicity:840
#, python-format
msgid "%d difference found"
msgid_plural "%d differences found"
msgstr[0] ""
msgstr[1] ""
-#: ../bin/duplicity:857
+#: ../bin/duplicity:859
msgid "No extraneous files found, nothing deleted in cleanup."
msgstr ""
-#: ../bin/duplicity:862
+#: ../bin/duplicity:864
msgid "Deleting this file from backend:"
msgid_plural "Deleting these files from backend:"
msgstr[0] ""
msgstr[1] ""
-#: ../bin/duplicity:874
+#: ../bin/duplicity:876
msgid "Found the following file to delete:"
msgid_plural "Found the following files to delete:"
msgstr[0] ""
msgstr[1] ""
-#: ../bin/duplicity:878
+#: ../bin/duplicity:880
msgid "Run duplicity again with the --force option to actually delete."
msgstr ""
-#: ../bin/duplicity:921
+#: ../bin/duplicity:923
msgid "There are backup set(s) at time(s):"
msgstr ""
-#: ../bin/duplicity:923
+#: ../bin/duplicity:925
msgid "Which can't be deleted because newer sets depend on them."
msgstr ""
-#: ../bin/duplicity:927
+#: ../bin/duplicity:929
msgid ""
"Current active backup chain is older than specified time. However, it will "
"not be deleted. To remove all your backups, manually purge the repository."
msgstr ""
-#: ../bin/duplicity:933
+#: ../bin/duplicity:935
msgid "No old backup sets found, nothing deleted."
msgstr ""
-#: ../bin/duplicity:936
+#: ../bin/duplicity:938
msgid "Deleting backup chain at time:"
msgid_plural "Deleting backup chains at times:"
msgstr[0] ""
msgstr[1] ""
-#: ../bin/duplicity:947
+#: ../bin/duplicity:949
#, python-format
msgid "Deleting incremental signature chain %s"
msgstr ""
-#: ../bin/duplicity:949
+#: ../bin/duplicity:951
#, python-format
msgid "Deleting incremental backup chain %s"
msgstr ""
-#: ../bin/duplicity:952
+#: ../bin/duplicity:954
#, python-format
msgid "Deleting complete signature chain %s"
msgstr ""
-#: ../bin/duplicity:954
+#: ../bin/duplicity:956
#, python-format
msgid "Deleting complete backup chain %s"
msgstr ""
-#: ../bin/duplicity:960
+#: ../bin/duplicity:962
msgid "Found old backup chain at the following time:"
msgid_plural "Found old backup chains at the following times:"
msgstr[0] ""
msgstr[1] ""
-#: ../bin/duplicity:964
+#: ../bin/duplicity:966
msgid "Rerun command with --force option to actually delete."
msgstr ""
-#: ../bin/duplicity:1041
+#: ../bin/duplicity:1043
#, python-format
msgid "Deleting local %s (not authoritative at backend)."
msgstr ""
-#: ../bin/duplicity:1045
+#: ../bin/duplicity:1047
#, python-format
msgid "Unable to delete %s: %s"
msgstr ""
-#: ../bin/duplicity:1073 ../duplicity/dup_temp.py:263
+#: ../bin/duplicity:1075 ../duplicity/dup_temp.py:263
#, python-format
msgid "Failed to read %s: %s"
msgstr ""
-#: ../bin/duplicity:1087
+#: ../bin/duplicity:1089
#, python-format
msgid "Copying %s to local cache."
msgstr ""
-#: ../bin/duplicity:1135
+#: ../bin/duplicity:1137
msgid "Local and Remote metadata are synchronized, no sync needed."
msgstr ""
-#: ../bin/duplicity:1140
+#: ../bin/duplicity:1142
msgid "Synchronizing remote metadata to local cache..."
msgstr ""
-#: ../bin/duplicity:1155
+#: ../bin/duplicity:1157
msgid "Sync would copy the following from remote to local:"
msgstr ""
-#: ../bin/duplicity:1158
+#: ../bin/duplicity:1160
msgid "Sync would remove the following spurious local files:"
msgstr ""
-#: ../bin/duplicity:1201
+#: ../bin/duplicity:1203
msgid "Unable to get free space on temp."
msgstr ""
-#: ../bin/duplicity:1209
+#: ../bin/duplicity:1211
#, python-format
msgid "Temp space has %d available, backup needs approx %d."
msgstr ""
-#: ../bin/duplicity:1212
+#: ../bin/duplicity:1214
#, python-format
msgid "Temp has %d available, backup will use approx %d."
msgstr ""
-#: ../bin/duplicity:1220
+#: ../bin/duplicity:1222
msgid "Unable to get max open files."
msgstr ""
-#: ../bin/duplicity:1224
+#: ../bin/duplicity:1226
#, python-format
msgid ""
"Max open files of %s is too low, should be >= 1024.\n"
"Use 'ulimit -n 1024' or higher to correct.\n"
msgstr ""
-#: ../bin/duplicity:1273
+#: ../bin/duplicity:1275
msgid ""
"RESTART: The first volume failed to upload before termination.\n"
" Restart is impossible...starting backup from beginning."
msgstr ""
-#: ../bin/duplicity:1279
+#: ../bin/duplicity:1281
#, python-format
msgid ""
"RESTART: Volumes %d to %d failed to upload before termination.\n"
" Restarting backup at volume %d."
msgstr ""
-#: ../bin/duplicity:1286
+#: ../bin/duplicity:1288
#, python-format
msgid ""
"RESTART: Impossible backup state: manifest has %d vols, remote has %d vols.\n"
@@ -314,7 +318,7 @@
" backup then restart the backup from the beginning."
msgstr ""
-#: ../bin/duplicity:1308
+#: ../bin/duplicity:1310
msgid ""
"\n"
"PYTHONOPTIMIZE in the environment causes duplicity to fail to\n"
@@ -324,54 +328,54 @@
"See https://bugs.launchpad.net/duplicity/+bug/931175\n";
msgstr ""
-#: ../bin/duplicity:1399
+#: ../bin/duplicity:1401
#, python-format
msgid "Last %s backup left a partial set, restarting."
msgstr ""
-#: ../bin/duplicity:1403
+#: ../bin/duplicity:1405
#, python-format
msgid "Cleaning up previous partial %s backup set, restarting."
msgstr ""
-#: ../bin/duplicity:1414
+#: ../bin/duplicity:1416
msgid "Last full backup date:"
msgstr ""
-#: ../bin/duplicity:1416
+#: ../bin/duplicity:1418
msgid "Last full backup date: none"
msgstr ""
-#: ../bin/duplicity:1418
+#: ../bin/duplicity:1420
msgid "Last full backup is too old, forcing full backup"
msgstr ""
-#: ../bin/duplicity:1461
+#: ../bin/duplicity:1463
msgid ""
"When using symmetric encryption, the signing passphrase must equal the "
"encryption passphrase."
msgstr ""
-#: ../bin/duplicity:1514
+#: ../bin/duplicity:1516
msgid "INT intercepted...exiting."
msgstr ""
-#: ../bin/duplicity:1522
+#: ../bin/duplicity:1524
#, python-format
msgid "GPG error detail: %s"
msgstr ""
-#: ../bin/duplicity:1532
+#: ../bin/duplicity:1534
#, python-format
msgid "User error detail: %s"
msgstr ""
-#: ../bin/duplicity:1542
+#: ../bin/duplicity:1544
#, python-format
msgid "Backend error detail: %s"
msgstr ""
-#: ../bin/rdiffdir:56 ../duplicity/commandline.py:233
+#: ../bin/rdiffdir:56 ../duplicity/commandline.py:238
#, python-format
msgid "Error opening file %s"
msgstr ""
@@ -381,33 +385,33 @@
msgid "File %s already exists, will not overwrite."
msgstr ""
-#: ../duplicity/selection.py:121
+#: ../duplicity/selection.py:119
#, python-format
msgid "Skipping socket %s"
msgstr ""
-#: ../duplicity/selection.py:125
+#: ../duplicity/selection.py:123
#, python-format
msgid "Error initializing file %s"
msgstr ""
-#: ../duplicity/selection.py:129 ../duplicity/selection.py:150
+#: ../duplicity/selection.py:127 ../duplicity/selection.py:148
#, python-format
msgid "Error accessing possibly locked file %s"
msgstr ""
-#: ../duplicity/selection.py:165
+#: ../duplicity/selection.py:163
#, python-format
msgid "Warning: base %s doesn't exist, continuing"
msgstr ""
-#: ../duplicity/selection.py:168 ../duplicity/selection.py:186
-#: ../duplicity/selection.py:189
+#: ../duplicity/selection.py:166 ../duplicity/selection.py:184
+#: ../duplicity/selection.py:187
#, python-format
msgid "Selecting %s"
msgstr ""
-#: ../duplicity/selection.py:270
+#: ../duplicity/selection.py:268
#, python-format
msgid ""
"Fatal Error: The file specification\n"
@@ -418,14 +422,14 @@
"pattern (such as '**') which matches the base directory."
msgstr ""
-#: ../duplicity/selection.py:278
+#: ../duplicity/selection.py:276
#, python-format
msgid ""
"Fatal Error while processing expression\n"
"%s"
msgstr ""
-#: ../duplicity/selection.py:288
+#: ../duplicity/selection.py:286
#, python-format
msgid ""
"Last selection expression:\n"
@@ -435,49 +439,49 @@
"probably isn't what you meant."
msgstr ""
-#: ../duplicity/selection.py:313
+#: ../duplicity/selection.py:311
#, python-format
msgid "Reading filelist %s"
msgstr ""
-#: ../duplicity/selection.py:316
+#: ../duplicity/selection.py:314
#, python-format
msgid "Sorting filelist %s"
msgstr ""
-#: ../duplicity/selection.py:343
+#: ../duplicity/selection.py:341
#, python-format
msgid ""
"Warning: file specification '%s' in filelist %s\n"
"doesn't start with correct prefix %s. Ignoring."
msgstr ""
-#: ../duplicity/selection.py:347
+#: ../duplicity/selection.py:345
msgid "Future prefix errors will not be logged."
msgstr ""
-#: ../duplicity/selection.py:363
+#: ../duplicity/selection.py:361
#, python-format
msgid "Error closing filelist %s"
msgstr ""
-#: ../duplicity/selection.py:430
+#: ../duplicity/selection.py:428
#, python-format
msgid "Reading globbing filelist %s"
msgstr ""
-#: ../duplicity/selection.py:463
+#: ../duplicity/selection.py:461
#, python-format
msgid "Error compiling regular expression %s"
msgstr ""
-#: ../duplicity/selection.py:479
+#: ../duplicity/selection.py:477
msgid ""
"Warning: exclude-device-files is not the first selector.\n"
"This may not be what you intended"
msgstr ""
-#: ../duplicity/commandline.py:70
+#: ../duplicity/commandline.py:68
#, python-format
msgid ""
"Warning: Option %s is pending deprecation and will be removed in a future "
@@ -485,11 +489,22 @@
"Use of default filenames is strongly suggested."
msgstr ""
+#: ../duplicity/commandline.py:216
+#, python-format
+msgid "Unable to load gio backend: %s"
+msgstr ""
+
#. Used in usage help to represent a Unix-style path name. Example:
#. --archive-dir <path>
+<<<<<<< TREE
#: ../duplicity/commandline.py:254 ../duplicity/commandline.py:264
#: ../duplicity/commandline.py:281 ../duplicity/commandline.py:347
#: ../duplicity/commandline.py:552 ../duplicity/commandline.py:768
+=======
+#: ../duplicity/commandline.py:259 ../duplicity/commandline.py:269
+#: ../duplicity/commandline.py:286 ../duplicity/commandline.py:352
+#: ../duplicity/commandline.py:557 ../duplicity/commandline.py:773
+>>>>>>> MERGE-SOURCE
msgid "path"
msgstr ""
@@ -499,9 +514,15 @@
#. --hidden-encrypt-key <gpg_key_id>
#. Used in usage help to represent an ID for a GnuPG key. Example:
#. --encrypt-key <gpg_key_id>
+<<<<<<< TREE
#: ../duplicity/commandline.py:276 ../duplicity/commandline.py:283
#: ../duplicity/commandline.py:369 ../duplicity/commandline.py:533
#: ../duplicity/commandline.py:741
+=======
+#: ../duplicity/commandline.py:281 ../duplicity/commandline.py:288
+#: ../duplicity/commandline.py:372 ../duplicity/commandline.py:538
+#: ../duplicity/commandline.py:746
+>>>>>>> MERGE-SOURCE
msgid "gpg-key-id"
msgstr ""
@@ -509,43 +530,66 @@
#. matching one or more files, as described in the documentation.
#. Example:
#. --exclude <shell_pattern>
+<<<<<<< TREE
#: ../duplicity/commandline.py:291 ../duplicity/commandline.py:395
#: ../duplicity/commandline.py:791
+=======
+#: ../duplicity/commandline.py:296 ../duplicity/commandline.py:398
+#: ../duplicity/commandline.py:796
+>>>>>>> MERGE-SOURCE
msgid "shell_pattern"
msgstr ""
#. Used in usage help to represent the name of a file. Example:
#. --log-file <filename>
+<<<<<<< TREE
#: ../duplicity/commandline.py:297 ../duplicity/commandline.py:304
#: ../duplicity/commandline.py:309 ../duplicity/commandline.py:397
#: ../duplicity/commandline.py:402 ../duplicity/commandline.py:413
#: ../duplicity/commandline.py:737
+=======
+#: ../duplicity/commandline.py:302 ../duplicity/commandline.py:309
+#: ../duplicity/commandline.py:314 ../duplicity/commandline.py:400
+#: ../duplicity/commandline.py:405 ../duplicity/commandline.py:416
+#: ../duplicity/commandline.py:742
+>>>>>>> MERGE-SOURCE
msgid "filename"
msgstr ""
#. Used in usage help to represent a regular expression (regexp).
-#: ../duplicity/commandline.py:316 ../duplicity/commandline.py:404
+#: ../duplicity/commandline.py:321 ../duplicity/commandline.py:407
msgid "regular_expression"
msgstr ""
#. Used in usage help to represent a time spec for a previous
#. point in time, as described in the documentation. Example:
#. duplicity remove-older-than time [options] target_url
+<<<<<<< TREE
#: ../duplicity/commandline.py:359 ../duplicity/commandline.py:475
#: ../duplicity/commandline.py:823
+=======
+#: ../duplicity/commandline.py:364 ../duplicity/commandline.py:478
+#: ../duplicity/commandline.py:828
+>>>>>>> MERGE-SOURCE
msgid "time"
msgstr ""
#. Used in usage help. (Should be consistent with the "Options:"
#. header.) Example:
#. duplicity [full|incremental] [options] source_dir target_url
+<<<<<<< TREE
#: ../duplicity/commandline.py:365 ../duplicity/commandline.py:455
#: ../duplicity/commandline.py:478 ../duplicity/commandline.py:544
#: ../duplicity/commandline.py:756
+=======
+#: ../duplicity/commandline.py:368 ../duplicity/commandline.py:458
+#: ../duplicity/commandline.py:481 ../duplicity/commandline.py:549
+#: ../duplicity/commandline.py:761
+>>>>>>> MERGE-SOURCE
msgid "options"
msgstr ""
-#: ../duplicity/commandline.py:380
+#: ../duplicity/commandline.py:383
#, python-format
msgid ""
"Running in 'ignore errors' mode due to %s; please re-consider if this was "
@@ -553,156 +597,257 @@
msgstr ""
#. Used in usage help to represent an imap mailbox
-#: ../duplicity/commandline.py:393
+#: ../duplicity/commandline.py:396
msgid "imap_mailbox"
msgstr ""
-#: ../duplicity/commandline.py:407
+#: ../duplicity/commandline.py:410
msgid "file_descriptor"
msgstr ""
#. Used in usage help to represent a desired number of
#. something. Example:
#. --num-retries <number>
+<<<<<<< TREE
#: ../duplicity/commandline.py:418 ../duplicity/commandline.py:440
#: ../duplicity/commandline.py:452 ../duplicity/commandline.py:461
#: ../duplicity/commandline.py:499 ../duplicity/commandline.py:504
#: ../duplicity/commandline.py:508 ../duplicity/commandline.py:582
#: ../duplicity/commandline.py:751
+=======
+#: ../duplicity/commandline.py:421 ../duplicity/commandline.py:443
+#: ../duplicity/commandline.py:455 ../duplicity/commandline.py:464
+#: ../duplicity/commandline.py:502 ../duplicity/commandline.py:507
+#: ../duplicity/commandline.py:511 ../duplicity/commandline.py:587
+#: ../duplicity/commandline.py:756
+>>>>>>> MERGE-SOURCE
msgid "number"
msgstr ""
#. Used in usage help (noun)
-#: ../duplicity/commandline.py:421
+#: ../duplicity/commandline.py:424
msgid "backup name"
msgstr ""
#. noun
+<<<<<<< TREE
#: ../duplicity/commandline.py:517 ../duplicity/commandline.py:520
#: ../duplicity/commandline.py:722
+=======
+#: ../duplicity/commandline.py:522 ../duplicity/commandline.py:525
+#: ../duplicity/commandline.py:727
+>>>>>>> MERGE-SOURCE
msgid "command"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:523
+=======
+#: ../duplicity/commandline.py:528
+>>>>>>> MERGE-SOURCE
msgid "pyrax|cloudfiles"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:541
+=======
+#: ../duplicity/commandline.py:546
+>>>>>>> MERGE-SOURCE
msgid "paramiko|pexpect"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:547
+=======
+#: ../duplicity/commandline.py:552
+>>>>>>> MERGE-SOURCE
msgid "pem formatted bundle of certificate authorities"
msgstr ""
#. Used in usage help. Example:
#. --timeout <seconds>
+<<<<<<< TREE
#: ../duplicity/commandline.py:557 ../duplicity/commandline.py:785
+=======
+#: ../duplicity/commandline.py:562 ../duplicity/commandline.py:790
+>>>>>>> MERGE-SOURCE
msgid "seconds"
msgstr ""
#. abbreviation for "character" (noun)
+<<<<<<< TREE
#: ../duplicity/commandline.py:563 ../duplicity/commandline.py:719
+=======
+#: ../duplicity/commandline.py:568 ../duplicity/commandline.py:724
+>>>>>>> MERGE-SOURCE
msgid "char"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:685
+=======
+#: ../duplicity/commandline.py:690
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Using archive dir: %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:686
+=======
+#: ../duplicity/commandline.py:691
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Using backup name: %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:693
+=======
+#: ../duplicity/commandline.py:698
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Command line error: %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:694
+=======
+#: ../duplicity/commandline.py:699
+>>>>>>> MERGE-SOURCE
msgid "Enter 'duplicity --help' for help screen."
msgstr ""
#. Used in usage help to represent a Unix-style path name. Example:
#. rsync://user[:password]@other_host[:port]//absolute_path
+<<<<<<< TREE
#: ../duplicity/commandline.py:707
+=======
+#: ../duplicity/commandline.py:712
+>>>>>>> MERGE-SOURCE
msgid "absolute_path"
msgstr ""
#. Used in usage help. Example:
#. tahoe://alias/some_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:711
+=======
+#: ../duplicity/commandline.py:716
+>>>>>>> MERGE-SOURCE
msgid "alias"
msgstr ""
#. Used in help to represent a "bucket name" for Amazon Web
#. Services' Simple Storage Service (S3). Example:
#. s3://other.host/bucket_name[/prefix]
+<<<<<<< TREE
#: ../duplicity/commandline.py:716
+=======
+#: ../duplicity/commandline.py:721
+>>>>>>> MERGE-SOURCE
msgid "bucket_name"
msgstr ""
#. Used in usage help to represent the name of a container in
#. Amazon Web Services' Cloudfront. Example:
#. cf+http://container_name
+<<<<<<< TREE
#: ../duplicity/commandline.py:727
+=======
+#: ../duplicity/commandline.py:732
+>>>>>>> MERGE-SOURCE
msgid "container_name"
msgstr ""
#. noun
+<<<<<<< TREE
#: ../duplicity/commandline.py:730
+=======
+#: ../duplicity/commandline.py:735
+>>>>>>> MERGE-SOURCE
msgid "count"
msgstr ""
#. Used in usage help to represent the name of a file directory
+<<<<<<< TREE
#: ../duplicity/commandline.py:733
+=======
+#: ../duplicity/commandline.py:738
+>>>>>>> MERGE-SOURCE
msgid "directory"
msgstr ""
#. Used in usage help, e.g. to represent the name of a code
#. module. Example:
#. rsync://user[:password]@other.host[:port]::/module/some_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:746
+=======
+#: ../duplicity/commandline.py:751
+>>>>>>> MERGE-SOURCE
msgid "module"
msgstr ""
#. Used in usage help to represent an internet hostname. Example:
#. ftp://user[:password]@other.host[:port]/some_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:760
+=======
+#: ../duplicity/commandline.py:765
+>>>>>>> MERGE-SOURCE
msgid "other.host"
msgstr ""
#. Used in usage help. Example:
#. ftp://user[:password]@other.host[:port]/some_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:764
+=======
+#: ../duplicity/commandline.py:769
+>>>>>>> MERGE-SOURCE
msgid "password"
msgstr ""
#. Used in usage help to represent a TCP port number. Example:
#. ftp://user[:password]@other.host[:port]/some_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:772
+=======
+#: ../duplicity/commandline.py:777
+>>>>>>> MERGE-SOURCE
msgid "port"
msgstr ""
#. Used in usage help. This represents a string to be used as a
#. prefix to names for backup files created by Duplicity. Example:
#. s3://other.host/bucket_name[/prefix]
+<<<<<<< TREE
#: ../duplicity/commandline.py:777
+=======
+#: ../duplicity/commandline.py:782
+>>>>>>> MERGE-SOURCE
msgid "prefix"
msgstr ""
#. Used in usage help to represent a Unix-style path name. Example:
#. rsync://user[:password]@other.host[:port]/relative_path
+<<<<<<< TREE
#: ../duplicity/commandline.py:781
+=======
+#: ../duplicity/commandline.py:786
+>>>>>>> MERGE-SOURCE
msgid "relative_path"
msgstr ""
#. Used in usage help to represent the name of a single file
#. directory or a Unix-style path to a directory. Example:
#. file:///some_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:796
+=======
+#: ../duplicity/commandline.py:801
+>>>>>>> MERGE-SOURCE
msgid "some_dir"
msgstr ""
@@ -710,14 +855,22 @@
#. directory or a Unix-style path to a directory where files will be
#. coming FROM. Example:
#. duplicity [full|incremental] [options] source_dir target_url
+<<<<<<< TREE
#: ../duplicity/commandline.py:802
+=======
+#: ../duplicity/commandline.py:807
+>>>>>>> MERGE-SOURCE
msgid "source_dir"
msgstr ""
#. Used in usage help to represent a URL files will be coming
#. FROM. Example:
#. duplicity [restore] [options] source_url target_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:807
+=======
+#: ../duplicity/commandline.py:812
+>>>>>>> MERGE-SOURCE
msgid "source_url"
msgstr ""
@@ -725,75 +878,127 @@
#. directory or a Unix-style path to a directory. where files will be
#. going TO. Example:
#. duplicity [restore] [options] source_url target_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:813
+=======
+#: ../duplicity/commandline.py:818
+>>>>>>> MERGE-SOURCE
msgid "target_dir"
msgstr ""
#. Used in usage help to represent a URL files will be going TO.
#. Example:
#. duplicity [full|incremental] [options] source_dir target_url
+<<<<<<< TREE
#: ../duplicity/commandline.py:818
+=======
+#: ../duplicity/commandline.py:823
+>>>>>>> MERGE-SOURCE
msgid "target_url"
msgstr ""
#. Used in usage help to represent a user name (i.e. login).
#. Example:
#. ftp://user[:password]@other.host[:port]/some_dir
+<<<<<<< TREE
#: ../duplicity/commandline.py:828
+=======
+#: ../duplicity/commandline.py:833
+>>>>>>> MERGE-SOURCE
msgid "user"
msgstr ""
#. Header in usage help
+<<<<<<< TREE
#: ../duplicity/commandline.py:845
+=======
+#: ../duplicity/commandline.py:850
+>>>>>>> MERGE-SOURCE
msgid "Backends and their URL formats:"
msgstr ""
#. Header in usage help
+<<<<<<< TREE
#: ../duplicity/commandline.py:871
+=======
+#: ../duplicity/commandline.py:875
+>>>>>>> MERGE-SOURCE
msgid "Commands:"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:895
+=======
+#: ../duplicity/commandline.py:899
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Specified archive directory '%s' does not exist, or is not a directory"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:904
+=======
+#: ../duplicity/commandline.py:908
+>>>>>>> MERGE-SOURCE
#, python-format
msgid ""
"Sign key should be an 8 character hex string, like 'AA0E73D2'.\n"
"Received '%s' instead."
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:964
+=======
+#: ../duplicity/commandline.py:968
+>>>>>>> MERGE-SOURCE
#, python-format
msgid ""
"Restore destination directory %s already exists.\n"
"Will not overwrite."
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:969
+=======
+#: ../duplicity/commandline.py:973
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Verify directory %s does not exist"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:975
+=======
+#: ../duplicity/commandline.py:979
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Backup source directory %s does not exist."
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:1004
+=======
+#: ../duplicity/commandline.py:1008
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Command line warning: %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:1004
+=======
+#: ../duplicity/commandline.py:1008
+>>>>>>> MERGE-SOURCE
msgid ""
"Selection options --exclude/--include\n"
"currently work only when backing up,not restoring."
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:1052
+=======
+#: ../duplicity/commandline.py:1056
+>>>>>>> MERGE-SOURCE
#, python-format
msgid ""
"Bad URL '%s'.\n"
@@ -801,40 +1006,76 @@
"\"file:///usr/local\". See the man page for more information."
msgstr ""
+<<<<<<< TREE
#: ../duplicity/commandline.py:1077
+=======
+#: ../duplicity/commandline.py:1081
+>>>>>>> MERGE-SOURCE
msgid "Main action: "
msgstr ""
+<<<<<<< TREE
#: ../duplicity/backend.py:111
+=======
+#: ../duplicity/backend.py:88
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Import of %s %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/backend.py:213
+=======
+#: ../duplicity/backend.py:165
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Could not initialize backend: %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/backend.py:374
+=======
+#: ../duplicity/backend.py:320
+#, python-format
+msgid "Attempt %s failed: %s: %s"
+msgstr ""
+
+#: ../duplicity/backend.py:322 ../duplicity/backend.py:352
+#: ../duplicity/backend.py:359
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Backtrace of previous error: %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/backend.py:389
+=======
+#: ../duplicity/backend.py:350
+#, python-format
+msgid "Attempt %s failed. %s: %s"
+msgstr ""
+
+#: ../duplicity/backend.py:361
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Giving up after %s attempts. %s: %s"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/backend.py:393
#, python-format
msgid "Attempt %s failed. %s: %s"
msgstr ""
#: ../duplicity/backend.py:476
+=======
+#: ../duplicity/backend.py:546 ../duplicity/backend.py:570
+>>>>>>> MERGE-SOURCE
#, python-format
msgid "Reading results of '%s'"
msgstr ""
+<<<<<<< TREE
#: ../duplicity/backend.py:503
#, python-format
msgid "Writing %s"
@@ -844,6 +1085,28 @@
#, python-format
msgid "File %s not found locally after get from backend"
msgstr ""
+=======
+#: ../duplicity/backend.py:585
+#, python-format
+msgid "Running '%s' failed with code %d (attempt #%d)"
+msgid_plural "Running '%s' failed with code %d (attempt #%d)"
+msgstr[0] ""
+msgstr[1] ""
+
+#: ../duplicity/backend.py:589
+#, python-format
+msgid ""
+"Error is:\n"
+"%s"
+msgstr ""
+
+#: ../duplicity/backend.py:591
+#, python-format
+msgid "Giving up trying to execute '%s' after %d attempt"
+msgid_plural "Giving up trying to execute '%s' after %d attempts"
+msgstr[0] ""
+msgstr[1] ""
+>>>>>>> MERGE-SOURCE
#: ../duplicity/asyncscheduler.py:66
#, python-format
@@ -881,142 +1144,142 @@
msgid "task execution done (success: %s)"
msgstr ""
-#: ../duplicity/patchdir.py:76 ../duplicity/patchdir.py:81
+#: ../duplicity/patchdir.py:74 ../duplicity/patchdir.py:79
#, python-format
msgid "Patching %s"
msgstr ""
-#: ../duplicity/patchdir.py:510
+#: ../duplicity/patchdir.py:508
#, python-format
msgid "Error '%s' patching %s"
msgstr ""
-#: ../duplicity/patchdir.py:582
+#: ../duplicity/patchdir.py:581
#, python-format
msgid "Writing
