duplicity-team team mailing list archive
-
duplicity-team team
-
Mailing list archive
-
Message #01627
[Merge] lp:~kvdveer/duplicity/fix-for-1155500 into lp:duplicity
Ondergetekende has proposed merging lp:~kvdveer/duplicity/fix-for-1155500 into lp:duplicity.
Requested reviews:
duplicity-team (duplicity-team)
For more details, see:
https://code.launchpad.net/~kvdveer/duplicity/fix-for-1155500/+merge/153534
Implemented Openstack Swift
--
https://code.launchpad.net/~kvdveer/duplicity/fix-for-1155500/+merge/153534
Your team duplicity-team is requested to review the proposed merge of lp:~kvdveer/duplicity/fix-for-1155500 into lp:duplicity.
=== modified file 'CHANGELOG'
--- CHANGELOG 2013-02-18 00:33:16 +0000
+++ CHANGELOG 2013-03-15 11:40:35 +0000
@@ -19,6 +19,7 @@
* Merged in lp:~ed.so/duplicity/verify.data
- add switch --compare-data, to selectively enable formerly always disabled
data comparison on verify runs
+* Implemented an Openstack Swift backend.
New in v0.6.21 (2013/01/23)
=== modified file 'Changelog.GNU'
--- Changelog.GNU 2013-02-18 00:33:16 +0000
+++ Changelog.GNU 2013-03-15 11:40:35 +0000
@@ -1,3 +1,7 @@
+2013-03-15 Koert van der Veer <koert@xxxxxxxxxxxx>
+
+ * Implemented an Openstack swift backend.
+
2013-02-17 Kenneth Loafman <kenneth@xxxxxxxxxxx>
* Applied patches from Laszlo Ersek to rdiffdir to "consume a chain of sigtar
=== modified file 'bin/duplicity.1'
--- bin/duplicity.1 2013-02-13 16:29:48 +0000
+++ bin/duplicity.1 2013-03-15 11:40:35 +0000
@@ -9,9 +9,9 @@
duplicity \- Encrypted incremental backup to local or remote storage.
.SH REQUIREMENTS
-Duplicity requires a POSIX-like operating system with a
+Duplicity requires a POSIX-like operating system with a
.B python
-interpreter version 2.4+ installed.
+interpreter version 2.4+ installed.
It is best used under GNU/Linux.
Some backends also require additional components (probably available as packages for your specific platform):
@@ -24,6 +24,10 @@
.B Cloud Files Python API
- http://www.rackspace.com/knowledge_center/article/python-api-installation-for-cloud-files
.TP
+.BR "swift backend"
+.B Python swiftclient Library
+- https://pypi.python.org/pypi/python-swiftclient
+.TP
.B "ftp backend"
.B NcFTP Client
- http://www.ncftp.com/
@@ -47,18 +51,18 @@
.B rsync client binary
- http://rsync.samba.org/
.PP
-There are two
+There are two
.B ssh backends
-for scp/sftp/ssh access (also see
+for scp/sftp/ssh access (also see
.BR "A NOTE ON SSH BACKENDS" ).
.TP
.BR "ssh paramiko backend" " (enabled by default)"
.B paramiko
-(SSH2 for python)
+(SSH2 for python)
- http://www.lag.net/paramiko/
.br
.B pycrypto
-(Python Cryptography Toolkit)
+(Python Cryptography Toolkit)
- http://www.dlitz.net/software/pycrypto/
.TP
.B ssh pexpect backend
@@ -79,7 +83,7 @@
for ssl certificate verification of HTTPS connections
- http://curl.haxx.se/docs/caextract.html
.br
-(also see
+(also see
.BR "A NOTE ON SSL CERTIFICATE VERIFICATION" ).
.TP
.B "dpbx backend"
@@ -374,7 +378,7 @@
.BI "--encrypt-key " key-id
When backing up, encrypt to the given public key, instead of using
symmetric (traditional) encryption. Can be specified multiple times.
-The key-id can be given in any of the formats supported by GnuPG; see
+The key-id can be given in any of the formats supported by GnuPG; see
.BR gpg (1),
section "HOW TO SPECIFY A USER ID" for details.
@@ -391,9 +395,9 @@
.TP
.BI "--encrypt-sign-key " key-id
Convenience parameter. Same as
-.BR --encrypt-key
+.BR --encrypt-key
.IR key-id
-.BR --sign-key
+.BR --sign-key
.IR "key-id" .
.TP
@@ -505,8 +509,8 @@
.BR --encrypt-key ,
but it hides user's key id from encrypted file. It uses the gpg's
.B --hidden-recipient
-command to obfuscate the owner of the backup. On restore, gpg will
-automatically try all available secret keys in order to decrypt the
+command to obfuscate the owner of the backup. On restore, gpg will
+automatically try all available secret keys in order to decrypt the
backup. See gpg(1) for more details.
@@ -634,9 +638,9 @@
.TP
.B --numeric-owner
-On restore always use the numeric uid/gid from the archive and not the
+On restore always use the numeric uid/gid from the archive and not the
archived user/group names, which is the default behaviour.
-Recommended for restoring from live cds which might have the users with
+Recommended for restoring from live cds which might have the users with
identical names but different uids/gids.
.TP
@@ -718,7 +722,7 @@
.B (only ssh pexpect backend)
The
.I command
-will be used instead of "sftp".
+will be used instead of "sftp".
.br
See also
.B "A NOTE ON SSH BACKENDS"
@@ -734,15 +738,15 @@
.TP
.BI "--sign-key " key-id
-This option can be used when backing up, restoring or verifying.
+This option can be used when backing up, restoring or verifying.
When backing up, all backup files will be signed with keyid
.IR key .
When restoring, duplicity will signal an error if any remote file is
-not signed with the given key-id. The key-id can be givein in any of
-the formats supported by GnuPG; see
+not signed with the given key-id. The key-id can be givein in any of
+the formats supported by GnuPG; see
.BR gpg (1),
section "HOW TO SPECIFY A USER ID" for details.
-Should be specified only once because currently only
+Should be specified only once because currently only
.B one
signing key is supported. Last entry overrides all other entries.
.br
@@ -751,15 +755,15 @@
.TP
.B --ssh-askpass
-Tells the ssh backend to prompt the user for the remote system password,
+Tells the ssh backend to prompt the user for the remote system password,
if it was not defined in target url and no FTP_PASSWORD env var is set.
This password is also used for passphrase-protected ssh keys.
.TP
.BI "--ssh-backend " backend
-Allows the explicit selection of a ssh backend. Defaults to
+Allows the explicit selection of a ssh backend. Defaults to
.BR paramiko .
-Alternatively you might choose
+Alternatively you might choose
.BR pexpect .
.br
See also
@@ -900,12 +904,12 @@
prompted for the passphrase.
.TP
.B SIGN_PASSPHRASE
-The passphrase to be used for
+The passphrase to be used for
.BR --sign-key .
-If ommitted
+If ommitted
.B and
-sign key is also one of the keys to encrypt against
-.B PASSPHRASE
+sign key is also one of the keys to encrypt against
+.B PASSPHRASE
will be reused instead.
Otherwise, if passphrase is needed but not set the user will be prompted for it.
@@ -920,8 +924,8 @@
It is not recommended to expose the password on the command line since
it could be revealed to anyone with permissions to do process listings,
it is permitted however.
-Consider setting the environment variable
-.B FTP_PASSWORD
+Consider setting the environment variable
+.B FTP_PASSWORD
instead, which is used by most, if not all backends, regardless of it's name.
.PP
In protocols that support it, the path may be preceded by a single
@@ -1141,8 +1145,8 @@
.IP 2.
the file is inside a directory matched by the option.
.PP
-Conversely, the
-.B "--include "
+Conversely, the
+.B "--include "
pattern matches a file if:
.IP 1.
.I pattern
@@ -1168,7 +1172,7 @@
specifies that /usr, /usr/local, /usr/local/lib, and
/usr/local/lib/netscape (but not /usr/doc) all be backed up. Thus you
don't have to worry about including parent directories to make sure
-that included subdirectories have somewhere to go.
+that included subdirectories have somewhere to go.
Finally,
.br
@@ -1226,7 +1230,7 @@
- /var
.RE
-then
+then
.B "--include-filelist list.txt"
would include /usr, /usr/local, and
/usr/local/bin. It would exclude /usr/local/doc,
@@ -1257,9 +1261,9 @@
- **
.RE
-Then
-.B "--include-globbing-filelist globbing-list.txt"
-would be exactly the same as specifying
+Then
+.B "--include-globbing-filelist globbing-list.txt"
+would be exactly the same as specifying
.B "--include dir/foo --include dir/bar --exclude **"
on the command line.
@@ -1290,8 +1294,8 @@
protocol.
The backend requires python-cloudfiles to be installed on the system.
-See
-.B REQUIREMENTS
+See
+.B REQUIREMENTS
above.
It uses three environment variables for authentification:
@@ -1299,13 +1303,33 @@
.BR CLOUDFILES_APIKEY " (required),"
.BR CLOUDFILES_AUTHURL " (optional)"
-If
-.B CLOUDFILES_AUTHURL
+If
+.B CLOUDFILES_AUTHURL
is unspecified it will default to the value
-provided by python-cloudfiles, which points to rackspace, hence this value
-.I must
+provided by python-cloudfiles, which points to rackspace, hence this value
+.I must
be set in order to use other cloud files providers.
+.SH A NOTE ON OPENSTACK SWIFT
+The backend requires python-swiftclient to be installed on the system.
+See
+.B REQUIREMENTS
+above.
+
+It uses three environment variables for authentification:
+.BR SWIFT_USERNAME " (required),"
+.BR SWIFT_PASSWORD " (required),"
+.BR SWIFT_AUTHURL " (required)"
+.BR SWIFT_AUTHVERSION " (optional)"
+
+If
+.B SWIFT_AUTHVERSION
+is unspecified, it will be auto-detected from the URL provided. Use
+.I 1
+for pre-keystone setups, use
+.I 2
+for any recent setup.
+
.SH A NOTE ON EUROPEAN S3 BUCKETS
Amazon S3 provides the ability to choose the location of a bucket upon
its creation. The purpose is to enable the user to choose a location
@@ -1355,7 +1379,7 @@
will distinguish between different backups.
.SH A NOTE ON SSH BACKENDS
-The
+The
.I ssh backends
support
.I sftp
@@ -1363,7 +1387,7 @@
.I scp/ssh
transport protocols.
This is a known user-confusing issue as these are fundamentally different.
-If you plan to access your backend via one of those please inform yourself
+If you plan to access your backend via one of those please inform yourself
about the requirements for a server to support
.IR sftp " or"
.I scp/ssh
@@ -1375,15 +1399,15 @@
.BR --use-scp ,
.BR --ssh-askpass " and"
.BR --ssh-options "."
-Only the
+Only the
.B pexpect
backend allows to define
.BR --scp-command " and"
.BR --sftp-command .
.PP
.BR "SSH paramiko backend " "(selected by default)"
-is a complete reimplementation of ssh protocols natively in python. Advantages
-are speed and maintainability. Minor disadvantage is that extra packages are
+is a complete reimplementation of ssh protocols natively in python. Advantages
+are speed and maintainability. Minor disadvantage is that extra packages are
needed as listed in
.B REQUIREMENTS
above. In
@@ -1406,12 +1430,12 @@
for all four supported
operations, unless the
.I --use-scp
-option is used to revert to old behavior.
+option is used to revert to old behavior.
.PP
.B Why use sftp instead of scp?
The change to sftp was made in order to allow the remote system to chroot the backup,
-thus providing better security and because it does not suffer from shell quoting issues like scp.
-Scp also does not support any kind of file listing, so sftp or ssh access will always be needed
+thus providing better security and because it does not suffer from shell quoting issues like scp.
+Scp also does not support any kind of file listing, so sftp or ssh access will always be needed
in addition for this backend mode to work properly. Sftp does not have these limitations but needs
an sftp service running on the backend server, which is sometimes not an option.
@@ -1419,7 +1443,7 @@
Certificate verification as implemented right now [01.2013] only in the webdav backend needs a file
based database of certification authority certificates (cacert file). It has to be a
.B PEM
-formatted text file as currently provided by the
+formatted text file as currently provided by the
.B CURL
project. See
.PP
@@ -1444,9 +1468,9 @@
.PP
Finally there is the
.B --ssl-no-check-certificate
-option to disable certificate verification alltogether, in case some ssl library
-is missing or verification is not wanted. Use it with care, as even with self signed
-servers manually providing the private ca certificate is definitely the safer option.
+option to disable certificate verification alltogether, in case some ssl library
+is missing or verification is not wanted. Use it with care, as even with self signed
+servers manually providing the private ca certificate is definitely the safer option.
.SH A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING
Signing and symmetrically encrypt at the same time with the gpg binary on the
@@ -1459,7 +1483,7 @@
.PP
2. Use a
.BI PASSPHRASE
-for symmetric encryption of your choice but the signing key has an
+for symmetric encryption of your choice but the signing key has an
.B empty
passphrase.
.PP
@@ -1469,15 +1493,15 @@
.SH A NOTE ON UBUNTU ONE
-To use Ubuntu One you must have an Ubuntu One OAuth access token. Such
-OAuth tokens have a practically unlimited lifetime; you can have multiple
+To use Ubuntu One you must have an Ubuntu One OAuth access token. Such
+OAuth tokens have a practically unlimited lifetime; you can have multiple
active tokens and you can revoke tokens using the Ubuntu One web interface.
.PP
-Duplicity expects the token in the environment variable
+Duplicity expects the token in the environment variable
.B FTP_PASSWORD
(in the format "consumer_key:consumer_secret:token:token_secret"). If no
token is present, duplicity asks for your Ubuntu One email address and password
-and requests an access token from the Ubuntu SSO service. The newly
+and requests an access token from the Ubuntu SSO service. The newly
acquired token is then printed to the console.
.PP
See https://one.ubuntu.com/ for more information about Ubuntu One.
@@ -1562,7 +1586,7 @@
Most backends were contributed individually.
Information about their authorship may be found in the according file's header.
.br
-Also we'd like to thank everybody posting issue to the mailing list or on
+Also we'd like to thank everybody posting issue to the mailing list or on
launchpad, sending in patches or contributing otherwise. Duplicity wouldn't
be as stable and useful if it weren't for you.
=== added file 'duplicity/backends/swiftbackend.py'
--- duplicity/backends/swiftbackend.py 1970-01-01 00:00:00 +0000
+++ duplicity/backends/swiftbackend.py 2013-03-15 11:40:35 +0000
@@ -0,0 +1,139 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2013 Koert van der Veer <koert@xxxxxxxxxxxx>
+#
+# This file is part of duplicity.
+#
+# Duplicity is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# Duplicity is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import os
+#import time
+
+import duplicity.backend
+#from duplicity import globals
+from duplicity import log
+from duplicity.errors import BackendException
+#from duplicity.util import exception_traceback
+#from duplicity.backend import retry
+
+try:
+ import swiftclient
+except ImportError:
+ swiftclient = None
+
+class SwiftBackend(duplicity.backend.Backend):
+ """
+ Backend for OpenStack Swift
+ """
+ def __init__(self, parsed_url):
+ if not swiftclient:
+ raise BackendException("This backend requires python-swiftclient")
+
+ conn_kwargs = {}
+
+ if not os.environ.has_key('SWIFT_USERNAME'):
+ raise BackendException('SWIFT_USERNAME environment variable not set.')
+
+ if not os.environ.has_key('SWIFT_PASSWORD'):
+ raise BackendException('SWIFT_PASSWORD environment variable not set.')
+
+ if not os.environ.has_key('SWIFT_AUTHURL'):
+ raise BackendException('SWIFT_AUTHURL environment variable not set.')
+
+ conn_kwargs['user'] = os.environ['SWIFT_USERNAME']
+ conn_kwargs['key'] = os.environ['SWIFT_PASSWORD']
+ conn_kwargs['authurl'] = os.environ['SWIFT_AUTHURL']
+
+ if os.environ.has_key('SWIFT_AUTHVERSION'):
+ conn_kwargs['auth_version'] = os.environ['SWIFT_AUTHVERSION']
+ elif '/v2.0' in conn_kwargs['authurl']:
+ # Url looks like a versioned keystone url, assume keystone auth.
+ conn_kwargs['auth_version'] = '2'
+
+ if os.environ.has_key('SWIFT_TENANTNAME'):
+ conn_kwargs['tenant_name'] = os.environ['SWIFT_TENANTNAME']
+ else:
+ raise BackendException('SWIFT_TENANTNAME environment variable not set.')
+
+ self.connection = swiftclient.Connection(**conn_kwargs)
+ self.container = parsed_url.path.strip('/')
+
+ try:
+ # try to create the container. This verifies that user can
+ # authenticate, and has write permissions. As a nice side effect,
+ # we can later assume the container exists. WinWin... ;-)
+ self.connection.put_container(self.container)
+ except Exception, e:
+ log.FatalError("Connection failed, please check your credentials: %s %s"
+ % (e.__class__.__name__, str(e)),
+ log.ErrorCode.connection_failed)
+
+
+ def put(self, source_path, remote_filename = None):
+ if not remote_filename:
+ remote_filename = source_path.get_filename()
+
+ source_file = None
+ try:
+ source_file = open(source_path.name)
+
+ self.connection.put_object(
+ self.container,
+ remote_filename,
+ source_file
+ )
+ finally:
+ if source_file:
+ source_file.close()
+
+ def get(self, remote_filename, local_path):
+ local_file = None
+ try:
+ local_file = open(local_path.name, 'w')
+ headers, data = self.connection.get_object(self.container,
+ remote_filename, resp_chunk_size=4096 )
+ for chunk in data:
+ local_file.write(chunk)
+ finally:
+ if local_file:
+ local_file.close()
+
+ def list(self):
+ try:
+ container, objects = self.connection.get_container(self.container,
+ full_listing=True)
+ return [obj['name'] for obj in objects]
+ except swiftclient.ClientException:
+ return []
+
+ def delete_one(self, remote_filename):
+ self.connection.delete_object(self.container, remote_filename)
+
+ def delete(self, filename_list):
+ for file in filename_list:
+ self.delete_one(file)
+ log.Debug("Deleted '%s/%s'" % (self.container, file))
+
+ def _query_list_info(self, filename_list):
+ container, objects = self.connection.get_container(self.container,
+ full_listing=True)
+
+ return dict(
+ (obj['name'], {"size": obj['bytes']})
+ for obj in objects)
+
+
+duplicity.backend.register_backend("swift+http", SwiftBackend)
+duplicity.backend.register_backend("swift", SwiftBackend)
=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py 2013-02-18 00:33:16 +0000
+++ duplicity/commandline.py 2013-03-15 11:40:35 +0000
@@ -815,6 +815,7 @@
rsync://%(user)s[:%(password)s]@%(other_host)s[:%(port)s]//%(absolute_path)s
s3://%(other_host)s/%(bucket_name)s[/%(prefix)s]
s3+http://%(bucket_name)s[/%(prefix)s]
+ swift://%(container_name)s
scp://%(user)s[:%(password)s]@%(other_host)s[:%(port)s]/%(some_dir)s
ssh://%(user)s[:%(password)s]@%(other_host)s[:%(port)s]/%(some_dir)s
tahoe://%(alias)s/%(directory)s
