duplicity-team team mailing list archive

[Xavier Lucas <xavier.lucas@xxxxxxxxxxxx>]

Xavier Lucas has proposed merging lp:~xlucas/duplicity/swift-storage-policies into lp:duplicity.

Requested reviews:
  duplicity-team (duplicity-team)

For more details, see:
https://code.launchpad.net/~xlucas/duplicity/swift-storage-policies/+merge/324008

This brings support for Swift storage policies: when using a Swift backend, you can specify the  policy containers should be operating on. This is similar to the idea of AWS Cloud Storage classes  (ia, rrs, glacier and so on).

For instance at OVH we built solutions for hot/cold storage atop Swift that typically differentiate for the user, when manipulating the Swift API, by the storage policy in use.
-- 
Your team duplicity-team is requested to review the proposed merge of lp:~xlucas/duplicity/swift-storage-policies into lp:duplicity.

=== modified file 'bin/duplicity.1'
--- bin/duplicity.1	2017-05-04 15:06:41 +0000
+++ bin/duplicity.1	2017-05-12 22:39:46 +0000
@@ -927,6 +927,13 @@
 .BR "A NOTE ON SSL CERTIFICATE VERIFICATION" .
 
 .TP
+.BI --swift-storage-policy
+Use this storage policy when operating on Swift containers.
+.br
+See also
+.BR "A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS" .
+
+.TP
 .BI "--tempdir " directory
 Use this existing directory for duplicity temporary files instead of
 the system default, which is usually the /tmp directory. This option

=== modified file 'duplicity/backends/swiftbackend.py'
--- duplicity/backends/swiftbackend.py	2017-05-05 20:59:45 +0000
+++ duplicity/backends/swiftbackend.py	2017-05-12 22:39:46 +0000
@@ -21,6 +21,7 @@
 import os
 
 import duplicity.backend
+from duplicity import globals
 from duplicity import log
 from duplicity import util
 from duplicity.errors import BackendException
@@ -108,6 +109,9 @@
         else:
             self.prefix = ''
 
+        policy = globals.swift_storage_policy
+        policy_header = 'X-Storage-Policy'
+
         container_metadata = None
         try:
             self.conn = Connection(**conn_kwargs)
@@ -122,11 +126,15 @@
         if container_metadata is None:
             log.Info("Creating container %s" % self.container)
             try:
-                self.conn.put_container(self.container)
+                headers = dict([[policy_header, policy]]) if policy else None
+                self.conn.put_container(self.container, headers=headers)
             except Exception as e:
                 log.FatalError("Container creation failed: %s %s"
                                % (e.__class__.__name__, str(e)),
                                log.ErrorCode.connection_failed)
+        elif policy and container_metadata[policy_header.lower()] != policy:
+            log.FatalError("Container '%s' exists but its storage policy is '%s' not '%s'."
+                           % (self.container, container_metadata[policy_header.lower()], policy))
 
     def _error_code(self, operation, e):
         if isinstance(e, self.resp_exc):

=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py	2017-05-04 15:52:42 +0000
+++ duplicity/commandline.py	2017-05-12 22:39:46 +0000
@@ -563,6 +563,9 @@
     # Option to allow use of server side encryption in s3
     parser.add_option("--s3-use-server-side-encryption", action="store_true", dest="s3_use_sse")
 
+    # Option to specify a Swift container storage policy.
+    parser.add_option("--swift-storage-policy", type="string", metavar=_("policy"))
+
     # Number of the largest supported upload size where the Azure library makes only one put call.
     # This is used to upload a single block if the content length is known and is less than this value.
     # The default is 67108864 (64MiB)

=== modified file 'duplicity/globals.py'
--- duplicity/globals.py	2017-05-11 10:45:03 +0000
+++ duplicity/globals.py	2017-05-12 22:39:46 +0000
@@ -216,6 +216,9 @@
 # Use server side encryption in s3
 s3_use_sse = False
 
+# Which storage policy to use for Swift containers
+swift_storage_policy = ""
+
 # The largest size upload supported in a single put call for azure
 azure_max_single_put_size = None