duplicity-team team mailing list archive

[Steve Tynor <stynor@xxxxxxxxx>]

Steve Tynor has proposed merging lp:~stynor/duplicity/multi-backend into lp:duplicity.

Requested reviews:
  duplicity-team (duplicity-team)

For more details, see:
https://code.launchpad.net/~stynor/duplicity/multi-backend/+merge/255911

A new backend that allows use of more than one backend stores (e.g. to combine the available space from more than one cloud provider to make a larger store available to duplicity).  

Changed file:
   bin/duplicity.1
New file:
   duplicity/backends/multibackend.py

I've tested this with the local, gdocs and pydrive backends.   Usage instructions are in the manpage.

-- 
Your team duplicity-team is requested to review the proposed merge of lp:~stynor/duplicity/multi-backend into lp:duplicity.

=== modified file 'bin/duplicity.1' (properties changed: -x to +x)
--- bin/duplicity.1	2015-03-12 21:43:25 +0000
+++ bin/duplicity.1	2015-04-11 22:06:20 +0000
@@ -157,6 +157,13 @@
 (also see
 .BR "A NOTE ON PYDRIVE BACKEND"
 ) below.
+.TP
+.BR "multi backend"
+.B Multi -- store to more than one backend
+.br
+(also see
+.BR "A NOTE ON MULTI BACKEND"
+) below.
 .br
 .SH DESCRIPTION
 Duplicity incrementally backs up files and folders into
@@ -1272,6 +1279,15 @@
 .B "A NOTE ON PYDRIVE BACKEND"
 below.
 .RE
+.BR "multi"
+.PP
+.RS
+multi:///path/to/config.json
+.PP
+See also
+.B "A NOTE ON MULTI BACKEND"
+below.
+.RE
 
 .SH TIME FORMATS
 duplicity uses time strings in two places.  Firstly, many of the files
@@ -1862,6 +1878,43 @@
 .BR GOOGLE_DRIVE_ACCOUNT_KEY
 environment variable for authentification.
 
+.SH A NOTE ON MULTI BACKEND
+
+The multi backend allows duplicity to combine the storage available in
+more than one backend store (e.g., you can store across a google drive
+account and a onedrive account to get effectively the combined storage
+available in both).  The URL path specifies a JSON formated config
+file containing a list of the backends it will use. Multibackend then
+round-robins across the given backends.  Each element of the list must
+have a "url" element, and may also contain an optional "description"
+and an optional "env" list of environment variables used to configure
+that backend. 
+.PP
+For example:
+.nf
+.RS
+[
+ {
+  "description": "a comment about the backend
+  "url": "abackend://myuser@xxxxxxxxxx/backup",
+  "env": [
+    {
+     "name" : "MYENV",
+     "value" : "xyz"
+    },
+    {
+     "name" : "FOO",
+     "value" : "bar"
+    }
+   ]
+ },
+ {
+  "url": "file:///path/to/dir"
+ }
+]
+.RE
+.fi
+
 .SH A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING
 Signing and symmetrically encrypt at the same time with the gpg binary on the
 command line, as used within duplicity, is a specifically challenging issue.

=== added file 'duplicity/backends/multibackend.py'
--- duplicity/backends/multibackend.py	1970-01-01 00:00:00 +0000
+++ duplicity/backends/multibackend.py	2015-04-11 22:06:20 +0000
@@ -0,0 +1,190 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2015 Steve Tynor <steve.tynor@xxxxxxxxx>
+#
+# This file is part of duplicity.
+#
+# Duplicity is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# Duplicity is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+#
+
+import os
+import os.path
+import string
+import urllib
+import json
+
+import duplicity.backend
+from duplicity.errors import BackendException
+from duplicity import log
+
+class MultiBackend(duplicity.backend.Backend):
+    """Store files across multiple remote stores. URL is a path to a local file containing URLs/other config defining the remote store"""
+
+        
+    # the stores we are managing
+    __stores = []
+
+    # when we write, we "stripe" via a simple round-robin across
+    # remote stores.  It's hard to get too much more sophisticated
+    # since we can't rely on the backend to give us any useful meta
+    # data (e.g. sizes of files, capacity of the store (quotas)) to do
+    # a better job of balancing load across stores.
+    __write_cursor = 0
+
+    def __init__(self, parsed_url):
+        duplicity.backend.Backend.__init__(self, parsed_url)
+        
+        # Init each of the wrapped stores
+        #
+        # config file is a json formatted collection of values, one for
+        # each backend.  We will 'stripe' data across all the given stores:
+        #
+        #  'url'  - the URL used for the backend store
+        #  'env' - an optional list of enviroment variable values to set
+        #      during the intialization of the backend
+        #
+        # Example:
+        #
+        # [
+        #  {
+        #   "url": "abackend://myuser@xxxxxxxxxx/backup",
+        #   "env": [
+        #     {
+        #      "name" : "MYENV",
+        #      "value" : "xyz"
+        #     },
+        #     {
+        #      "name" : "FOO",
+        #      "value" : "bar"
+        #     }
+        #    ]
+        #  },
+        #  {
+        #   "url": "file:///path/to/dir"
+        #  }
+        # ]
+                
+        try:
+            with open(parsed_url.path) as f:
+                configs = json.load(f)
+        except IOError as e:
+            log.Log(_("MutliBackend: Could not load config file %s: %s ")
+                    % (parsed_url.path, e),
+                    log.ERROR)
+            raise BackendException('Could not load config file')
+
+        for config in configs:
+            url = config['url']
+            log.Log(_("MultiBackend: use store %s")
+                    % (url),
+                    log.INFO)
+            if 'env' in config:
+                for env in config['env']:
+                    log.Log(_("MultiBackend: set env %s = %s")
+                            % (env['name'],env['value']),
+                            log.INFO)
+                    os.environ[env['name']] = env['value'];
+                
+            store = duplicity.backend.get_backend(url)
+        	self.__stores.append(store)
+            store_list = store.list()
+            log.Log(_("MultiBackend: at init, store %s has %s files")
+                    % (url, len(store_list)),
+                    log.INFO)
+    
+    def _put(self, source_path, remote_filename):
+        first = self.__write_cursor
+        while True:
+            store = self.__stores[self.__write_cursor]
+            try:
+                next = self.__write_cursor + 1;
+                if (next > len(self.__stores) -1):
+                    next = 0
+                log.Log(_("MultiBackend: _put: write to store #%s (%s)")
+                        % (self.__write_cursor, store.backend.parsed_url.url_string),
+                        log.DEBUG)
+                store.put(source_path, remote_filename)
+                self.__write_cursor = next
+                break
+            except Exception, e:            
+                log.Log(_("MultiBackend: failed to write to store #%s (%s), try #%s, Exception: %s")
+                        % (self.__write_cursor, store.backend.parsed_url.url_string, next, e),
+                        log.INFO)
+                self.__write_cursor = next
+
+                if (self.__write_cursor == first):
+                    log.Log(_("MultiBackend: failed to write %s. Tried all backing stores and none succeeded")
+                            % (source_path),
+                            log.ERROR)
+                    raise BackendException("failed to write");
+    
+    def _get(self, remote_filename, local_path):
+        # since the backend operations will be retried, we can't
+        # simply try to get from the store, if not found, move to the
+        # next store (since each failure will be retried n times
+        # before finally giving up).  So we need to get the list first
+        # before we try to fetch
+        # ENHANCEME: maintain a cached list for each store
+        for s in self.__stores:
+            list = s.list()
+            if remote_filename in list:
+                s.get(remote_filename, local_path)
+                return
+            log.Log(_("MultiBackend: failed to get %s to %s from %s")
+                    % (remote_filename, local_path, s.backend.parsed_url.url_string),
+                    log.INFO)
+        log.Log(_("MultiBackend: failed to get %s. Tried all backing stores and none succeeded")
+                % (remote_filename),
+                log.ERROR)
+        raise BackendException("failed to get")
+
+    def _list(self):
+        lists = []
+        for s in self.__stores:
+            l = s.list()
+            log.Log(_("MultiBackend: list from %s: %s")
+                    % (s.backend.parsed_url.url_string, l),
+                    log.DEBUG)
+            lists.append(s.list())
+        # combine the lists into a single flat list:
+        result = [item for sublist in lists for item in sublist]
+        log.Log(_("MultiBackend: combined list: %s")
+                % (result),
+                log.DEBUG)
+        return result
+        
+    def _delete(self, filename):
+        # since the backend operations will be retried, we can't
+        # simply try to get from the store, if not found, move to the
+        # next store (since each failure will be retried n times
+        # before finally giving up).  So we need to get the list first
+        # before we try to delete
+        # ENHANCEME: maintain a cached list for each store
+        for s in self.__stores:
+            list = s.list()
+            if filename in list:
+                s._do_delete(filename)
+                return
+            log.Log(_("MultiBackend: failed to delete %s from %s")
+                    % (filename, s.backend.parsed_url.url_string),
+                    log.INFO)
+        log.Log(_("MultiBackend: failed to delete %s. Tried all backing stores and none succeeded")
+                % (filename),
+                log.ERROR)
+#        raise BackendException("failed to delete")
+    
+duplicity.backend.register_backend('multi', MultiBackend)
+