duplicity-team team mailing list archive

[westerngateguard <yigala@xxxxxxxxxxxxxxxxx>]

westerngateguard has proposed merging lp:~user3942934/duplicity/duplicity into lp:duplicity.

Requested reviews:
  duplicity-team (duplicity-team)

For more details, see:
https://code.launchpad.net/~user3942934/duplicity/duplicity/+merge/246258

Hello,

Currently duplicity uses gdocs backend for Google Drive backups.
gdocs uses deprecated API and don't allow backups for managed Google accounts.
(see https://bugs.launchpad.net/duplicity/+bug/1315684)

I added pydrive backend that solves both of those problems. I published it also on GitHub: https://github.com/westerngateguard/duplicity-pydrive-backend.

It works fine on my Debian (Wheezy) machine.
-- 
Your team duplicity-team is requested to review the proposed merge of lp:~user3942934/duplicity/duplicity into lp:duplicity.

=== modified file 'bin/duplicity.1'
--- bin/duplicity.1	2015-01-10 19:38:59 +0000
+++ bin/duplicity.1	2015-01-13 07:31:12 +0000
@@ -56,10 +56,6 @@
 
 Some backends also require additional components (probably available as packages for your specific platform):
 .TP
-.BR "azure backend" " (Azure Blob Storage Service)"
-.B Microsoft Azure SDK for Python
-- https://github.com/Azure/azure-sdk-for-python
-.TP
 .BR "boto backend" " (S3 Amazon Web Services, Google Cloud Storage)"
 .B boto version 2.0+
 - http://github.com/boto/boto
@@ -149,7 +145,21 @@
 .br
 (also see
 .BR "A NOTE ON SSL CERTIFICATE VERIFICATION" ).
-
+.TP
+.BR "pydrive backend" " (Google PyDrive)"
+.B Google Drive APIs Python Client Library
+- https://github.com/westerngateguard/duplicity-pydrive-backend
+To enable pydrive backend you need:
+1. To install PyDrive
+python pip install PyDrive
+2. To create a service account in "Google developers console" at
+https://console.developers.google.com
+2.1 Copy the email address of the created account
+2.2 Download the .p12 keyfile, then convert it to the .pem format:
+openssl pkcs12 -in XXX.p12  -nodes -nocerts > pydriveprivatekey.pem
+3. To use the backend, the remote URL will be in the form of
+pydrive://<service account' email address>@developer.gserviceaccount.com/remote/path/on/google/drive?keyfile=/path/to/your/pydriveprivatekey.pem
+.br
 .SH DESCRIPTION
 Duplicity incrementally backs up files and folders into
 tar-format volumes encrypted with GnuPG and places them to a
@@ -1102,15 +1112,6 @@
 Formats of each of the URL schemes follow:
 
 .PP
-.BR "Azure"
-.PP
-.RS
-azure://container_name
-.PP
-See also
-.B "A NOTE ON AZURE ACCESS"
-.RE
-.PP
 .BR "Cloud Files" " (Rackspace)"
 .PP
 .RS
@@ -1575,17 +1576,6 @@
 which aren't followed by 'foo'.  However, it wouldn't match /home even
 if /home/ben/1234567 existed.
 
-.SH A NOTE ON AZURE ACCESS
-The Azure backend requires the Microsoft Azure SDK for Python to be installed
-on the system.
-See
-.B REQUIREMENTS
-above.
-
-It uses two environment variables for authentification:
-.BR AZURE_ACCOUNT_NAME " (required),"
-.BR AZURE_ACCOUNT_KEY " (required)"
-
 .SH A NOTE ON CLOUD FILES ACCESS
 Pyrax is Rackspace's next-generation Cloud management API, including
 Cloud Files access.  The cfpyrax backend requires the pyrax library to

=== modified file 'duplicity/backend.py'
--- duplicity/backend.py	2014-12-17 10:35:11 +0000
+++ duplicity/backend.py	2015-01-13 07:31:12 +0000
@@ -252,6 +252,12 @@
         except Exception:
             raise InvalidBackendURL("Syntax error in: %s" % url_string)
 
+        if pu.query:     
+            try:
+                self.keyfile = urlparse.parse_qs(pu.query)['keyfile']
+            except Exception:
+                raise InvalidBackendURL("Syntax error (keyfile) in: %s" % url_string)
+                
         try:
             self.scheme = pu.scheme
         except Exception:

=== added file 'duplicity/backends/pydrivebackend.py'
--- duplicity/backends/pydrivebackend.py	1970-01-01 00:00:00 +0000
+++ duplicity/backends/pydrivebackend.py	2015-01-13 07:31:12 +0000
@@ -0,0 +1,107 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2015 Yigal Asnis 
+#
+# This file is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# It is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import string
+import duplicity.backend
+from duplicity.errors import BackendException
+
+class PyDriveBackend(duplicity.backend.Backend):
+    """Connect to remote store using PyDrive API"""
+
+    def __init__(self, parsed_url):
+        duplicity.backend.Backend.__init__(self, parsed_url)
+        try:
+            global pydrive
+            import httplib2
+            from apiclient.discovery import build
+            from oauth2client.client import SignedJwtAssertionCredentials
+            from pydrive.auth import GoogleAuth
+            from pydrive.drive import GoogleDrive
+        except ImportError:
+            raise BackendException('PyDrive backend requires PyDrive installation'
+                                   'python pip install PyDrive')
+        try: 
+            keyFilePath = parsed_url.keyfile
+            keyfile = open(keyFilePath[0], "r+")
+            key = keyfile.read();
+            keyfile.close()
+        except:
+            raise BackendException("Can't find/open/read the keyfile. Please read the manpage (man duplicity) for configuration info.") 
+        credentials = SignedJwtAssertionCredentials(parsed_url.username + '@' + parsed_url.hostname, key, scope='https://www.googleapis.com/auth/drive')
+        credentials.authorize(httplib2.Http())
+        gauth = GoogleAuth()
+        gauth.credentials = credentials
+        self.drive = GoogleDrive(gauth)
+        
+        # Dirty way to find root folder id
+        file_list = self.drive.ListFile({'q': "'Root' in parents"}).GetList()
+        if file_list:
+            parent_folder_id = file_list[0]['parents'][0]['id']
+        else:
+            file_in_root = self.drive.CreateFile({'title': 'i_am_in_root'})
+            file_in_root.Upload()
+            parent_folder_id = file_in_root['parents'][0]['id']
+                         
+        # Fetch destination folder entry (and create hierarchy if required).
+        folder_names = string.split(parsed_url.path, '/')
+        for folder_name in folder_names:
+            if not folder_name:
+                continue
+            file_list = self.drive.ListFile({'q': "'" + parent_folder_id + "' in parents"}).GetList()
+            folder =  next((item for item in file_list if item['title'] == folder_name and item['mimeType'] == 'application/vnd.google-apps.folder'), None)
+            if folder is None:
+                folder = self.drive.CreateFile({'title': folder_name, 'mimeType': "application/vnd.google-apps.folder", 'parents': [{'id': parent_folder_id}]})
+                folder.Upload()
+            parent_folder_id = folder['id']
+        self.folder = parent_folder_id
+        
+    def FilesList(self):
+        return self.drive.ListFile({'q': "'" + self.folder + "' in parents"}).GetList()
+    
+    def id_by_name(self, filename):
+        try:
+            return next(item for item in self.FilesList() if item['title'] == filename)['id']
+        except:
+            return ''
+            
+    def _put(self, source_path, remote_filename):
+        drive_file = self.drive.CreateFile({'title': remote_filename, 'parents': [{"kind": "drive#fileLink", "id": self.folder}]})
+        drive_file.SetContentFile(source_path.name)
+        drive_file.Upload()
+
+    def _get(self, remote_filename, local_path):
+        drive_file = self.drive.CreateFile({'id': self.id_by_name(remote_filename)})
+        drive_file.GetContentFile(local_path.name)
+        
+    def _list(self):
+        return [item['title'] for item in self.FilesList()]
+       
+    def _delete(self, filename):
+        file_id = self.id_by_name(filename)
+        drive_file = self.drive.CreateFile({'id': file_id})
+        drive_file.auth.service.files().delete(fileId=drive_file['id']).execute()
+        
+    def _query(self, filename):
+        try:
+            size = int((item for item in self.FilesList() if item['title'] == filename).next()['fileSize'])
+        except:
+            size = -1
+        return {'size': size}
+                   
+duplicity.backend.register_backend('pydrive', PyDriveBackend)
+duplicity.backend.uses_netloc.extend([ 'pydrive' ])

=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py	2015-01-10 19:38:59 +0000
+++ duplicity/commandline.py	2015-01-13 07:31:12 +0000
@@ -856,6 +856,7 @@
   webdav://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   webdavs://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   gdocs://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
+  pydrive://%(user)s@%(other_host)s/%(some_dir)s
   mega://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   copy://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   dpbx:///%(some_dir)s