launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:librarian-new-container-workaround into launchpad:master.

Commit message:
Allow librarian-feed-swift to sleep after creating container

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/405744

radosgw apparently has a bug where newly-created containers need to be left idle for a short time before they can be used reliably.  For now, allow working around this by configuring `librarian_server.new_container_delay` to a number of seconds to sleep after creating a new container.

See https://portal.admin.canonical.com/C131836.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:librarian-new-container-workaround into launchpad:master.

diff --git a/lib/lp/services/config/schema-lazr.conf b/lib/lp/services/config/schema-lazr.conf
index 749ecaa..894fac0 100644
--- a/lib/lp/services/config/schema-lazr.conf
+++ b/lib/lp/services/config/schema-lazr.conf
@@ -1247,6 +1247,11 @@ restricted_logfile: -
 # datatype: string
 root: none
 
+# Allow working around a radosgw bug by sleeping for a short time (measured
+# in seconds) after creating new containers.
+# datatype: integer
+new_container_delay: 0
+
 # Swift connection information and secret.
 #
 # datatype: urlbase
diff --git a/lib/lp/services/librarianserver/swift.py b/lib/lp/services/librarianserver/swift.py
index 662b2fa..81df074 100644
--- a/lib/lp/services/librarianserver/swift.py
+++ b/lib/lp/services/librarianserver/swift.py
@@ -180,6 +180,15 @@ def _to_swift_file(log, swift_connection, lfc_id, fs_path):
             raise
         log.info('Creating {0} container'.format(container))
         swift_connection.put_container(container)
+        # radosgw apparently has a bug where newly-created containers need
+        # to be left idle for a short time before they can be used reliably.
+        # If configured to do so, work around this by sleeping for a while
+        # before continuing.
+        if config.librarian_server.new_container_delay:
+            log.info(
+                'Sleeping for %d seconds after creating container',
+                config.librarian_server.new_container_delay)
+            time.sleep(config.librarian_server.new_container_delay)
 
     try:
         headers = quiet_swiftclient(
diff --git a/lib/lp/services/librarianserver/tests/test_swift.py b/lib/lp/services/librarianserver/tests/test_swift.py
index 61a4a3d..5555f01 100644
--- a/lib/lp/services/librarianserver/tests/test_swift.py
+++ b/lib/lp/services/librarianserver/tests/test_swift.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2017 Canonical Ltd.  This software is licensed under the
+# Copyright 2013-2021 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Librarian disk to Swift storage tests."""
@@ -367,6 +367,23 @@ class TestFeedSwift(TestCase):
             finally:
                 swift_client.close()
 
+    def test_new_container_delay_unconfigured(self):
+        # If not explicitly configured to do so, _to_swift_file does not
+        # sleep after creating a new container.
+        log = BufferLogger()
+        with patch('time.sleep') as mock_sleep:
+            swift.to_swift(log)
+            mock_sleep.assert_not_called()
+
+    def test_new_container_delay(self):
+        # _to_swift_file sleeps for a configurable number of seconds after
+        # creating a new container, to work around a radosgw bug.
+        log = BufferLogger()
+        self.pushConfig('librarian_server', new_container_delay=60)
+        with patch('time.sleep') as mock_sleep:
+            swift.to_swift(log)
+            mock_sleep.assert_called_once_with(60)
+
 
 class TestHashStream(TestCase):
     layer = BaseLayer