curtin-dev team mailing list archive

[Olivier Gayot <mp+442650@xxxxxxxxxxxxxxxxxx>]

Olivier Gayot has proposed merging curtin:integration-memory-error into curtin:master.

Commit message:
tests: fix memory error in integration test when we allocate a 2GiB string

When create=True, the function add_image does something similar to what
'dd if=/dev/zero bs=1 count={size}' would do.

That said, our implementation allocates a string of the requested size
in memory. This fails with MemoryError on systems with limited RAM
available.

Instead, we now call 'dd', which is more memory efficient. It also
allows to create a sparse file, which /might/ result in a slight speedup
in test execution.


Requested reviews:
  Server Team CI bot (server-team-bot): continuous-integration
  curtin developers (curtin-dev)

For more details, see:
https://code.launchpad.net/~curtin-dev/curtin/+git/curtin/+merge/442650

Fix MemoryError when running integration tests on systems with limited RAM.

Because of our implementation of `add_image`, fhe following test allocates space for a 2GiB string in memory:

    def test_raw_image(self):
        img = self.tmp_path('image.img')
        config = StorageConfigBuilder(version=1)
        config.add_image(path=img, size='2G', ptable='gpt', create=True)

On a LXD VM with default memory settings, this results in a MemoryError:

________________________________________________ TestBlockMeta.test_raw_image _________________________________________________

self = <integration.test_block_meta.TestBlockMeta testMethod=test_raw_image>

    def test_raw_image(self):
        img = self.tmp_path('image.img')
        config = StorageConfigBuilder(version=1)
>       config.add_image(path=img, size='2G', ptable='gpt', create=True)

tests/integration/test_block_meta.py:573: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <integration.test_block_meta.StorageConfigBuilder object at 0x7fa27099e860>
path = '/tmp/curtin-ci-TestBlockMeta.27b4jc0r/image.img', size = '2G', create = True, kw = {'ptable': 'gpt'}
f = <_io.BufferedWriter name='/tmp/curtin-ci-TestBlockMeta.27b4jc0r/image.img'>

    def add_image(self, *, path, size, create=False, **kw):
        if create:
            with open(path, "wb") as f:
>               f.write(b"\0" * int(util.human2bytes(size)))
E               MemoryError

tests/integration/test_block_meta.py:172: MemoryError

NOTE: On a default LXD VM, the test would still fail because the bcache kernel module is not present on the kvm kernel flavor. This is however a different problem that we can treat separately.
-- 
Your team curtin developers is requested to review the proposed merge of curtin:integration-memory-error into curtin:master.

diff --git a/tests/integration/test_block_meta.py b/tests/integration/test_block_meta.py
index b25d2ce..36a6a8c 100644
--- a/tests/integration/test_block_meta.py
+++ b/tests/integration/test_block_meta.py
@@ -7,6 +7,7 @@ import json
 import os
 from parameterized import parameterized
 import re
+import subprocess
 import sys
 from typing import Optional
 import yaml
@@ -168,8 +169,12 @@ class StorageConfigBuilder:
 
     def add_image(self, *, path, size, create=False, **kw):
         if create:
-            with open(path, "wb") as f:
-                f.write(b"\0" * int(util.human2bytes(size)))
+            # Create a sparse file of the specified size.
+            subprocess.run([
+                'dd',
+                'if=/dev/zero', f'of={path}',
+                'bs=1', 'count=0', f'seek={size}',
+            ])
         action = self._add(type='image', path=path, size=size, **kw)
         self.cur_image = action['id']
         return action
diff --git a/tests/unittests/helpers.py b/tests/unittests/helpers.py
index 819e2c5..93b5b5b 100644
--- a/tests/unittests/helpers.py
+++ b/tests/unittests/helpers.py
@@ -152,8 +152,8 @@ class CiTestCase(TestCase):
         return os.path.normpath(
             os.path.abspath(os.path.sep.join((_dir, path))))
 
-    @classmethod
-    def random_string(cls, length=8):
+    @staticmethod
+    def random_string(length=8):
         """ return a random lowercase string with default length of 8"""
         return ''.join(
             random.choice(string.ascii_lowercase) for _ in range(length))