duplicity-team team mailing list archive

[Kenneth Loafman <kenneth@xxxxxxxxxxx>]

Hi Scott,

I would say that all your errors are from not being able to build
*_librsync.so* properly.  There is one thing that may help in lines 65-76
of *setup.py*.  You can supply *LIBRSYNC_DIR* as an env variable, or on the
command line as *--librsync-dir=*.  I have *export LIBRSYNC_DIR=/usr/local* in
my .bash_profile.  That came from *homebrew* installing into /usr/local
instead of /usr.  Seems none of these installers care much for standards.

Try exporting *LIBRSYNC_DIR* and let me know how it goes after that.

...Ken


On Sun, Oct 9, 2022 at 4:46 PM Scott Hannahs <sth@xxxxxxxxxx> wrote:

> So there are two major failures in testing.  One is four failures in the
> tahoe backend, (which I don’t use).
>
> testing/unit/test_backend_instance.py ..............................FF.. [
> 10%]
> FF........................ssssssssss                                     [
> 18%]
>
>
> This shows a “Key Error” so is something wrong with my GPG version?
>
> self = <duplicity.backends.tahoebackend.TAHOEBackend object at 0x10f58deb0>
> commandline = 'tahoe cp
> /var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/local var:folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output/file-a'
>
>     def subprocess_popen(self, commandline):
>         u"""
>         Execute the given command line with error check.
>         Returns int Exitcode, string StdOut, string StdErr
>
>         Raise a BackendException on failure.
>         """
>         import shlex
>
>         if isinstance(commandline, (list, tuple)):
>             logstr = u' '.join(commandline)
>             args = commandline
>         else:
>             logstr = commandline
>             args = shlex.split(commandline)
>
>         logstr = self.munge_password(logstr)
>         log.Info(_(u"Reading results of '%s'") % logstr)
>
>         result, stdout, stderr = self.__subprocess_popen(args)
>         if result != 0:
>             try:
> >               ignores = self.popen_breaks[args[0]]
> E               KeyError:
> '/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/overrides/bin/tahoe'
>
> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:522:
> KeyError
>
> During handling of the above exception, another exception occurred:
>
> The second testing error is more general, so many of the tests pass and
> then starting at the “bad upload” test it all goes south.
>
> testing/functional/test_badupload.py F                                   [
> 70%]
> testing/functional/test_cleanup.py FFF                                   [
> 71%]
> testing/functional/test_final.py FFFFFFFFFFFFFFFFFFFFFFFFFFF             [
> 77%]
> testing/functional/test_log.py F                                         [
> 77%]
> testing/functional/test_rdiffdir.py F                                    [
> 77%]
> testing/functional/test_replicate.py FFF                                 [
> 78%]
> testing/functional/test_restart.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF    [
> 86%]
> testing/functional/test_selection.py .FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [
> 94%]
> FFFFssFFFFFFFssss                                                        [
> 98%]
>
>
> This test has a long back trace, But I think that the basic problem is
> that duplicity cannot import “librsynch”???  The end of the traceback has
> the text "b"ImportError: cannot import name '_librsync' from 'duplicity'
> (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)"
> ...return_val: 1"
>
> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:527:
> BackendException
> _______________________ BadUploadTest.test_missing_file
> ________________________
>
> self = <testing.functional.test_badupload.BadUploadTest
> testMethod=test_missing_file>
>
>     @pytest.mark.slow
>     def test_missing_file(self):
>         u"""
>         Test basic lost file
>         """
>         try:
> >           self.backup(u"full",
> u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"])
>
>
> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:44:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _
>
> self = <testing.functional.test_badupload.BadUploadTest
> testMethod=test_missing_file>
> type = 'full'
> input_dir =
> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1'
> options = ['full',
> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', '
> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output',
> '--volsize', '1', '--skip-volume=1']
> kwargs = {}, before_files = set(), now = 1664937609.016365
>
>     def backup(self, type, input_dir, options=[], **kwargs):  # pylint:
> disable=redefined-builtin
>         u"""Run duplicity backup to default directory"""
>         options = [type, input_dir, self.backend_url, u"--volsize", u"1"]
> + options
>         before_files = self.get_backend_files()
>
>         # If a chain ends with time X and the next full chain begins at
> time X,
>         # we may trigger an assert in dup_collections.py.  If needed,
> sleep to
>         # avoid such problems
>         now = time.time()
>         if self.last_backup == int(now):
>             time.sleep(1)
>
> >       self.run_duplicity(options=options, **kwargs)
>
>
> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:189:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _
>
> self = <testing.functional.test_badupload.BadUploadTest
> testMethod=test_missing_file>
> options = ['full',
> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', '
> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output',
> '--volsize', '1', '--skip-volume=1']
> current_time = None, fail = None, passphrase_input = []
>
>     def run_duplicity(self, options=[], current_time=None, fail=None,
>                       passphrase_input=[]):
>         u"""
>         Run duplicity binary with given arguments and options
>         """
>         # We run under setsid and take input from /dev/null (below) because
>         # this way we force a failure if duplicity tries to read from the
>         # console unexpectedly (like for gpg password or such).
>
>         # Check all string inputs are unicode -- we will convert to system
> encoding before running the command
>         for item in options:
>             if sys.version_info.major == 2:
>                 assert not isinstance(item, str), u"item " + unicode(item)
> + u" in options is not unicode"
>
>         for item in passphrase_input:
>             assert isinstance(item, u"".__class__), u"item " +
> unicode(item) + u" in passphrase_input is not unicode"
>
>         if platform.platform().startswith(u'Linux'):
>             cmd_list = [u'setsid']
>             if self._setsid_w:
>                 cmd_list.extend([u"-w"])
>         else:
>             cmd_list = []
>         basepython = os.environ.get(u'TOXPYTHON', None)
>         if basepython is not None:
>             cmd_list.extend([basepython])
>         run_coverage = os.environ.get(u'RUN_COVERAGE', None)
>         if run_coverage is not None:
>             cmd_list.extend([u"-m", u"coverage", u"run",
> u"--source=duplicity", u"-p"])
>         cmd_list.extend([u"{0}/bin/duplicity".format(_top_dir)])
>         cmd_list.extend(options)
>         cmd_list.extend([u"-v0"])
>         cmd_list.extend([u"--no-print-statistics"])
>         cmd_list.extend([u"--allow-source-mismatch"])
>
> cmd_list.extend([u"--archive-dir={0}/testfiles/cache".format(_runtest_dir)])
>         if current_time:
>             cmd_list.extend([u"--current-time", current_time])
>         cmd_list.extend(self.class_args)
>         if fail:
>             cmd_list.extend([u"--fail", u"".__class__(fail)])
>         cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
>
>         cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
>
>         if not passphrase_input:
>             cmdline += u" < /dev/null"
>
>         # The immediately following block is the nicer way to execute
> pexpect with
>         # unicode strings, but we need to have the pre-4.0 version for
> some time yet,
>         # so for now this is commented out so tests execute the same way
> on all systems.
>
>         # if parse_version(pexpect.__version__) >= parse_version("4.0"):
>         #     # pexpect.spawn only supports unicode from version 4.0
>         #     # there was a separate pexpect.spawnu in 3.x, but it has an
> error on readline
>         #     child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline],
> timeout=None, encoding=sys.getfilesystemencoding())
>         #
>         #     for passphrase in passphrase_input:
>         #         child.expect(u'passphrase.*:')
>         #         child.sendline(passphrase)
>         # else:
>
>         # Manually encode to filesystem encoding and send to spawn as bytes
>         # ToDo: Remove this once we no longer have to support systems with
> pexpect < 4.0
>         if sys.version_info.major > 2:
>             child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline],
> timeout=None)
>         else:
>             child = pexpect.spawn(b'/bin/sh', [b'-c',
> cmdline.encode(sys.getfilesystemencoding(),
>
>  u'replace')], timeout=None)
>
>         for passphrase in passphrase_input:
>             child.expect(b'passphrase.*:')
>             child.sendline(passphrase)
>
>         # if the command fails, we need to clear its output
>         # so it will terminate cleanly.
>         child.expect_exact(pexpect.EOF)
>         lines = child.before.splitlines()
>         child.wait()
>         child.ptyproc.delayafterclose = 0.0
>         return_val = child.exitstatus
>
>         if fail:
>             self.assertEqual(30, return_val)
>         elif return_val:
>             print(u"\n...command:", cmdline, file=sys.stderr)
>             print(u"...cwd:", os.getcwd(), file=sys.stderr)
>             print(u"...output:", file=sys.stderr)
>             for line in lines:
>                 line = line.rstrip()
>                 if line:
>                     print(line, file=sys.stderr)
>             print(u"...return_val:", return_val, file=sys.stderr)
>
> >           raise CmdError(return_val)
> E           testing.functional.CmdError: 1
>
> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:175:
> CmdError
>
> During handling of the above exception, another exception occurred:
>
> self = <testing.functional.test_badupload.BadUploadTest
> testMethod=test_missing_file>
>
>     @pytest.mark.slow
>     def test_missing_file(self):
>         u"""
>         Test basic lost file
>         """
>         try:
>             self.backup(u"full",
> u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"])
>             self.fail()
>         except CmdError as e:
> >           self.assertEqual(e.exit_status, 44, str(e))
> E           AssertionError: 1 != 44 : 1
>
> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:47:
> AssertionError
> ----------------------------- Captured stderr call
> -----------------------------
>
> ...command:
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity"
> "full" "/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1" "
> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output"
> "--volsize" "1" "--skip-volume=1" "-v0" "--no-print-statistics"
> "--allow-source-mismatch"
> "--archive-dir=/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/cache"
> < /dev/null
> ...cwd: /private/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T
> ...output:
> b'Traceback (most recent call last):'
> b'  File
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity",
> line 37, in <module>'
> b'    from duplicity.dup_main import main'
> b'  File
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_main.py",
> line 50, in <module>'
> b'    from duplicity import commandline'
> b'  File
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/commandline.py",
> line 43, in <module>'
> b'    from duplicity import backend'
> b'  File
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py",
> line 44, in <module>'
> b'    from duplicity import dup_temp'
> b'  File
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_temp.py",
> line 34, in <module>'
> b'    from duplicity import path'
> b'  File
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/path.py",
> line 49, in <module>'
> b'    from duplicity import librsync'
> b'  File
> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/librsync.py",
> line 36, in <module>'
> b'    from . import _librsync'
> b"ImportError: cannot import name '_librsync' from 'duplicity'
> (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)"
> ...return_val: 1
>
>
> On Oct 9, 2022, at 11:26 AM, Kenneth Loafman <kenneth@xxxxxxxxxxx> wrote:
>
> Ok, sounds good.  Any progress?
>
> ...Ken
>
>
> On Thu, Oct 6, 2022 at 3:15 PM Scott Hannahs <sth@xxxxxxxxxx> wrote:
>
>> I will try to reinstall Xcode and the command line tools.  I had been
>> using py38 which is installed in both the standard and an updated version
>> in the non-standard location.
>>
>> On Oct 6, 2022, at 14:57, Kenneth Loafman <kenneth@xxxxxxxxxxx> wrote:
>>
>> OK,
>>
>> I could not remember what version of Python 3 you were running, so py3x
>> meant for you to fill it in.  I'm guessing either py38 or py39 for Python
>> 3.8 and Python 3.9 respectively.  Others follow the same model,
>> py27,py35,...py310.
>>
>> I am not sure why it's trying to build for anything but x86_64 if you
>> have Intel silicone and not Apple silicone.
>>
>> Here's what mine looks like for py38:
>>
>> ken@dione:~/workspace/duplicity-origin$ python3.8 ./setup.py build_ext
>> Unable to get SCM version: defaulting to 1.0.1
>> running build_ext
>> Building extension for librsync...
>> building 'duplicity._librsync' extension
>> creating build/temp.macosx-12-x86_64-cpython-38
>> creating build/temp.macosx-12-x86_64-cpython-38/duplicity
>> clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common
>> -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot
>> /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/usr/local/include
>> -I/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/include/python3.8
>> -c duplicity/_librsyncmodule.c -o
>> build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o
>> creating build/lib.macosx-12-x86_64-cpython-38
>> creating build/lib.macosx-12-x86_64-cpython-38/duplicity
>> clang -bundle -undefined dynamic_lookup -isysroot
>> /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
>> build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o
>> -L/usr/local/lib -lrsync -o
>> build/lib.macosx-12-x86_64-cpython-38/duplicity/_
>> librsync.cpython-38-darwin.so
>> ld: warning: -undefined dynamic_lookup may not work with chained fixups
>> copying build/lib.macosx-12-x86_64-cpython-38/duplicity/_
>> librsync.cpython-38-darwin.so -> duplicity
>>
>>
>> It looks like your xcode libs are out of date, and it also looks like
>> you're not using the standard locations for software.  That's gonna
>> complicate matters.
>>
>> I'm on Monterey 12.6 with the current Xcode from Apple, all in standard
>> places.
>>
>> ...Ken
>>
>>
>>
>>
>> On Thu, Oct 6, 2022 at 1:13 PM Scott Hannahs <sth@xxxxxxxxxx> wrote:
>>
>>> Ken,
>>>
>>> I will leave fink out, but it should be doing exactly the same thing!
>>>  :-)
>>>
>>> the following doesn’t even build duplicity correctly, so I don’t get to
>>> the tox command.
>>>
>>> % tar xf duplicity-1.0.1.tar
>>> % cd duplicity-1.0.1
>>> % % C_INCLUDE_PATH=/opt/sw/include LIBRARY_PATH=/opt/sw/lib python3
>>> ./setup.py build_ext > build.out.txt 2>&1
>>>
>>>
>>> This gives the following output.
>>>
>>> Unable to get SCM version: defaulting to 1.0.1
>>> running build_ext
>>> Building extension for librsync...
>>> building 'duplicity._librsync' extension
>>> clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common
>>> -dynamic -DNDEBUG -g -fwrapv -O3 -Wall
>>> -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders
>>> -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers
>>> -arch arm64 -arch x86_64 -Werror=implicit-function-declaration
>>> -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/include/python3.9
>>> -c duplicity/_librsyncmodule.c -o
>>> build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o
>>> clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64
>>> -Wl,-headerpad,0x1000
>>> build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o -lrsync
>>> -o build/lib.macosx-10.9-universal2-3.9/duplicity/_
>>> librsync.cpython-39-darwin.so
>>> ld: warning: ignoring file /opt/sw/lib/librsync.dylib, building for
>>> macOS-arm64 but attempting to link with file built for macOS-x86_64
>>> ld: warning: dylib (/opt/sw/lib/librsync.dylib) was built for newer
>>> macOS version (12.4) than being linked (10.9)
>>> copying build/lib.macosx-10.9-universal2-3.9/duplicity/_
>>> librsync.cpython-39-darwin.so -> duplicity
>>>
>>>
>>> I am building on an X86 machine and my libraries are x86 and I am not
>>> trying to build a universal arm 64 system.   But with warnings it tries to
>>> build.
>>>
>>> tox does not like the py3x value
>>>
>>> % tox -epy3
>>> py3 create: /Users/sth/Downloads/duplicity-1.0.1/.tox/py3
>>> py3 installdeps: -rrequirements.txt
>>>
>>> and I ran out of time this AM waiting for more output.
>>>
>>> -Scott
>>>
>>>
>>> On Oct 5, 2022, at 11:54 AM, Kenneth Loafman <kenneth@xxxxxxxxxxx>
>>> wrote:
>>>
>>> Could you try just building and running the tests in a 'normal' user
>>> environment?  Leave Fink out of the picture.
>>>
>>> do this:
>>>
>>>    1. $ tar xf duplicity-1.01.tar.gz
>>>    2. $ ./setup.py build_ext
>>>    3. $ tox -e py3x
>>>
>>> ...Thanks,
>>> ...Ken
>>>
>>>
>>>
>>
>