duplicity-team team mailing list archive

[Kenneth Loafman <kenneth@xxxxxxxxxxx>]

Scott,

I think you are confused.

When I refer to *_librsync.so* it is really *_librsync.cpython-39-darwin.so
<http://librsync.cpython-39-darwin.so>* that I'm referring to.  The middle
part will change depending on which python and its version it's compiled
against.  And no, _librsyncmodule.o is not needed at all once the compile
and link have finished.  The build directory is strictly transitory.

*librsync* is a requirement for duplicity.  See *README.md*.  So yes,
testing would fail, but setup would fail before that.

*_librsyncmodule.c* is the C interface to librsync.so in the system
libraries.  It is compiled and linked against the system *librsync.so*
producing
*duplicity/_librsync.so* as a shim to the library.  On most systems, both
/usr/lib and /usr/local/lib are in the path for linker.  On the Mac and
some other systems, only /usr/lib is in the path for the linker, thus we
add *LIBRSYNC_DIR* during setup to account for that.  This is passed to
setup.py to add extra dirs to link against and include in compiles.

...Ken



On Mon, Oct 17, 2022 at 9:15 AM Scott Hannahs <sth@xxxxxxxxxx> wrote:

> Ken,
>
> Just as a followup, the only reference I can find to _librsync.cpython is
> in the build _librsync library that is built and in the build folder.
>
> It seems that your use of LIBRSYN_DIR is using an installed possibly
> previous version of librsync rather than the one built in the distribution
> folder?  Should the testing have LIBRSYNC_DIR set to some folder in the
> build directory structure rather than in the file system in general?
>
> Would the testing fail if this was an install in a clean system where
> duplicity (and librsync) had not already been installed?
>
> *-Scott*
>
>
> On Oct 10, 2022, at 11:42 AM, Kenneth Loafman <kenneth@xxxxxxxxxxx> wrote:
>
> Hi Scott,
>
> I would say that all your errors are from not being able to build
> *_librsync.so* properly.  There is one thing that may help in lines 65-76
> of *setup.py*.  You can supply *LIBRSYNC_DIR* as an env variable, or on
> the command line as *--librsync-dir=*.  I have
> *export LIBRSYNC_DIR=/usr/local* in my .bash_profile.  That came from
> *homebrew* installing into /usr/local instead of /usr.  Seems none of
> these installers care much for standards.
>
> Try exporting *LIBRSYNC_DIR* and let me know how it goes after that.
>
> ...Ken
>
>
> On Sun, Oct 9, 2022 at 4:46 PM Scott Hannahs <sth@xxxxxxxxxx> wrote:
>
>> So there are two major failures in testing.  One is four failures in the
>> tahoe backend, (which I don’t use).
>>
>> testing/unit/test_backend_instance.py ..............................FF..
>> [ 10%]
>> FF........................ssssssssss
>> [ 18%]
>>
>>
>> This shows a “Key Error” so is something wrong with my GPG version?
>>
>> self = <duplicity.backends.tahoebackend.TAHOEBackend object at
>> 0x10f58deb0>
>> commandline = 'tahoe cp
>> /var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/local var:folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output/file-a'
>>
>>     def subprocess_popen(self, commandline):
>>         u"""
>>         Execute the given command line with error check.
>>         Returns int Exitcode, string StdOut, string StdErr
>>
>>         Raise a BackendException on failure.
>>         """
>>         import shlex
>>
>>         if isinstance(commandline, (list, tuple)):
>>             logstr = u' '.join(commandline)
>>             args = commandline
>>         else:
>>             logstr = commandline
>>             args = shlex.split(commandline)
>>
>>         logstr = self.munge_password(logstr)
>>         log.Info(_(u"Reading results of '%s'") % logstr)
>>
>>         result, stdout, stderr = self.__subprocess_popen(args)
>>         if result != 0:
>>             try:
>> >               ignores = self.popen_breaks[args[0]]
>> E               KeyError:
>> '/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/overrides/bin/tahoe'
>>
>> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:522:
>> KeyError
>>
>> During handling of the above exception, another exception occurred:
>>
>> The second testing error is more general, so many of the tests pass and
>> then starting at the “bad upload” test it all goes south.
>>
>> testing/functional/test_badupload.py F
>> [ 70%]
>> testing/functional/test_cleanup.py FFF
>> [ 71%]
>> testing/functional/test_final.py FFFFFFFFFFFFFFFFFFFFFFFFFFF
>> [ 77%]
>> testing/functional/test_log.py F
>> [ 77%]
>> testing/functional/test_rdiffdir.py F
>> [ 77%]
>> testing/functional/test_replicate.py FFF
>> [ 78%]
>> testing/functional/test_restart.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
>> [ 86%]
>> testing/functional/test_selection.py .FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
>> [ 94%]
>> FFFFssFFFFFFFssss
>> [ 98%]
>>
>>
>> This test has a long back trace, But I think that the basic problem is
>> that duplicity cannot import “librsynch”???  The end of the traceback has
>> the text "b"ImportError: cannot import name '_librsync' from 'duplicity'
>> (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)"
>> ...return_val: 1"
>>
>> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py:527:
>> BackendException
>> _______________________ BadUploadTest.test_missing_file
>> ________________________
>>
>> self = <testing.functional.test_badupload.BadUploadTest
>> testMethod=test_missing_file>
>>
>>     @pytest.mark.slow
>>     def test_missing_file(self):
>>         u"""
>>         Test basic lost file
>>         """
>>         try:
>> >           self.backup(u"full",
>> u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"])
>>
>>
>> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:44:
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>> _ _ _
>>
>> self = <testing.functional.test_badupload.BadUploadTest
>> testMethod=test_missing_file>
>> type = 'full'
>> input_dir =
>> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1'
>> options = ['full',
>> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', '
>> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output',
>> '--volsize', '1', '--skip-volume=1']
>> kwargs = {}, before_files = set(), now = 1664937609.016365
>>
>>     def backup(self, type, input_dir, options=[], **kwargs):  # pylint:
>> disable=redefined-builtin
>>         u"""Run duplicity backup to default directory"""
>>         options = [type, input_dir, self.backend_url, u"--volsize", u"1"]
>> + options
>>         before_files = self.get_backend_files()
>>
>>         # If a chain ends with time X and the next full chain begins at
>> time X,
>>         # we may trigger an assert in dup_collections.py.  If needed,
>> sleep to
>>         # avoid such problems
>>         now = time.time()
>>         if self.last_backup == int(now):
>>             time.sleep(1)
>>
>> >       self.run_duplicity(options=options, **kwargs)
>>
>>
>> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:189:
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>> _ _ _
>>
>> self = <testing.functional.test_badupload.BadUploadTest
>> testMethod=test_missing_file>
>> options = ['full',
>> '/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1', '
>> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output',
>> '--volsize', '1', '--skip-volume=1']
>> current_time = None, fail = None, passphrase_input = []
>>
>>     def run_duplicity(self, options=[], current_time=None, fail=None,
>>                       passphrase_input=[]):
>>         u"""
>>         Run duplicity binary with given arguments and options
>>         """
>>         # We run under setsid and take input from /dev/null (below)
>> because
>>         # this way we force a failure if duplicity tries to read from the
>>         # console unexpectedly (like for gpg password or such).
>>
>>         # Check all string inputs are unicode -- we will convert to
>> system encoding before running the command
>>         for item in options:
>>             if sys.version_info.major == 2:
>>                 assert not isinstance(item, str), u"item " +
>> unicode(item) + u" in options is not unicode"
>>
>>         for item in passphrase_input:
>>             assert isinstance(item, u"".__class__), u"item " +
>> unicode(item) + u" in passphrase_input is not unicode"
>>
>>         if platform.platform().startswith(u'Linux'):
>>             cmd_list = [u'setsid']
>>             if self._setsid_w:
>>                 cmd_list.extend([u"-w"])
>>         else:
>>             cmd_list = []
>>         basepython = os.environ.get(u'TOXPYTHON', None)
>>         if basepython is not None:
>>             cmd_list.extend([basepython])
>>         run_coverage = os.environ.get(u'RUN_COVERAGE', None)
>>         if run_coverage is not None:
>>             cmd_list.extend([u"-m", u"coverage", u"run",
>> u"--source=duplicity", u"-p"])
>>         cmd_list.extend([u"{0}/bin/duplicity".format(_top_dir)])
>>         cmd_list.extend(options)
>>         cmd_list.extend([u"-v0"])
>>         cmd_list.extend([u"--no-print-statistics"])
>>         cmd_list.extend([u"--allow-source-mismatch"])
>>
>> cmd_list.extend([u"--archive-dir={0}/testfiles/cache".format(_runtest_dir)])
>>         if current_time:
>>             cmd_list.extend([u"--current-time", current_time])
>>         cmd_list.extend(self.class_args)
>>         if fail:
>>             cmd_list.extend([u"--fail", u"".__class__(fail)])
>>         cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
>>
>>         cmdline = u" ".join([u'"%s"' % x for x in cmd_list])
>>
>>         if not passphrase_input:
>>             cmdline += u" < /dev/null"
>>
>>         # The immediately following block is the nicer way to execute
>> pexpect with
>>         # unicode strings, but we need to have the pre-4.0 version for
>> some time yet,
>>         # so for now this is commented out so tests execute the same way
>> on all systems.
>>
>>         # if parse_version(pexpect.__version__) >= parse_version("4.0"):
>>         #     # pexpect.spawn only supports unicode from version 4.0
>>         #     # there was a separate pexpect.spawnu in 3.x, but it has an
>> error on readline
>>         #     child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline],
>> timeout=None, encoding=sys.getfilesystemencoding())
>>         #
>>         #     for passphrase in passphrase_input:
>>         #         child.expect(u'passphrase.*:')
>>         #         child.sendline(passphrase)
>>         # else:
>>
>>         # Manually encode to filesystem encoding and send to spawn as
>> bytes
>>         # ToDo: Remove this once we no longer have to support systems
>> with pexpect < 4.0
>>         if sys.version_info.major > 2:
>>             child = pexpect.spawn(u'/bin/sh', [u'-c', cmdline],
>> timeout=None)
>>         else:
>>             child = pexpect.spawn(b'/bin/sh', [b'-c',
>> cmdline.encode(sys.getfilesystemencoding(),
>>
>>  u'replace')], timeout=None)
>>
>>         for passphrase in passphrase_input:
>>             child.expect(b'passphrase.*:')
>>             child.sendline(passphrase)
>>
>>         # if the command fails, we need to clear its output
>>         # so it will terminate cleanly.
>>         child.expect_exact(pexpect.EOF)
>>         lines = child.before.splitlines()
>>         child.wait()
>>         child.ptyproc.delayafterclose = 0.0
>>         return_val = child.exitstatus
>>
>>         if fail:
>>             self.assertEqual(30, return_val)
>>         elif return_val:
>>             print(u"\n...command:", cmdline, file=sys.stderr)
>>             print(u"...cwd:", os.getcwd(), file=sys.stderr)
>>             print(u"...output:", file=sys.stderr)
>>             for line in lines:
>>                 line = line.rstrip()
>>                 if line:
>>                     print(line, file=sys.stderr)
>>             print(u"...return_val:", return_val, file=sys.stderr)
>>
>> >           raise CmdError(return_val)
>> E           testing.functional.CmdError: 1
>>
>> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/__init__.py:175:
>> CmdError
>>
>> During handling of the above exception, another exception occurred:
>>
>> self = <testing.functional.test_badupload.BadUploadTest
>> testMethod=test_missing_file>
>>
>>     @pytest.mark.slow
>>     def test_missing_file(self):
>>         u"""
>>         Test basic lost file
>>         """
>>         try:
>>             self.backup(u"full",
>> u"{0}/testfiles/dir1".format(_runtest_dir), options=[u"--skip-volume=1"])
>>             self.fail()
>>         except CmdError as e:
>> >           self.assertEqual(e.exit_status, 44, str(e))
>> E           AssertionError: 1 != 44 : 1
>>
>> /opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/testing/functional/test_badupload.py:47:
>> AssertionError
>> ----------------------------- Captured stderr call
>> -----------------------------
>>
>> ...command:
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity"
>> "full" "/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/dir1" "
>> file:///var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/output"
>> "--volsize" "1" "--skip-volume=1" "-v0" "--no-print-statistics"
>> "--allow-source-mismatch"
>> "--archive-dir=/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T/testfiles/cache"
>> < /dev/null
>> ...cwd: /private/var/folders/6b/85yg9bp94rv55dzbdp78thp40000kr/T
>> ...output:
>> b'Traceback (most recent call last):'
>> b'  File
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/bin/duplicity",
>> line 37, in <module>'
>> b'    from duplicity.dup_main import main'
>> b'  File
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_main.py",
>> line 50, in <module>'
>> b'    from duplicity import commandline'
>> b'  File
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/commandline.py",
>> line 43, in <module>'
>> b'    from duplicity import backend'
>> b'  File
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/backend.py",
>> line 44, in <module>'
>> b'    from duplicity import dup_temp'
>> b'  File
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/dup_temp.py",
>> line 34, in <module>'
>> b'    from duplicity import path'
>> b'  File
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/path.py",
>> line 49, in <module>'
>> b'    from duplicity import librsync'
>> b'  File
>> "/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/librsync.py",
>> line 36, in <module>'
>> b'    from . import _librsync'
>> b"ImportError: cannot import name '_librsync' from 'duplicity'
>> (/opt/sw/src/fink.build/duplicity-1.0.1-1/duplicity-1.0.1/duplicity/__init__.py)"
>> ...return_val: 1
>>
>>
>> On Oct 9, 2022, at 11:26 AM, Kenneth Loafman <kenneth@xxxxxxxxxxx> wrote:
>>
>> Ok, sounds good.  Any progress?
>>
>> ...Ken
>>
>>
>> On Thu, Oct 6, 2022 at 3:15 PM Scott Hannahs <sth@xxxxxxxxxx> wrote:
>>
>>> I will try to reinstall Xcode and the command line tools.  I had been
>>> using py38 which is installed in both the standard and an updated version
>>> in the non-standard location.
>>>
>>> On Oct 6, 2022, at 14:57, Kenneth Loafman <kenneth@xxxxxxxxxxx> wrote:
>>>
>>> OK,
>>>
>>> I could not remember what version of Python 3 you were running, so py3x
>>> meant for you to fill it in.  I'm guessing either py38 or py39 for Python
>>> 3.8 and Python 3.9 respectively.  Others follow the same model,
>>> py27,py35,...py310.
>>>
>>> I am not sure why it's trying to build for anything but x86_64 if you
>>> have Intel silicone and not Apple silicone.
>>>
>>> Here's what mine looks like for py38:
>>>
>>> ken@dione:~/workspace/duplicity-origin$ python3.8 ./setup.py build_ext
>>> Unable to get SCM version: defaulting to 1.0.1
>>> running build_ext
>>> Building extension for librsync...
>>> building 'duplicity._librsync' extension
>>> creating build/temp.macosx-12-x86_64-cpython-38
>>> creating build/temp.macosx-12-x86_64-cpython-38/duplicity
>>> clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common
>>> -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot
>>> /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/usr/local/include
>>> -I/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/include/python3.8
>>> -c duplicity/_librsyncmodule.c -o
>>> build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o
>>> creating build/lib.macosx-12-x86_64-cpython-38
>>> creating build/lib.macosx-12-x86_64-cpython-38/duplicity
>>> clang -bundle -undefined dynamic_lookup -isysroot
>>> /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
>>> build/temp.macosx-12-x86_64-cpython-38/duplicity/_librsyncmodule.o
>>> -L/usr/local/lib -lrsync -o
>>> build/lib.macosx-12-x86_64-cpython-38/duplicity/_
>>> librsync.cpython-38-darwin.so
>>> ld: warning: -undefined dynamic_lookup may not work with chained fixups
>>> copying build/lib.macosx-12-x86_64-cpython-38/duplicity/_
>>> librsync.cpython-38-darwin.so -> duplicity
>>>
>>>
>>> It looks like your xcode libs are out of date, and it also looks like
>>> you're not using the standard locations for software.  That's gonna
>>> complicate matters.
>>>
>>> I'm on Monterey 12.6 with the current Xcode from Apple, all in standard
>>> places.
>>>
>>> ...Ken
>>>
>>>
>>>
>>>
>>> On Thu, Oct 6, 2022 at 1:13 PM Scott Hannahs <sth@xxxxxxxxxx> wrote:
>>>
>>>> Ken,
>>>>
>>>> I will leave fink out, but it should be doing exactly the same thing!
>>>>  :-)
>>>>
>>>> the following doesn’t even build duplicity correctly, so I don’t get to
>>>> the tox command.
>>>>
>>>> % tar xf duplicity-1.0.1.tar
>>>> % cd duplicity-1.0.1
>>>> % % C_INCLUDE_PATH=/opt/sw/include LIBRARY_PATH=/opt/sw/lib python3
>>>> ./setup.py build_ext > build.out.txt 2>&1
>>>>
>>>>
>>>> This gives the following output.
>>>>
>>>> Unable to get SCM version: defaulting to 1.0.1
>>>> running build_ext
>>>> Building extension for librsync...
>>>> building 'duplicity._librsync' extension
>>>> clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common
>>>> -dynamic -DNDEBUG -g -fwrapv -O3 -Wall
>>>> -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders
>>>> -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers
>>>> -arch arm64 -arch x86_64 -Werror=implicit-function-declaration
>>>> -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/include/python3.9
>>>> -c duplicity/_librsyncmodule.c -o
>>>> build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o
>>>> clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64
>>>> -Wl,-headerpad,0x1000
>>>> build/temp.macosx-10.9-universal2-3.9/duplicity/_librsyncmodule.o -lrsync
>>>> -o build/lib.macosx-10.9-universal2-3.9/duplicity/_
>>>> librsync.cpython-39-darwin.so
>>>> ld: warning: ignoring file /opt/sw/lib/librsync.dylib, building for
>>>> macOS-arm64 but attempting to link with file built for macOS-x86_64
>>>> ld: warning: dylib (/opt/sw/lib/librsync.dylib) was built for newer
>>>> macOS version (12.4) than being linked (10.9)
>>>> copying build/lib.macosx-10.9-universal2-3.9/duplicity/_
>>>> librsync.cpython-39-darwin.so -> duplicity
>>>>
>>>>
>>>> I am building on an X86 machine and my libraries are x86 and I am not
>>>> trying to build a universal arm 64 system.   But with warnings it tries to
>>>> build.
>>>>
>>>> tox does not like the py3x value
>>>>
>>>> % tox -epy3
>>>> py3 create: /Users/sth/Downloads/duplicity-1.0.1/.tox/py3
>>>> py3 installdeps: -rrequirements.txt
>>>>
>>>> and I ran out of time this AM waiting for more output.
>>>>
>>>> -Scott
>>>>
>>>>
>>>> On Oct 5, 2022, at 11:54 AM, Kenneth Loafman <kenneth@xxxxxxxxxxx>
>>>> wrote:
>>>>
>>>> Could you try just building and running the tests in a 'normal' user
>>>> environment?  Leave Fink out of the picture.
>>>>
>>>> do this:
>>>>
>>>>    1. $ tar xf duplicity-1.01.tar.gz
>>>>    2. $ ./setup.py build_ext
>>>>    3. $ tox -e py3x
>>>>
>>>> ...Thanks,
>>>> ...Ken
>>>>
>>>>
>>>>
>>>
>>
>