aims team mailing list archive
-
aims team
-
Mailing list archive
-
Message #01654
[Bug 1384503] Re: rsync fails on large files with compression
We have a few servers that are exhibiting this behaviour while copying
backups sets. I cannot make any of that data available, but have tried
producing some /dev/urandom produced data files that exhibit the error
and have succeeded in getting something I hope is useful.
I discovered by accident while running a loop that generated and rsyncd
test data that interrupting the transfer with ctrl-c produces a failing
test case that works without compression but fails with compression.
I did not test rigorously, but the files do seem to need to be large,
but zero filled seems to be okay. So the test data files I'm attaching
are zero filled and only the beginning is random data.
Here is a recipe for producing the data
# note: needs working ssh to localhost, does not error on a local rsync
# setup for test
cd ~/
mkdir source dest
# create a large file
dd if=/dev/zero of=source/random-data-0 bs=1M count=1500
# only need first part of file with random data
dd if=/dev/urandom of=source/random-data-0 bs=1M count=50 conv=notrunc
# copy it once successfully
rsync -vv -ia --inplace --compress --checksum -P source/ localhost:dest/
# change the first part of the file with new data
dd if=/dev/urandom of=source/random-data-0 bs=1M count=50 conv=notrunc
# copy it again, but break transfer within the first part of file
# by typing ctrl-c key during this rsync
rsync -vv -ia --inplace --compress --checksum -P source/ localhost:dest/
# rsync again should will exhibit error near transfer break point
rsync -vv -ia --inplace --compress --checksum -P source/ localhost:dest/
--
You received this bug notification because you are a member of AIMS,
which is subscribed to a duplicate bug report (1431709).
https://bugs.launchpad.net/bugs/1384503
Title:
rsync fails on large files with compression
Status in rsync package in Ubuntu:
Fix Released
Status in rsync source package in Trusty:
Incomplete
Bug description:
[Status]
This issue is currently Incomplete.
We're currently blocked on obtaining access to a reliable, and publicly
available dataset that can be shared to reproduce the issue. The test
is critical to being able to evaluate the impact of the change on other
users when assessing if the fix can be SRU'ed to trusty.
If you are affected by this bug and can reliably reproduce the issue
with a specific dataset that you are willing and permitted to share please
comment and specify the dataset location, and how you reproduce the issue.
[Original Description]
Copying large (>10GB) files with rsync -z (compression) leads to a long hang and eventual error after transferring part of the file. The error is consistent. The file copies at normal speed until it reaches its maximum size (1.4 GB out of 20 GB for one, 6.9 GB out of 29 GB for another). Then nothing happens for a while (many minutes). Finally, there is an error:
[....]
jh/.VirtualBox/win7/win7.vbox
jh/.VirtualBox/win7/win7.vbox-prev
jh/.VirtualBox/win7/win7.vdi
rsync: [sender] write error: Broken pipe (32)
rsync error: error in rsync protocol data stream (code 12) at io.c(837) [sender=3.1.0]
In this case, 6.9 GB of 29 GB transferred. Without -z, it works.
See the following upstream report, with a comment at the end from the
rsync maintainer:
https://bugzilla.samba.org/show_bug.cgi?id=10372
According to this report, version 3.1.0 (included in 14.04) uses a
different compression package from prior versions. Prior versions did
not have this problem for me using the same command on the same
systems. Both hosts ran Ubuntu 11.10 at the time, and all run 14.04
now, in each case with all updates applied, Intel hardware. Network
connection between them is gigabit ethernet through one switch. A
shell ssh between them in a terminal works and stays up during the
failure, so it is not a network issue. There are no relevant entries
in syslog on either machine. There is sufficient capacity on the
receiving disk. All filesystems are ext4.
rsync command:
/usr/bin/rsync -aHSxvz --delete --stats --exclude=lost+found
--exclude=.gvfs --exclude=/nonlaptop /home/
backup.host.edu:/bu/host/home/
(yes, I changed the machine names)
Current release (both hosts):
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Current package (both hosts):
rsync:
Installed: 3.1.0-2ubuntu0.1
Candidate: 3.1.0-2ubuntu0.1
Version table:
*** 3.1.0-2ubuntu0.1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
100 /var/lib/dpkg/status
3.1.0-2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
Thanks,
--jh--
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions