openstack team mailing list archive

Thread
Date

Bad rsync return code after adding two new disks

To: openstack@xxxxxxxxxxxxxxxxxxx
From: Sharif Islam <islamsh@xxxxxxxxxxx>
Date: Fri, 11 Nov 2011 12:18:09 -0500
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110921 Thunderbird/3.1.15

I added two new new disks: sdb1 in 192.168.0.12 and sdb1 in
192.168.0.13. (both of these storage nodes already had sda3 which was in
zone 2  and 3 respectively). I added a new zone and these two new disks
in that zone. Then did rebalance and distributed the ring files all
across the cluster. Did I miss any other step?

I started noticing these errors (there are some successful rsync as well).


> Nov 11 12:12:48 b001 object-replicator Successful rsync of /srv/node/sda3/objects/19498/86d at [192.168.0.12]::object/sdb1/objects/19498 (0.209)
> Nov 11 12:12:48 b001 object-replicator Bad rsync return code: ['rsync', '--recursive', '--whole-file', '--human-readable', '--xattrs', '--itemize-changes', '--ignore-existing', '--timeout=30', '--contimeout=30', '/srv/node/sda3/objects/51119/350', '[192.168.0.12]::object/sdb1/objects/51119'] -> 5
> Nov 11 12:12:48 b001 object-replicator Bad rsync return code: ['rsync', '--recursive', '--whole-file', '--human-readable', '--xattrs', '--itemize-changes', '--ignore-existing', '--timeout=30', '--contimeout=30', '/srv/node/sda3/objects/32937/efe', '[192.168.0.12]::object/sdb1/objects/32937'] -> 5


Same with .13:

> Nov 11 12:13:43 b001 object-replicator Successful rsync of /srv/node/sda3/objects/11047/10d at [192.168.0.13]::object/sdb1/objects/11047 (0.168)
> Nov 11 12:13:43 b001 object-replicator Bad rsync return code: ['rsync', '--recursive', '--whole-file', '--human-readable', '--xattrs', '--itemize-changes', '--ignore-existing', '--timeout=30', '--contimeout=30', '/srv/node/sda3/objects/92247/966', '[192.168.0.13]::object/sdb1/objects/92247'] -> 5
> Nov 11 12:13:43 b001 object-replicator Bad rsync return code: ['rsync', '--recursive', '--whole-file', '--human-readable', '--xattrs', '--itemize-changes', '--ignore-existing', '--timeout=30', '--contimeout=30', '/srv/node/sda3/objects/50409/be4', '[192.168.0.13]::object/sdb1/objects/50409'] -> 5


any suggestions?

I also noticed this in .12:

Nov 11 12:16:16 b002 object-replicator @ERROR: max connections (2)
reached -- try again later


--sharif


-- 
Sharif Islam
Senior Systems Analyst/Programmer
FutureGrid (http://futuregrid.org)
Pervasive Technology Institute, Indiana University Bloomington