← Back to team overview

holland-discuss team mailing list archive

Re: rsync plugin

 

Yep no worries on the time. I'm there quite often as well.

Thanks for the subprocess info! That was quite helpful. While those a try, I ended up beating it by using '-e' instead of '--rsh'. Seems to behave better, although building the sub-command is kinda ugly, though it does seem to work. So I sort of didn't solve the underlying issue I was running into so shame on me, but I did at least come up with an acceptable work-around for this.

I was thinking today about perhaps looking at splitting out the SSH stuff into an [ssh] section should a future plugin need it (to promote code reuse and all that) but I think the first goal is to get things functional first.

Haha I forgot you actually dislike the estimation functions :) I'm generally cool having it return 0 until a better solution is realized (which could be up to and including never). My only gripe is this bit in the holland.log:

2015-09-17 17:54:01,867 [INFO] Estimated Backup Size: 0.00B

To avoid that, I modified some Holland core code but it's an easy modification to the 'check_available_space' function to only do space checking when estimated_bytes_required is not None. I'd imagine that /could/ have implications elsewhere though (I can't think of any offhand though)? Thus I'd consider
that both optional and more aesthetic.

Tim

On 09/17/2015 07:40 PM, Andrew Garner wrote:
I haven't had a lot of time this week to look at this in detail.  But
here are some inline comments.

On Thu, Sep 17, 2015 at 6:09 PM, Tim Soderstrom
<tim@xxxxxxxxxxxxxxxxxxxxx> wrote:
I still haven't figured out the annoying lists quote issue noted below,
subprocess.list2cmdline is pretty basic and for complex stuff the
output is pretty bad.  I don't think it was really designed to handle
complex bits.  Per the subprocess module documentation "list2cmdline()
is designed for applications using the same rules as the MS C runtime"

I use one of three approach to deal with it:

- (most common) ignore it and treat the commandline output from
list2cmdline as "close enough" :)
- rely on some third party library that handles quoting args a little
better, like sarge
- roll my own quoting function.

You can look at the sarge.shell_quote implementation which is
relatively simple and see how it handles it:

https://bitbucket.org/vinay.sajip/sarge/src/1408b253c82da3dcaaeb7b1704a5e08d895dce34/sarge/__init__.py?at=default&fileviewer=file-view-default#__init__.py-68

To emulate list2cmdline I sometime do something like:

' '.join(map(sarge.shell_quote, args))

Which might give output like this:

rsync -avx --link-dest=/var/spool/holland/ssh/20150915_175342
--exclude=Private --exclude=.cache --exclude=Steam
--exclude=.thunderbird --exclude=.mozilla '--rsh=ssh -p' --bwlimit=1k
--one-file-system filedawg.moocow.home://home/tim/tmp
/var/spool/holland/ssh/20150915_175354

I also disabled backup size estimation since that was proving to be really
expensive, and could be inaccurate when using --one-file-system, not to
mention won't work at all for remote sources (rsync's --dry-run could be
used for that case). So for now it just returns 0, which is fugly. Need a
more elegant solution I think.
Yeah.  I think the backup size estimation is one of the main
misfeatures in holland.   It can be really expensive and is almost
always inaccurate and leads to false failures or sometimes
underestimates.  I wouldn't feel too bad about always returning 0 and
just documenting the behavior.



Follow ups

References