holland-discuss team mailing list archive

[Tim Soderstrom <tim@xxxxxxxxxxxxxxxxxxxxx>]

Anyone had time to look at this and/or the PR I sent? Rsync plugin has been running for quite a while on my NAS and working rather well:

root@filedawg:/var/spool/holland/rsync-home# du -sch *
1.2T	20151003_020717
365M	20151004_020621
819M	20151005_020659
302M	20151006_020705
261M	20151007_020627
423M	20151008_020658
0	newest
0	oldest
1.2T	total

Tim


> On Sep 17, 2015, at 9:59 PM, Tim Soderstrom <tim@xxxxxxxxxxxxxxxxxxxxx> wrote:
> 
> Yep no worries on the time. I'm there quite often as well.
> 
> Thanks for the subprocess info! That was quite helpful. While those a try, I ended up beating it by using '-e' instead of '--rsh'. Seems to behave better, although building the sub-command is kinda ugly, though it does seem to work. So I sort of didn't solve the underlying issue I was running into so shame on me, but I did at least come up with an acceptable work-around for this.
> 
> I was thinking today about perhaps looking at splitting out the SSH stuff into an [ssh] section should a future plugin need it (to promote code reuse and all that) but I think the first goal is to get things functional first.
> 
> Haha I forgot you actually dislike the estimation functions :) I'm generally cool having it return 0 until a better solution is realized (which could be up to and including never).  My only gripe is this bit in the holland.log:
> 
> 2015-09-17 17:54:01,867 [INFO] Estimated Backup Size: 0.00B
> 
> To avoid that, I modified some Holland core code but it's an easy modification to the 'check_available_space' function to only do space checking when estimated_bytes_required is not None. I'd imagine that /could/ have implications elsewhere though (I can't think of any offhand though)? Thus I'd consider
> that both optional and more aesthetic.
> 
> Tim
> 
> On 09/17/2015 07:40 PM, Andrew Garner wrote:
>> I haven't had a lot of time this week to look at this in detail.  But
>> here are some inline comments.
>> 
>> On Thu, Sep 17, 2015 at 6:09 PM, Tim Soderstrom
>> <tim@xxxxxxxxxxxxxxxxxxxxx> wrote:
>>> I still haven't figured out the annoying lists quote issue noted below,
>> subprocess.list2cmdline is pretty basic and for complex stuff the
>> output is pretty bad.  I don't think it was really designed to handle
>> complex bits.  Per the subprocess module documentation "list2cmdline()
>> is designed for applications using the same rules as the MS C runtime"
>> 
>> I use one of three approach to deal with it:
>> 
>> - (most common) ignore it and treat the commandline output from
>> list2cmdline as "close enough" :)
>> - rely on some third party library that handles quoting args a little
>> better, like sarge
>> - roll my own quoting function.
>> 
>> You can look at the sarge.shell_quote implementation which is
>> relatively simple and see how it handles it:
>> 
>> https://bitbucket.org/vinay.sajip/sarge/src/1408b253c82da3dcaaeb7b1704a5e08d895dce34/sarge/__init__.py?at=default&fileviewer=file-view-default#__init__.py-68
>> 
>> To emulate list2cmdline I sometime do something like:
>> 
>> ' '.join(map(sarge.shell_quote, args))
>> 
>> Which might give output like this:
>> 
>> rsync -avx --link-dest=/var/spool/holland/ssh/20150915_175342
>> --exclude=Private --exclude=.cache --exclude=Steam
>> --exclude=.thunderbird --exclude=.mozilla '--rsh=ssh -p' --bwlimit=1k
>> --one-file-system filedawg.moocow.home://home/tim/tmp
>> /var/spool/holland/ssh/20150915_175354
>> 
>>> I also disabled backup size estimation since that was proving to be really
>>> expensive, and could be inaccurate when using --one-file-system, not to
>>> mention won't work at all for remote sources (rsync's --dry-run could be
>>> used for that case). So for now it just returns 0, which is fugly. Need a
>>> more elegant solution I think.
>> Yeah.  I think the backup size estimation is one of the main
>> misfeatures in holland.   It can be really expensive and is almost
>> always inaccurate and leads to false failures or sometimes
>> underestimates.  I wouldn't feel too bad about always returning 0 and
>> just documenting the behavior.
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~holland-discuss
> Post to     : holland-discuss@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~holland-discuss
> More help   : https://help.launchpad.net/ListHelp