holland-discuss team mailing list archive

[Tim Soderstrom <tim@xxxxxxxxxxxxxxxxxxxxx>]

I still haven't figured out the annoying lists quote issue noted below, 
but I did add logging of rsync's output (to a file in the backup-dir as 
that made the most sense to me since it'd be super cluttered in the 
holland log itself), fixed it so it builds .DEBs using the build_debs.py 
script, and some tweaks to better handle defaults in the config.

I also disabled backup size estimation since that was proving to be 
really expensive, and could be inaccurate when using --one-file-system, 
not to mention won't work at all for remote sources (rsync's --dry-run 
could be used for that case). So for now it just returns 0, which is 
fugly. Need a more elegant solution I think.

Apart from that, it does what I want it to do, e.g.:

############################################################
root@filedawg:/var/spool/holland/rsync-boot# du -sch *
142M    20150917_165926
24K    20150917_165934
24K    20150917_170055
24K    20150917_170056
24K    20150917_175400
24K    20150917_175401
0    newest
0    oldest
142M    total
root@filedawg:/var/spool/holland/rsync-boot# du -sch 20150917_175401
142M    20150917_175401
142M    total
root@filedawg:/var/spool/holland/rsync-boot# find newest/ | wc -l
311
root@filedawg:/var/spool/holland/rsync-boot# find oldest/ | wc -l
311
############################################################

Tim


On 09/15/2015 05:59 PM, Tim Soderstrom wrote:
> I've made more progress with it. It now supports remote sync (SSH, 
> rsync) as well as some one-file-system and bandwidth-limit options. 
> Those are basically -- style flags I've exposed as a specific config 
> option. As I recall the idea was to sort of move away from that in 
> future plugins and, instead letting users specifying their own flags 
> in a more free-flow method? I think I'd want to have that for some 
> obscure flags but some of the more useful ones I think I like having 
> as a specific option. I find it makes things easier to read? But I dunno.
>
> Currently lacking is the ability to specify a custom port. Having 
> trouble with Python wanting to do things like this:
>
> ###
> Executing: rsync -avx 
> --link-dest=/var/spool/holland/ssh/20150915_175342 --exclude=Private 
> --exclude=.cache --exclude=Steam --exclude=.thunderbird 
> --exclude=.mozilla "--rsh='ssh \-p" --bwlimit=1k --one-file-system 
> filedawg.moocow.home://home/tim/tmp 
> /var/spool/holland/ssh/20150915_175354
> ###
>
> I tried quite a few ways to avoid quotes being added. It seems to not 
> like when I add the - in -p and I'm not sure how to disable that.
>
> Otherwise, while I haven't been doing an exhaustive test of all 
> combinations but things seem to be working ok, save for testing the 
> SSH method. It should work, but I just haven't gotten around to it 
> yet. The rsync method (whether authenticated or anonymous) does seem 
> to work however. I'd like to have a way to test things in a more 
> automated away but it could get somewhat complicated.
>
> I've also added the provider and default backup configs with some 
> docs, save for the extended options since I'm not sure if I'll keep 
> those yet.
>
> Tim
>
> On 09/14/2015 06:29 PM, Tim Soderstrom wrote:
> > It's still pretty bare but I've setup a basic rsync plugin which 
> > supports hardlinking. I had to do some interesting things in regards 
> > to looking at previous backup-sets and could use some feedback if 
> > anyone wants to take a look:
> >
> > https://github.com/m00dawg/holland/tree/master/plugins/holland.backup.rsync 
> >
> >
> > Hardlinking seems to work - it basically sets --link-dest to the 
> > previous backup set if it exsts and if hardlinking is enabled.
> >
> > Hideously lacking is the ability to setup exclusions, and I'm not 
> > sure if I should just offer flags/extended_flags options or be nice 
> > and make certain features explicit (such as --one-file-system). Also 
> > not sure if I should support --exclude-file= or just use the 
> > --exclude flag and iterate over a list (similar to how mysqldump 
> > works as I recall). I should also probably support remote sources (I 
> > don't have a need for that in my case but I'm sure lots of folks would).
> >
> > Tim
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~holland-discuss
> > Post to     : holland-discuss@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~holland-discuss
> > More help   : https://help.launchpad.net/ListHelp
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~holland-discuss
> Post to     : holland-discuss@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~holland-discuss
> More help   : https://help.launchpad.net/ListHelp