← Back to team overview

c4c-dev team mailing list archive

Re: Large Binary Files

 

HI Eric,

You don't need the QT Opensource Installer .. that's for QT application
development, which we are not doing at this point; it was just an
example of lengthy downloads and accepting the EULA.

Everything we need QT wise will be provided by the base Installation
ISO. We don't need to be concerned about those packages.

best regards,

Greg.



On 05/18/2015 11:07 AM, Eric Bradshaw wrote:
> Guys,
> 
> Again, no expertise - but, my thought is we may want to use the QT Open
> Source installer simply because Lubuntu is transitioning to LXQt
> (supposedly) by 15.10 - so maybe we'd be prepared? Apologies if one
> would have nothing to do with the other.
> 
> Eric
> 
> On 05/18/2015 09:14 AM, KI7MT wrote:
>> Hi Israel, All,
>>
>> Yes, zsync, rsync or even curl could do that job easily. A simple shell
>> script could do what is needed here.
>>
>> If you have ever used MSYS/MinGW, this is exactly how their installer
>> works for pulling packages to install form the SF server. The same for
>> the QT Open Source installer, and it pull a is HUGE amount of data at
>> install.
>>
>>
>> best regards,
>>
>> Greg.
>>
>> On 05/18/2015 07:43 AM, Israel wrote:
>>> Hi everyone...
>>> I have thought about this issue.
>>> 2 things
>>> zsync is the appropriate tool for this!
>>>
>>> we could host all our content on a c4c server.
>>> then the package could simply be a dummy package that requires zsync,
>>> and then postinst
>>> runs something like:
>>> cd /usr/share/c4c/Music
>>> zsync http:://c4cserver.org/ot.tar.xz.zsync
>>>
>>> then unpacks the tarball and syncs it to the user's Music folder.  As
>>> long as the user can read the files (not modify them... file
>>> permissions)  we will be good to go.
>>>
>>> Another good idea would be to include in the MetaData of the music a
>>> copyright notice
>>>
>>> Also, including some sort of dialog (zenity or xterm + dialog would be
>>> great for this) to say:
>>> "$LICENSE_AGREEMENT
>>> Do you accept these terms of use?"
>>> where LICENSE_AGREEMENT=""#whatever you want to put as the agreement
>>>
>>> zsync will easily mitigate the HUGE amount of stuff that needs to be
>>> downloaded, and also automatically checks md5sum and can resume if
>>> internet connection is broken
>>>
>>>
>>>
>>> On 05/17/2015 04:50 PM, KI7MT wrote:
>>>> Hi Eric,
>>>>
>>>> It's not so much the change in material itself that is the issue,
>>>> rather, it's the history of change within the branch that will cause us
>>>> problems.
>>>>
>>>> I've been looking for solutions too this, but have not found a suitable
>>>> one alternative yet, aside from a downloading system that the user
>>>> initiates. Not to mention, having restricted contest in an open source
>>>> management system is really not appropriate.
>>>>
>>>> Including the restricted content with each system, by some means of
>>>> downloading after usage acceptance would be fine, as long as the
>>>> location is outside of the VCS system on Launchpad. Mixing and matching
>>>> Restricted / Copyrighted material with Open Source License material in
>>>> the same branch location should be avoided if at all possible.
>>>>
>>>> This happens allot with application like Wine (Microsoft DLL's),
>>>> restricted codecs (MP3, MP4 etc), Microsoft True fonts, but those files
>>>> have a small disk space foot print. It's very grey area to say the
>>>> least.
>>>>
>>>> If it were my choice, I would not post restricted material in the VCS
>>>> system.
>>>>
>>>>
>>>> best regards,
>>>>
>>>> Greg
>>>>
>>>> On 05/17/2015 03:13 PM, Eric Bradshaw wrote:
>>>>> Guys,
>>>>>
>>>>> Many of the C4C-Specific files really are huge, but none of them have
>>>>> much a chance of being updated, especially between major releases.
>>>>> Here's the top 7 potential packages - biggest first - (size after
>>>>> expansion/install):
>>>>>
>>>>> 662.5 MB The Gospel of John, and 1st, 2nd, and 3rd John in MP3. It's
>>>>> included because it is one of the best, if not the best book in
>>>>> which to
>>>>> start reading the Bible and it's wonderfully read and explained (as
>>>>> this
>>>>> church does), in order, verse by verse. This wouldn't even be possible
>>>>> to change or add to for years and years.
>>>>> 637.3 MB The Entire Old Testament in MP3, World English Bible (WEB)
>>>>> version. This will not be added to or changed.
>>>>> 303.8 MB Christian Music MP3s. There is a possibility that this may be
>>>>> updated - expanded only - if we get permission from additional
>>>>> Christian
>>>>> recording artists. But, they'd have to offer because I haven't asked
>>>>> anyone for permission to use their music in a good while. It took
>>>>> many,
>>>>> many months before I got as music as I did - and frankly, it was
>>>>> exhausting. If we get any additional music, we can certainly make sure
>>>>> we don't add it in until the next LTS release.
>>>>> 191.7 MB The pre-installed and indexed Bibles, Commentaries,
>>>>> Devotionals, Dictionaries, General Books and Maps for use with Bible
>>>>> Desktop and Xiphos. There is a possibility one or two modules will be
>>>>> updated between LTS releases, but we can still choose not to update
>>>>> our
>>>>> package.
>>>>> 170.7 MB The Entire New Testament in MP3, World English Bible (WEB)
>>>>> version. This will not be added to or changed.
>>>>> 54.6 MB OGG Videos. No reason we would need to change these videos -
>>>>> even add to them, except or until an LTS release.
>>>>> 30.5 MB SWF Videos. Same as the OGG Videos above.
>>>>>
>>>>> Eric
>>>>>
>>>>> On 05/17/2015 12:17 PM, KI7MT wrote:
>>>>>> HI Eric,
>>>>>>
>>>>>> Wow, these are *huge* files, but it's good to have this breakdown, as
>>>>>> each is potentially a package that would need creating
>>>>>> specifically for
>>>>>> C4C or at least, some level of customization.
>>>>>>
>>>>>> Here's the problem (at least one, but its a big one). When you
>>>>>> check out
>>>>>> a branch, using BZR, Git etc, you get the full history each time.
>>>>>> If a
>>>>>> package is updated say two or three time over the course of the
>>>>>> release
>>>>>> cycle, you would have 2 to 3 time that much disk space usage and
>>>>>> Upload
>>>>>> / Download time while branching, merging, and pushing back to the
>>>>>> branch.
>>>>>>
>>>>>> For christian-john-123.tar.gz (600+ MB), that would equate to
>>>>>> =>2.4GB of
>>>>>> storage if the updates were stored in whole at each revision. And
>>>>>> that
>>>>>> is for just the one package.
>>>>>>
>>>>>> I did not realize the content of these packages was so large.
>>>>>> Clearly we
>>>>>> need a better solution for dealing with these large files, as the
>>>>>> problem of download / upload time for the branch will grow
>>>>>> exponentially
>>>>>> using a version control system.
>>>>>>
>>>>>> best regards,
>>>>>>
>>>>>> Greg.
>>>>>>
>>>>>> On 05/17/2015 11:58 AM, Eric Bradshaw wrote:
>>>>>>> Guys,
>>>>>>>
>>>>>>> I know it took me an hour to upload the huge 1404-2.zip (2.1 GB)
>>>>>>> file to
>>>>>>> the shared C4C Folder in Dropbox, so it's probably going to take
>>>>>>> triple
>>>>>>> that for anyone to download via DSL, or satellite. So, I've uploaded
>>>>>>> "byte-size" chunks of it too, tar-gz files, split by where it
>>>>>>> ends up
>>>>>>> (but, still in the same order as the bullet lists). Israel; this may
>>>>>>> also help explain why there are a ton of binary files. The list
>>>>>>> below is
>>>>>>> also a text file called TAR-GZ-FILES. Oh, and the big one is
>>>>>>> still up
>>>>>>> there too if you've got a fast connection and/or just want
>>>>>>> everything at
>>>>>>> once.
>>>>>>>
>>>>>>> ADD BIBLES, COMMENTARIES, MAPS, BOOKMARKS & FILE SETTINGS
>>>>>>> dot-jsword.tar.gz [15.3MB] expands to dot-jsword, the contents of
>>>>>>> which
>>>>>>> go into ~/.jsword
>>>>>>> dot-mozilla.tar.gz [3.4MB] expands to dot-mozilla, the contents
>>>>>>> of which
>>>>>>> go into ~/.mozilla
>>>>>>> dot-sword.tar.gz [144.8MB] expands to dot-sword, the contents of
>>>>>>> which
>>>>>>> go into ~/.sword
>>>>>>> dot-xiphos.tar.gz [2.7KiB] expands to dot-xiphos, the contents of
>>>>>>> which
>>>>>>> go into ~/.xiphos
>>>>>>> etc-gnome.tar.gz [1.7KiB] expands to etc-gnome, the contents of
>>>>>>> which
>>>>>>> (defaults.list) go into/etc/gnome
>>>>>>> etc-xdg-lubuntu-applications.tar.gz [1.7KiB] expands to
>>>>>>> etc-xdg-lubuntu-applications, the contents of which
>>>>>>> (defaults.list) go
>>>>>>> into/etc/xdg/lubuntu/applications
>>>>>>>
>>>>>>> ADD C4C SPECIFIC STUFF
>>>>>>> dot-config.tar.gz [19.2KiB] expands to dot-config, the contents
>>>>>>> of which
>>>>>>> go into ~/.conf
>>>>>>> share.tar.gz [3.7KiB] expands to share, the contents of which go
>>>>>>> into
>>>>>>> ~/.local/share
>>>>>>> christian-12-apostles.tar.gz [592KiB] expands to 12-apostles,
>>>>>>> which goes
>>>>>>> into /usr/share/christian
>>>>>>> christian-audio-bible.tar.gz [16KiB] expands to audio-bible,
>>>>>>> which goes
>>>>>>> into /usr/share/christian
>>>>>>> christian-believe.tar.gz [3.3MB] expands to believe, which goes into
>>>>>>> /usr/share/christian
>>>>>>> christian-bkgs.tar.gz [8.2MB] expands to bkgs, which goes into
>>>>>>> /usr/share/christian
>>>>>>> christian-bvmq.tar.gz [528KiB] expands to bvmq, which goes into
>>>>>>> /usr/share/christian
>>>>>>> christian-christian-music.tar.gz [296.8MB] expands to
>>>>>>> christian-music,
>>>>>>> which goes into /usr/share/christian
>>>>>>> christian-christian-videos.tar.gz [16KiB] expands to
>>>>>>> christian-videos,
>>>>>>> which goes into /usr/share/christian
>>>>>>> christian-disciple.tar.gz [2MB] expands to disciple, which goes into
>>>>>>> /usr/share/christian
>>>>>>> christian-fbgs.tar.gz [7.8MB] expands to fbgs, which goes into
>>>>>>> /usr/share/christian
>>>>>>> christian-icons.tar.gz [176KiB] expands to icons, which goes into
>>>>>>> /usr/share/christian
>>>>>>> christian-john-123.tar.gz [644.6MB] expands to john-123, which
>>>>>>> goes into
>>>>>>> /usr/share/christian
>>>>>>> christian-new-testament.tar.gz [158.2MB] expands to
>>>>>>> new-testament, which
>>>>>>> goes into /usr/share/christian
>>>>>>> christian-ogg-video.tar.gz [54.2MB] expands to ogg-video, which goes
>>>>>>> into /usr/share/christian
>>>>>>> christian-old-testament.tar.gz [591.5MB] expands to
>>>>>>> old-testament, which
>>>>>>> goes into /usr/share/christian
>>>>>>> christian-swf-video.tar.gz [24.6MB] expands to swf-video, which goes
>>>>>>> into /usr/share/christian
>>>>>>> doc.tar.gz [816KiB] expands to doc, the contents of which go into
>>>>>>> /usr/share/doc
>>>>>>> lubuntu.tar.gz [21.8MB] expands to lubuntu, the contents of which go
>>>>>>> into /usr/share/lubuntu
>>>>>>> themes.tar.gz [32KiB] expands to themes, the contents of which go
>>>>>>> into
>>>>>>> /lib/plymouth/themes/
>>>>>>> applications.tar.gz [16KiB] expands to applications, the contents of
>>>>>>> which go into /usr/share/applications
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>> On 05/17/2015 08:23 AM, KI7MT wrote:
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> Nice work!
>>>>>>>>
>>>>>>>> I've only browsed through each section briefly, but I think
>>>>>>>> most, if
>>>>>>>> not
>>>>>>>> all could be done with a single shell script, at least, when
>>>>>>>> building a
>>>>>>>> box locally. Doing all this from the command line would make
>>>>>>>> things a
>>>>>>>> bit easier also.
>>>>>>>>
>>>>>>>> Now we need to look at each action and determine if is a system
>>>>>>>> wide
>>>>>>>> task or user specific task then determine the best way
>>>>>>>> accomplish the
>>>>>>>> task programatically. A good could be, after the box is
>>>>>>>> configured the
>>>>>>>> way you like, create a new user and determine what needs doing.
>>>>>>>>
>>>>>>>>
>>>>>>>> best regards
>>>>>>>>
>>>>>>>> Greg
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 05/16/2015 09:57 PM, Eric Bradshaw wrote:
>>>>>>>>> Guys,
>>>>>>>>>
>>>>>>>>> I've installed the C4C ReSpin of 14.04.2 on a 64-bit machine
>>>>>>>>> and made
>>>>>>>>> both a High Level Action List called "Bullet-Points" and one
>>>>>>>>> with more
>>>>>>>>> detail called "Bullet-List-Expanded" (sorry - couldn't help it)
>>>>>>>>> attached
>>>>>>>>> to this email and uploaded to the C4C shared folder. I've also
>>>>>>>>> uploaded
>>>>>>>>> a zipped 1404-2 folder with all the files I used and reference
>>>>>>>>> in the
>>>>>>>>> Bullet-List-Expanded text file.
>>>>>>>>>
>>>>>>>>> I release now why Greg was confused about why I was un-installing
>>>>>>>>> apps.
>>>>>>>>> It was because I never could get what I wanted to start with by
>>>>>>>>> installing the Lubuntu-base - I always started with Lubuntu
>>>>>>>>> Desktop.
>>>>>>>>> Plus everything on my home network has a static IP, so it's
>>>>>>>>> easier to
>>>>>>>>> set the Internet connection from a Desktop ISO to do the initial
>>>>>>>>> install.
>>>>>>>>>
>>>>>>>>> Anyway - I hope I'm helping.
>>>>>>>>>
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>> On 05/16/2015 01:09 PM, KI7MT wrote:
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> Just a suggestion; I would recommend using a .txt document rather
>>>>>>>>>> than
>>>>>>>>>> .odt files for your high level doc. Text formatting is a bit
>>>>>>>>>> of an
>>>>>>>>>> art,
>>>>>>>>>> but when done properly, is reansferable to almost anything, likst
>>>>>>>>>> text2html for example.
>>>>>>>>>>
>>>>>>>>>> Most developers prefer .txt docs for Emails as well as
>>>>>>>>>> procedures,
>>>>>>>>>> as it
>>>>>>>>>> is much easier to read / edit / and merge.
>>>>>>>>>>
>>>>>>>>>> It can take a while to get used too, but in the long run,
>>>>>>>>>> allot of
>>>>>>>>>> documentation for things in Linux are written in plain-txt:
>>>>>>>>>> How-To's,
>>>>>>>>>> README, INSTALL, all that stuff is always in plain-txt. It
>>>>>>>>>> also makes
>>>>>>>>>> copy & past work much better :-)
>>>>>>>>>>
>>>>>>>>>> best regards,
>>>>>>>>>>
>>>>>>>>>> Greg.
>>>>>>>>>>
>>>
> 
> 


Follow ups

References