← Back to team overview

c4c-dev team mailing list archive

Re: Large Binary Files

 

Hi Eric,

It's not so much the change in material itself that is the issue,
rather, it's the history of change within the branch that will cause us
problems.

I've been looking for solutions too this, but have not found a suitable
one alternative yet, aside from a downloading system that the user
initiates. Not to mention, having restricted contest in an open source
management system is really not appropriate.

Including the restricted content with each system, by some means of
downloading after usage acceptance would be fine, as long as the
location is outside of the VCS system on Launchpad. Mixing and matching
Restricted / Copyrighted material with Open Source License material in
the same branch location should be avoided if at all possible.

This happens allot with application like Wine (Microsoft DLL's),
restricted codecs (MP3, MP4 etc), Microsoft True fonts, but those files
have a small disk space foot print. It's very grey area to say the least.

If it were my choice, I would not post restricted material in the VCS
system.


best regards,

Greg

On 05/17/2015 03:13 PM, Eric Bradshaw wrote:
> Guys,
> 
> Many of the C4C-Specific files really are huge, but none of them have
> much a chance of being updated, especially between major releases.
> Here's the top 7 potential packages - biggest first - (size after
> expansion/install):
> 
> 662.5 MB The Gospel of John, and 1st, 2nd, and 3rd John in MP3. It's
> included because it is one of the best, if not the best book in which to
> start reading the Bible and it's wonderfully read and explained (as this
> church does), in order, verse by verse. This wouldn't even be possible
> to change or add to for years and years.
> 637.3 MB The Entire Old Testament in MP3, World English Bible (WEB)
> version. This will not be added to or changed.
> 303.8 MB Christian Music MP3s. There is a possibility that this may be
> updated - expanded only - if we get permission from additional Christian
> recording artists. But, they'd have to offer because I haven't asked
> anyone for permission to use their music in a good while. It took many,
> many months before I got as music as I did - and frankly, it was
> exhausting. If we get any additional music, we can certainly make sure
> we don't add it in until the next LTS release.
> 191.7 MB The pre-installed and indexed Bibles, Commentaries,
> Devotionals, Dictionaries, General Books and Maps for use with Bible
> Desktop and Xiphos. There is a possibility one or two modules will be
> updated between LTS releases, but we can still choose not to update our
> package.
> 170.7 MB The Entire New Testament in MP3, World English Bible (WEB)
> version. This will not be added to or changed.
> 54.6 MB OGG Videos. No reason we would need to change these videos -
> even add to them, except or until an LTS release.
> 30.5 MB SWF Videos. Same as the OGG Videos above.
> 
> Eric
> 
> On 05/17/2015 12:17 PM, KI7MT wrote:
>> HI Eric,
>>
>> Wow, these are *huge* files, but it's good to have this breakdown, as
>> each is potentially a package that would need creating specifically for
>> C4C or at least, some level of customization.
>>
>> Here's the problem (at least one, but its a big one). When you check out
>> a branch, using BZR, Git etc, you get the full history each time. If a
>> package is updated say two or three time over the course of the release
>> cycle, you would have 2 to 3 time that much disk space usage and Upload
>> / Download time while branching, merging, and pushing back to the branch.
>>
>> For christian-john-123.tar.gz (600+ MB), that would equate to =>2.4GB of
>> storage if the updates were stored in whole at each revision. And that
>> is for just the one package.
>>
>> I did not realize the content of these packages was so large. Clearly we
>> need a better solution for dealing with these large files, as the
>> problem of download / upload time for the branch will grow exponentially
>> using a version control system.
>>
>> best regards,
>>
>> Greg.
>>
>> On 05/17/2015 11:58 AM, Eric Bradshaw wrote:
>>> Guys,
>>>
>>> I know it took me an hour to upload the huge 1404-2.zip (2.1 GB) file to
>>> the shared C4C Folder in Dropbox, so it's probably going to take triple
>>> that for anyone to download via DSL, or satellite. So, I've uploaded
>>> "byte-size" chunks of it too, tar-gz files, split by where it ends up
>>> (but, still in the same order as the bullet lists). Israel; this may
>>> also help explain why there are a ton of binary files. The list below is
>>> also a text file called TAR-GZ-FILES. Oh, and the big one is still up
>>> there too if you've got a fast connection and/or just want everything at
>>> once.
>>>
>>> ADD BIBLES, COMMENTARIES, MAPS, BOOKMARKS & FILE SETTINGS
>>> dot-jsword.tar.gz [15.3MB] expands to dot-jsword, the contents of which
>>> go into ~/.jsword
>>> dot-mozilla.tar.gz [3.4MB] expands to dot-mozilla, the contents of which
>>> go into ~/.mozilla
>>> dot-sword.tar.gz [144.8MB] expands to dot-sword, the contents of which
>>> go into ~/.sword
>>> dot-xiphos.tar.gz [2.7KiB] expands to dot-xiphos, the contents of which
>>> go into ~/.xiphos
>>> etc-gnome.tar.gz [1.7KiB] expands to etc-gnome, the contents of which
>>> (defaults.list) go into/etc/gnome
>>> etc-xdg-lubuntu-applications.tar.gz [1.7KiB] expands to
>>> etc-xdg-lubuntu-applications, the contents of which (defaults.list) go
>>> into/etc/xdg/lubuntu/applications
>>>
>>> ADD C4C SPECIFIC STUFF
>>> dot-config.tar.gz [19.2KiB] expands to dot-config, the contents of which
>>> go into ~/.conf
>>> share.tar.gz [3.7KiB] expands to share, the contents of which go into
>>> ~/.local/share
>>> christian-12-apostles.tar.gz [592KiB] expands to 12-apostles, which goes
>>> into /usr/share/christian
>>> christian-audio-bible.tar.gz [16KiB] expands to audio-bible, which goes
>>> into /usr/share/christian
>>> christian-believe.tar.gz [3.3MB] expands to believe, which goes into
>>> /usr/share/christian
>>> christian-bkgs.tar.gz [8.2MB] expands to bkgs, which goes into
>>> /usr/share/christian
>>> christian-bvmq.tar.gz [528KiB] expands to bvmq, which goes into
>>> /usr/share/christian
>>> christian-christian-music.tar.gz [296.8MB] expands to christian-music,
>>> which goes into /usr/share/christian
>>> christian-christian-videos.tar.gz [16KiB] expands to christian-videos,
>>> which goes into /usr/share/christian
>>> christian-disciple.tar.gz [2MB] expands to disciple, which goes into
>>> /usr/share/christian
>>> christian-fbgs.tar.gz [7.8MB] expands to fbgs, which goes into
>>> /usr/share/christian
>>> christian-icons.tar.gz [176KiB] expands to icons, which goes into
>>> /usr/share/christian
>>> christian-john-123.tar.gz [644.6MB] expands to john-123, which goes into
>>> /usr/share/christian
>>> christian-new-testament.tar.gz [158.2MB] expands to new-testament, which
>>> goes into /usr/share/christian
>>> christian-ogg-video.tar.gz [54.2MB] expands to ogg-video, which goes
>>> into /usr/share/christian
>>> christian-old-testament.tar.gz [591.5MB] expands to old-testament, which
>>> goes into /usr/share/christian
>>> christian-swf-video.tar.gz [24.6MB] expands to swf-video, which goes
>>> into /usr/share/christian
>>> doc.tar.gz [816KiB] expands to doc, the contents of which go into
>>> /usr/share/doc
>>> lubuntu.tar.gz [21.8MB] expands to lubuntu, the contents of which go
>>> into /usr/share/lubuntu
>>> themes.tar.gz [32KiB] expands to themes, the contents of which go into
>>> /lib/plymouth/themes/
>>> applications.tar.gz [16KiB] expands to applications, the contents of
>>> which go into /usr/share/applications
>>>
>>> Eric
>>>
>>> On 05/17/2015 08:23 AM, KI7MT wrote:
>>>> Hi Eric,
>>>>
>>>> Nice work!
>>>>
>>>> I've only browsed through each section briefly, but I think most, if
>>>> not
>>>> all could be done with a single shell script, at least, when building a
>>>> box locally. Doing all this from the command line would make things a
>>>> bit easier also.
>>>>
>>>> Now we need to look at each action and determine if is a system wide
>>>> task or user specific task then determine the best way accomplish the
>>>> task programatically. A good could be, after the box is configured the
>>>> way you like, create a new user and determine what needs doing.
>>>>
>>>>
>>>> best regards
>>>>
>>>> Greg
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 05/16/2015 09:57 PM, Eric Bradshaw wrote:
>>>>> Guys,
>>>>>
>>>>> I've installed the C4C ReSpin of 14.04.2 on a 64-bit machine and made
>>>>> both a High Level Action List called "Bullet-Points" and one with more
>>>>> detail called "Bullet-List-Expanded" (sorry - couldn't help it)
>>>>> attached
>>>>> to this email and uploaded to the C4C shared folder. I've also
>>>>> uploaded
>>>>> a zipped 1404-2 folder with all the files I used and reference in the
>>>>> Bullet-List-Expanded text file.
>>>>>
>>>>> I release now why Greg was confused about why I was un-installing
>>>>> apps.
>>>>> It was because I never could get what I wanted to start with by
>>>>> installing the Lubuntu-base - I always started with Lubuntu Desktop.
>>>>> Plus everything on my home network has a static IP, so it's easier to
>>>>> set the Internet connection from a Desktop ISO to do the initial
>>>>> install.
>>>>>
>>>>> Anyway - I hope I'm helping.
>>>>>
>>>>> Eric
>>>>>
>>>>> On 05/16/2015 01:09 PM, KI7MT wrote:
>>>>>> Hi Eric,
>>>>>>
>>>>>> Just a suggestion; I would recommend using a .txt document rather
>>>>>> than
>>>>>> .odt files for your high level doc. Text formatting is a bit of an
>>>>>> art,
>>>>>> but when done properly, is reansferable to almost anything, likst
>>>>>> text2html for example.
>>>>>>
>>>>>> Most developers prefer .txt docs for Emails as well as procedures,
>>>>>> as it
>>>>>> is much easier to read / edit / and merge.
>>>>>>
>>>>>> It can take a while to get used too, but in the long run, allot of
>>>>>> documentation for things in Linux are written in plain-txt: How-To's,
>>>>>> README, INSTALL, all that stuff is always in plain-txt. It also makes
>>>>>> copy & past work much better :-)
>>>>>>
>>>>>> best regards,
>>>>>>
>>>>>> Greg.
>>>>>>
>>>>>
>>>
> 
> 


Follow ups

References