← Back to team overview

c4c-dev team mailing list archive

Re: Large Binary Files

 

Guys,

Again, no expertise - but, my thought is we may want to use the QT Open Source installer simply because Lubuntu is transitioning to LXQt (supposedly) by 15.10 - so maybe we'd be prepared? Apologies if one would have nothing to do with the other.

Eric

On 05/18/2015 09:14 AM, KI7MT wrote:
Hi Israel, All,

Yes, zsync, rsync or even curl could do that job easily. A simple shell
script could do what is needed here.

If you have ever used MSYS/MinGW, this is exactly how their installer
works for pulling packages to install form the SF server. The same for
the QT Open Source installer, and it pull a is HUGE amount of data at
install.


best regards,

Greg.

On 05/18/2015 07:43 AM, Israel wrote:
Hi everyone...
I have thought about this issue.
2 things
zsync is the appropriate tool for this!

we could host all our content on a c4c server.
then the package could simply be a dummy package that requires zsync,
and then postinst
runs something like:
cd /usr/share/c4c/Music
zsync http:://c4cserver.org/ot.tar.xz.zsync

then unpacks the tarball and syncs it to the user's Music folder.  As
long as the user can read the files (not modify them... file
permissions)  we will be good to go.

Another good idea would be to include in the MetaData of the music a
copyright notice

Also, including some sort of dialog (zenity or xterm + dialog would be
great for this) to say:
"$LICENSE_AGREEMENT
Do you accept these terms of use?"
where LICENSE_AGREEMENT=""#whatever you want to put as the agreement

zsync will easily mitigate the HUGE amount of stuff that needs to be
downloaded, and also automatically checks md5sum and can resume if
internet connection is broken



On 05/17/2015 04:50 PM, KI7MT wrote:
Hi Eric,

It's not so much the change in material itself that is the issue,
rather, it's the history of change within the branch that will cause us
problems.

I've been looking for solutions too this, but have not found a suitable
one alternative yet, aside from a downloading system that the user
initiates. Not to mention, having restricted contest in an open source
management system is really not appropriate.

Including the restricted content with each system, by some means of
downloading after usage acceptance would be fine, as long as the
location is outside of the VCS system on Launchpad. Mixing and matching
Restricted / Copyrighted material with Open Source License material in
the same branch location should be avoided if at all possible.

This happens allot with application like Wine (Microsoft DLL's),
restricted codecs (MP3, MP4 etc), Microsoft True fonts, but those files
have a small disk space foot print. It's very grey area to say the least.

If it were my choice, I would not post restricted material in the VCS
system.


best regards,

Greg

On 05/17/2015 03:13 PM, Eric Bradshaw wrote:
Guys,

Many of the C4C-Specific files really are huge, but none of them have
much a chance of being updated, especially between major releases.
Here's the top 7 potential packages - biggest first - (size after
expansion/install):

662.5 MB The Gospel of John, and 1st, 2nd, and 3rd John in MP3. It's
included because it is one of the best, if not the best book in which to
start reading the Bible and it's wonderfully read and explained (as this
church does), in order, verse by verse. This wouldn't even be possible
to change or add to for years and years.
637.3 MB The Entire Old Testament in MP3, World English Bible (WEB)
version. This will not be added to or changed.
303.8 MB Christian Music MP3s. There is a possibility that this may be
updated - expanded only - if we get permission from additional Christian
recording artists. But, they'd have to offer because I haven't asked
anyone for permission to use their music in a good while. It took many,
many months before I got as music as I did - and frankly, it was
exhausting. If we get any additional music, we can certainly make sure
we don't add it in until the next LTS release.
191.7 MB The pre-installed and indexed Bibles, Commentaries,
Devotionals, Dictionaries, General Books and Maps for use with Bible
Desktop and Xiphos. There is a possibility one or two modules will be
updated between LTS releases, but we can still choose not to update our
package.
170.7 MB The Entire New Testament in MP3, World English Bible (WEB)
version. This will not be added to or changed.
54.6 MB OGG Videos. No reason we would need to change these videos -
even add to them, except or until an LTS release.
30.5 MB SWF Videos. Same as the OGG Videos above.

Eric

On 05/17/2015 12:17 PM, KI7MT wrote:
HI Eric,

Wow, these are *huge* files, but it's good to have this breakdown, as
each is potentially a package that would need creating specifically for
C4C or at least, some level of customization.

Here's the problem (at least one, but its a big one). When you check out
a branch, using BZR, Git etc, you get the full history each time. If a
package is updated say two or three time over the course of the release
cycle, you would have 2 to 3 time that much disk space usage and Upload
/ Download time while branching, merging, and pushing back to the branch.

For christian-john-123.tar.gz (600+ MB), that would equate to =>2.4GB of
storage if the updates were stored in whole at each revision. And that
is for just the one package.

I did not realize the content of these packages was so large. Clearly we
need a better solution for dealing with these large files, as the
problem of download / upload time for the branch will grow exponentially
using a version control system.

best regards,

Greg.

On 05/17/2015 11:58 AM, Eric Bradshaw wrote:
Guys,

I know it took me an hour to upload the huge 1404-2.zip (2.1 GB) file to
the shared C4C Folder in Dropbox, so it's probably going to take triple
that for anyone to download via DSL, or satellite. So, I've uploaded
"byte-size" chunks of it too, tar-gz files, split by where it ends up
(but, still in the same order as the bullet lists). Israel; this may
also help explain why there are a ton of binary files. The list below is
also a text file called TAR-GZ-FILES. Oh, and the big one is still up
there too if you've got a fast connection and/or just want everything at
once.

ADD BIBLES, COMMENTARIES, MAPS, BOOKMARKS & FILE SETTINGS
dot-jsword.tar.gz [15.3MB] expands to dot-jsword, the contents of which
go into ~/.jsword
dot-mozilla.tar.gz [3.4MB] expands to dot-mozilla, the contents of which
go into ~/.mozilla
dot-sword.tar.gz [144.8MB] expands to dot-sword, the contents of which
go into ~/.sword
dot-xiphos.tar.gz [2.7KiB] expands to dot-xiphos, the contents of which
go into ~/.xiphos
etc-gnome.tar.gz [1.7KiB] expands to etc-gnome, the contents of which
(defaults.list) go into/etc/gnome
etc-xdg-lubuntu-applications.tar.gz [1.7KiB] expands to
etc-xdg-lubuntu-applications, the contents of which (defaults.list) go
into/etc/xdg/lubuntu/applications

ADD C4C SPECIFIC STUFF
dot-config.tar.gz [19.2KiB] expands to dot-config, the contents of which
go into ~/.conf
share.tar.gz [3.7KiB] expands to share, the contents of which go into
~/.local/share
christian-12-apostles.tar.gz [592KiB] expands to 12-apostles, which goes
into /usr/share/christian
christian-audio-bible.tar.gz [16KiB] expands to audio-bible, which goes
into /usr/share/christian
christian-believe.tar.gz [3.3MB] expands to believe, which goes into
/usr/share/christian
christian-bkgs.tar.gz [8.2MB] expands to bkgs, which goes into
/usr/share/christian
christian-bvmq.tar.gz [528KiB] expands to bvmq, which goes into
/usr/share/christian
christian-christian-music.tar.gz [296.8MB] expands to christian-music,
which goes into /usr/share/christian
christian-christian-videos.tar.gz [16KiB] expands to christian-videos,
which goes into /usr/share/christian
christian-disciple.tar.gz [2MB] expands to disciple, which goes into
/usr/share/christian
christian-fbgs.tar.gz [7.8MB] expands to fbgs, which goes into
/usr/share/christian
christian-icons.tar.gz [176KiB] expands to icons, which goes into
/usr/share/christian
christian-john-123.tar.gz [644.6MB] expands to john-123, which goes into
/usr/share/christian
christian-new-testament.tar.gz [158.2MB] expands to new-testament, which
goes into /usr/share/christian
christian-ogg-video.tar.gz [54.2MB] expands to ogg-video, which goes
into /usr/share/christian
christian-old-testament.tar.gz [591.5MB] expands to old-testament, which
goes into /usr/share/christian
christian-swf-video.tar.gz [24.6MB] expands to swf-video, which goes
into /usr/share/christian
doc.tar.gz [816KiB] expands to doc, the contents of which go into
/usr/share/doc
lubuntu.tar.gz [21.8MB] expands to lubuntu, the contents of which go
into /usr/share/lubuntu
themes.tar.gz [32KiB] expands to themes, the contents of which go into
/lib/plymouth/themes/
applications.tar.gz [16KiB] expands to applications, the contents of
which go into /usr/share/applications

Eric

On 05/17/2015 08:23 AM, KI7MT wrote:
Hi Eric,

Nice work!

I've only browsed through each section briefly, but I think most, if
not
all could be done with a single shell script, at least, when building a
box locally. Doing all this from the command line would make things a
bit easier also.

Now we need to look at each action and determine if is a system wide
task or user specific task then determine the best way accomplish the
task programatically. A good could be, after the box is configured the
way you like, create a new user and determine what needs doing.


best regards

Greg












On 05/16/2015 09:57 PM, Eric Bradshaw wrote:
Guys,

I've installed the C4C ReSpin of 14.04.2 on a 64-bit machine and made
both a High Level Action List called "Bullet-Points" and one with more
detail called "Bullet-List-Expanded" (sorry - couldn't help it)
attached
to this email and uploaded to the C4C shared folder. I've also
uploaded
a zipped 1404-2 folder with all the files I used and reference in the
Bullet-List-Expanded text file.

I release now why Greg was confused about why I was un-installing
apps.
It was because I never could get what I wanted to start with by
installing the Lubuntu-base - I always started with Lubuntu Desktop.
Plus everything on my home network has a static IP, so it's easier to
set the Internet connection from a Desktop ISO to do the initial
install.

Anyway - I hope I'm helping.

Eric

On 05/16/2015 01:09 PM, KI7MT wrote:
Hi Eric,

Just a suggestion; I would recommend using a .txt document rather
than
.odt files for your high level doc. Text formatting is a bit of an
art,
but when done properly, is reansferable to almost anything, likst
text2html for example.

Most developers prefer .txt docs for Emails as well as procedures,
as it
is much easier to read / edit / and merge.

It can take a while to get used too, but in the long run, allot of
documentation for things in Linux are written in plain-txt: How-To's,
README, INSTALL, all that stuff is always in plain-txt. It also makes
copy & past work much better :-)

best regards,

Greg.





Follow ups

References