← Back to team overview

ubuntu-ngo team mailing list archive

Portable repositories

 

Hi all,

Apologies for the length of this. Here's the summary.

* I wanted to try out having an entire repo on one bootable USB stick.
* It works.
* Comments?

The idea of portable repositories (that is a removable, portable media
device containing a partial copy of the Ubuntu repo(s) ) has been in
my head for some time and recently I had some time to play with it.

Note: I talk about USB sticks mostly, but that's just because that's
what I used. It may well be more appropriate (and cheaper) to use USB
hard disks (the spinning kind). Please bear that in mind.

Here's the scenarios I've been thinking about.

1 System admin has a network of computer which have slow or no
internet connection, but would like new software packages and updates
to be installable.
2 Advocate would like to help a user get Ubuntu installed, but the
user has no or slow internet connection. Advocate would like to
install a standard desktop, but add on extra packages to improve the
experience.
3 Expert Ubuntu user would like to quickly install Ubuntu on a number
of unconnected machines but has no or slow internet connection.

My ideal solution consists of a portable media device (USB attached
stick or hard disk in an enclosure) with all that is required on the
stick. No software needs to be installed, and minimal configuration
necessary.

For Scenario 1 - System Admin.

System Admin boots a pre-configured USB stick in a spare machine which
immediately becomes a local repository. Client machines can either be
pointed directly to the hostname or IP address of the box, or (if
available) DNS can be spoofed for *.archive.ubuntu.com, pointing those
records at the server running the repo. Either way, all client
machines can access the repository using standard tools such as
synaptic and apt-get.

Enhancement 1 to Scenario 1.

System admin can have two USB sticks used alternately. Whilst one is
in the server providing service, the second one is in the Internet
Cafe being updated. It could of course be posted (being small and
light) to someone else with internet access who posts it back when
done.

Enhancement 2 to Scenario 1.

System admin may want a more permanent solution than booting off USB
sticks. System Admin could install a permanent local server, and
simply update the repository on the server from the USB stick. Again
the USB stick can be sent away for updates, and when it comes back,
the update process can be initiated to ensure the local server is up
to date. The USB stick can then be sent away again, or passed on to
someone else to update their network.

Enhancement 3 to Scenario 1.

The USB stick server could also be a PXE boot and (optionally) a DHCP
server. This would enable the System Admin to boot machines over the
LAN (PXE) into the mini (netinst) installation environment where they
can then install to local client machines, and obtain packages
directly from the server. This ensures that any client machines are up
to date as soon as they are finished installing.

For Scenario 2 - Advocate.

Advocate boots off the USB stick into a live environment to
demonstrate Ubuntu. The USB stick would have a live image of the
current release, along with a recent copy of the repository. Once the
advocate has demonstrated, they can install from the stick to the
local machine, and then install additional applications from the
repository onto the local hard disk.

Enhancement 1 to Scenario 2.

If there is sufficient space the entire repository could be copied
from the USB stick to the local hard disk on the users PC. This would
enable them to be 'self sufficient' in installing applications when
the advocate is not around. Next time the Advocate visits, they can
bring an updated USB key which they can use to install new apps, or
update the local repository on the users PC.

For Scenario 3 - Expert User

Similar to Enhancement 3 for Scenario 1, however booting locally
rather than via PXE. The Expert User boots to the mini (netinst) iso
directly from the USB stick and performs an installation using the
locally attached repository on the USB stick.

Technologies required:-

I have not yet completed all necessary steps to fulfil all three scenarios.

My current setup was simply created by plugging the USB stick in and
doing a server install from an 8.04 server CD. The only gotcha I had
was to make sure the grub boot loader was installed to the correct
location (i.e. not my local hard disk), and post install I cleared out
the grub entries for the installs of Ubuntu that are on my laptop -
which were detected during the installation (could have avoided this
by unplugging the hard disk in my laptop).

Created /srv/repostick/www/ubuntu and /srv/repostick/logs for the
webserver to use.

Installed lighttpd to serve up the repo over http to my lan and
configured it (very basic) to serve up /srv/repostick/www as the web
root, and use /srv/repostick/logs for logging.

Once I booted into Ubuntu I just installed "debmirror" and "patch"
(required by debmirror), and kicked off this command:-

debmirror --nosource -m --passive \
    --exclude=\-dbg_ \
    --host=gb.archive.ubuntu.com \
    --root=ubuntu/ \
    --method=http \
     --progress \
     --dist=karmic,karmic-updates,karmic-security,karmic-backports,karmic-proposed
\
     --section=main,restricted,universe,multiverse \
     --arch=i386 \
     --ignore-release-gpg
      /srv/repostick/www/ubuntu

This grabs the binary only packages, excluding the debug symbol ones,
from my local mirror in the UK via http. It grabs all the sections of
karmic but only i386 architecture. I left it overnight and it had
finished by the morning.

Side note: Actually it ran out of space on the USB stick, which was
why I then deleted all packages with -dbg_ in the name and re-ran the
debmirror it with the 'exclude'. It finished off the remaining
packages.

Notes: I know karmic isn't necessarily the right one to do for
everyone, hardy (as an LTS) would be more appropriate. I also know
that -proposed -updates and -security will effectively be empty right
now. I used karmic because I have a laptop running i386 karmic and
that gets lots of updates so there's plenty of opportunity to re-run
the scripts and make sure it all still works :)

$ du -hs /repo/srv/repostick/www
24G

2.9G free on the stick.

Testing:-

Boot off the USB stick. Point other machines at
http://repostick/ubuntu and observe a lovely repository.
Configure a machine to use the sources lines:-

deb http://repostick/ubuntu karmic main restricted universe multiverse
deb http://repostick/ubuntu karmic-updates main restricted universe multiverse
deb http://repostick/ubuntu karmic-security main restricted universe multiverse
deb http://repostick/ubuntu karmic-backports main restricted universe multiverse
deb http://repostick/ubuntu karmic-proposed main restricted universe multiverse

Do a usual apt-get update/upgrade and all seems in order. Only issue
is that apt complains that the packages are untrusted.

Next test was to plug the USB stick into a machine running karmic i386
desktop. Sadly it mounted automatically as
/media/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (the UUID of the
partition), so I remounted it:-

sudo mkdir /repo
sudo mount /dev/sdb1 /repo

Then edited the sources.list to use:-

deb file:/repo/srv/repostick/www/ubuntu karmic main restricted
universe multiverse
deb file:/repo/srv/repostick/www/ubuntu karmic-updates main restricted
universe multiverse
deb file:/repo/srv/repostick/www/ubuntu karmic-security main
restricted universe multiverse
deb file:/repo/srv/repostick/www/ubuntu karmic-backports main
restricted universe multiverse
deb file:/repo/srv/repostick/www/ubuntu karmic-proposed main
restricted universe multiverse

Again apt-get update/upgrade worked fine, along with the complaint
that packages are unsigned.

Once it was plugged in I could run debmirror to update it too :)

debmirror --nosource -m --passive \
    --exclude=\-dbg_ \
    --host=gb.archive.ubuntu.com \
    --root=ubuntu/ \
    --method=http \
     --progress \
     --dist=karmic,karmic-updates,karmic-security,karmic-backports,karmic-proposed
\
     --section=main,restricted,universe,multiverse \
     --arch=i386 \
     --ignore-release-gpg
      /repo/srv/repostick/www/ubuntu

Note: only the last line has changed.

Issues:-

* You need a fairly large USB stick to hold a copy of even part of the
repository. For example a 32GB USB drive is not big enough to hold the
installation of Ubuntu Server and Karmic for one architecture (i386)
including main, restricted, universe, multiverse, binary only (no
source code packages). I have managed to get it to fit by removing the
*-dbg_* packages (which used to be in a separate repo, but now seem to
be in the default one?). Clearly a USB hard disk offers much more
capacity for a lower cost, but of course it less robust and larger.

* Initial filling of the device takes an age if you want to pull in a
whole release or architecture. Even on my 20Gb/s home broadband it
took a good 12 hours to get the full karmic repo for one arch
downloaded.

* It's relatively complex to configure a USB stick to multiboot both
the Live environment, a server install and the mini (netinst) iso. So
a carefully crafted recipe is needed.

Further thoughts:-

My overnight download of karmic was over my broadband connection,
direct from my regions local mirror to the USB stick attached to my
laptop. If I kept an up to date mirror on a server, I could speed the
processes up by syncing from that mirror rather than from the Ubuntu
mirror. This would enable me to churn out a bunch of these keys/drives
in the period it took me to do one.

USB sticks potentially suffer from a lower lifetime compared to hard
disks but in some environments this may not hold completely true. For
a stick that gets a one-off update from the repo, then sits plugged
into a server only really ever doing read operations, is this likely
to last less time than a hard disk I wonder.

If someone wanted a full copy of the entire Ubuntu repository (all
currently supported releases and both architectures, and all the
source code) they'd need quite a bit of space. Somewhere in the region
of 250-300GB.

Would it also make sense to include all currently supported ISO images
for the chosen repositories downloaded, so that people could burn ISOs
if they had machines that would not boot off USB. That kick-starts
them, then they can use the USB stick to do further application
installs.

Is there a business in this? Could people in regions where there is
abundant high-speed broadband produce these USB keys/sticks and send
them for a fee to those for whom broadband isn't an option?

Any other comments/questions/suggestions? Is this worth doing?

Cheers,
Al.



Follow ups