yellow team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

OK, I'll describe the story for the lxc-start-ephemeral changes we need.

Here's how to demonstrate the problem.  Start up two ephemeral instances 
based on the same container.  Make a change in the home directory of 
one.  You'll see that this is reflected in the other ephemeral, and in 
the host.  That's because the home directory is mounted in the 
container, but without a tmpfs overlay.

In fact, right now we mount the sensitive parts of our home directory 
(the directory that buildbot is testing) twice.  First, we pass -b $PWD 
in to launchpad-lxc-test (as created by setuplxc).  This directly mounts 
the build directory, without any temporary overlay, in the ephemeral 
container.  Then, we have a copy of the base container's fstab (look in 
/var/lib/lxc/CONTAINER_NAME/fstab).  This will mount the entire home 
directory in the ephemeral container.

Working from some of Benji's ideas, here's what I propose we do.  I 
haven't tested this, so a good first step is to see if this might 
actually work. :-P

- Change the -b argument to lxc-start-ephemeral to be explicitly about 
binding a directory *ephemerally*.
- If -b is used, we need to add code that does something like the following.
   * We make a directory at the top of the ephemeral $LXC_DIR, next to 
rootfs, fstab, and so on, named something like "ephemeral-binding". 
Note that $LXC_DIR is already an overlayfs with a tmpfs on top, so this 
is already ephemeral.
   * We replace the current binding mount ("sudo mount --bind $LXC_BIND 
$LXC_DIR/rootfs$LXC_BIND") with something like this: "sudo mount -t 
overlayfs -oupperdir=$LXC_DIR/ephemeral-binding,lowerdir=$LXC_BIND none 
$LXC_DIR/rootfs$LXC_BIND".  That's putting an overlay on an overlay, 
using the overlay, so it may be insane.  Hopefully you can understand 
the goal, if that doesn't work, though: you could make another tmpfs 
directory somewhere else instead of the "$LXC_DIR/ephemeral-binding" 
from the previous step (see the example "OVERLAY_DIR=`mktemp -d 
/tmp/lxc-lp-XXXXXXX`; sudo mount -t tmpfs none $OVERLAY_DIR" in the 
script now).
   * If the same directory is mounted in the ephemeral container's 
fstab (it will be for us, see below), remove that line from the fstab.

Then, we would call lxc-start-ephemeral with -b /var/lib/buildbot.

That's the right direction, I think.

(Benji also mentioned worrying about the tmp directory, but that is 
already part of the original overlayfs in the container, so I think it 
is fine already.)

So, feel free to mess with that, or not!

Talk to you all tomorow,

Gary

On 03/08/12 18:06, Gary Poster wrote:
> Hi Graham and Francesco. I told you that Benji would write instructions
> on how to start the Launchpad buildbot infrastructure. The good news is
> that the instructions are now (again) simply what the buildbot master's
> README describes. The bad news, from the perspective of you all working
> on it, is good news: Benji and I believe we see another big thing to fix
> in lxc-start-ephemeral.
>
> The problem is that lxc-start-ephemeral shares the user's home directory
> with all other ephemerals, and changes there are *not* ephemeral. That's
> bad for our tests.
>
> If I have a chance this evening to describe the problem more in depth,
> and our thoughts about it, then you could tackle it. If not, I suggest
> slack time work if you are wondering what to do, or bug 663149.
>
> I could also describe my thoughts on 609986. I'll see what I get to
> later tonight.
>
> Gary