yellow team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

Thank you Graham.

Brad and I worked on this Friday, and then I worked on it this weekend. 
 It's farther along, and we have a demo that is pretty close to what we 
planned, at least for the buildbot sample Pyflakes demo.

Branches:
lp:~gary/charms/oneiric/buildbot-slave/dynamic-relationship
lp:~gary/charms/oneiric/buildbot-master/dynamic-relationship

Those have a lot of changes.

1) I came up with a crazy way to not have to templatize the master.cfg, 
and Brad convinced me it was a reasonably good kind of crazy. 
master.cfg is executed by buildbot, so we write our own.  It execs the 
original master.cfg, and then we mutate the data structure that the 
original one builds.  Our master.cfg gets our data from a JSON file, so 
we don't have to mutate the Python file itself.  I think it's nice, in a 
scary "buildbot's configuration language is Python!" kind of way. [Note 
that I need to explain why we have the placeholder slave. I think it is 
necessary, but we should challenge that to be sure.]

2) The master sets the password and name for the slave, as we had 
discussed.  This is done in -changed rather than -joined, which is a 
change from a sketch someone else had started.  This seemed like a good 
idea at the time, but the time was an unholy late time on Saturday 
night, so I need to think that through again: maybe it should go into 
-join after all.  That said, the handshake between slaves and masters 
needs to be [slave -> master: builders] [master -> slave: password and 
name] because the buildbot master needs to be ready for the slave before 
the slave starts.

3) I could not get uuencode and juju set to play nicely together.  I 
threw together an "encode" executable that does what we need.  See the 
next item....

4) I added a start at a README.txt which has an example usage.  Look in 
there for how to check out whether you can get my two branches to work 
together.  Here's where you can see how I'm using encode.  If you can 
get uuencode to work more attractively than this encode script, feel 
free to document here how to do it, and delete the encode script. 
Here's the steps that work for me.

juju bootstrap
juju deploy --repository=./charms local:buildbot-master
juju deploy --repository=./charms local:buildbot-slave
juju set buildbot-master extra-packages=git
juju set buildbot-master config-file=`./charms/oneiric/buildbot-master 
/encode charms/oneiric/buildbot-master/examples/master.cfg`
juju set buildbot-slave builders=runtests
juju add-relation buildbot-slave buildbot-master

5) I discovered that buildbot reconfig does not work when I actually 
changed something, at least with Oneiric/buildbot 0.8.3.  It generates a 
traceback and fails to process the changes.  Therefore I switched to 
stop + start.

6) I added a local.py in the master to hold helpers that are specific to 
this charm (see below for notes on Benji's vision of how the helpers.py 
script is supposed to be generic and shared).

7) I used uuid to generate names rather than random.choice.  Feel free 
to argue against this, but it seems like the right thing to do.


That's all I can think of.

There are a lot of things to do to clean up.  I can't even think of all 
of them.  Here are a few from memory.

1) For some reason the first attempt to set a buildbot name doesn't 
take.  Another is made, and it works, but it leaves a bad slave 
registration that you can see on the /buildslaves page of the buildbot 
master web interface (:8010 port).

2) Our master.cfg will only work with BuilderConfig classes.  lpbuildbot 
uses dicts.  We need to reconcile this one way or the other.

3) Benji had a vision that helpers.py would be a generic charms file 
that would be eventually shared with the equivalent to svn externals, as 
is coming in bzr 2.5.  I liked that vision, but he and I did not 
document or popularize it.  We need to reconcile that with what we have now.

4) Imports should be standardized.  Our files should be standardized. 
We should include our license boilerplate.  IOW, we should follow the 
standard LP practices.

5) I agree with Brad that it would be nice if our Python files had 
functions and the "if __name__ = '__main__':" pattern.

6) I'm using json to store the placeholder slave name, and I probably 
should just write it to file directly, because all I want is a string. 
I kinda wonder if json's care buys me anything though.

7) [Maybe] can we change our current pickle use to json?  It is much 
more transparent for debugging and generally seems cleaner.

There's more, I'm sure.

*More importantly*, we need to see how this works with lpbuildbot and 
old master/new slave, and resolve any problems we encounter.

So, looking ahead:

1) We need to get this current work to a point that we can review and 
land it ASAP.
2) Then we need to get this to work with lpbuildbot.
3) Then we can start addressing clean up issues like the ones above.

Thanks, and talk to you all tomorrow!

Gary



On 02/03/12 13:05, Graham Binns wrote:
> Hi folks,
>
> Since Brad and Gary were away at lunch when Francesco and I were
> wrapping up for the day, we thought we'd send you an email detailing
> what we'd done rather than just chucking it over the wall at you.
>
> We've updated both buildbot-master and -slave. You probably don't need
> to worry too much about the slave changes, so here are the things we did
> to the master charm:
>
>   - We've added a configuration template in the hooks directory. We
>     wanted to keep it separate from the one in configuration/, since that
>     one is used when the buildbot master is being set up in the install
>     hook. The template contains placeholders for slaves and builders,
>     which we'll replace on-the-fly.
>   - We've added a buildbot-relation-changed hook which handles the master
>     side of the master/slave handshake. It stores the slave details in a
>     pickle so that it can regenerate the whole config on the fly every
>     time.
>   - We've added a relation-joined hook, which generates a name and
>     password for each new slave that joins.
>   - We've added some (self-explanatory) helpers to helpers.py.
>
> The next step for the master is updating the c['builders'] section
> ofmaster.cfg. This is a bit more complicated, since the builders to
> which
> each slave belongs are stored with the slaves in the pickle.
> However,it's mostly just a case of iterating over the slaves and
> putting them in
> a {builder: [slave_names]} kind of structure (We think; but it's 6/7pm
> for us).
>
> That's about it. Hope it helps, and that the code doesn't conflict too
> much with yours. We've tested it out as follows:
>
>   $ juju deploy repository=./ local:buildbot-master
>   $ juju deploy repository=./ local:buildbot-slave
>   $ juju set buildbot-slave builders=BUILDER1,BUILDER2
>   $ juju add-relation buildbot-slave buildbot-master
>
> Watch the master log file and weep with joy.
>
> Cheers,
>
> Francesco&  Graham