openstack team mailing list archive

[Nati Ueno <nati.ueno@xxxxxxxxx>]

Monty++!

Just FYI

GridDynamics are working well for Packaging RPMs.
http://wiki.openstack.org/NovaInstall/RHEL6Notes-Diablo-3

Its also works on Scientific Linux6.1 (Personally, I prefer free OS :D )

Here is there repos.
http://yum.griddynamics.net/yum/diablo-3/openstack/

They maintains it on the github.
https://github.com/griddynamics/openstack-rhel

I hope their great work would become official repos.

Nati (NTT)

2011/8/24 Monty Taylor <mordred@xxxxxxxxxxxx>:
> Hi all!
>
> First of all, I spoke about something yesterday in terms of being a
> done-deal, when it should have been prefaced with "I'm looking at". My
> bad, and I certainly apologize for freaking anyone out.
>
> That being said - there are several issues with the current packaging
> setup that I've been tasked with tackling. This email will outline the
> issues, and then I'll discuss what I've been thinking in terms of solutions.
>
> *****
> This is a proposal. We want to get feedback on this before the summit
> but will not be changing the way we do packaging before the summit.
> *****
>
> For reference, the current setup is:
>  Jenkins creates source packages from a team
> (~openstack-ubuntu-packagers) packaging branch for every trunk commit
> for Ubuntu and uploads them to a PPA on Launchpad
>
> Let me start be enumerating the issues with the current setup that we're
> trying to solve. (These are not in order of importance)
>
> - Lack of Debian support. One of the largest current deployments of
> OpenStack is Rackspace, and they deploy on Debian. (disclaimer, they
> also pay my salary) This means they are internally maintaining their own
> set of packages, and not interacting with ours at all. That sort of sucks.
>
> - Lack of collaboration. Related to the above, but slightly different.
> Since the Rackspace internal deployment team have to maintain their own
> packages, it means they are not using the published packages and we are
> not benefiting from their real-world deployment experience.
> Additionally, the Cloudbuilders consulting team doesn't use our
> published packages, and similarly we aren't really benefiting from them
> directly working with us on packaging.
>
> - PPAs are Async. This makes integration testing harder. We're about to
> roll out deployment/integration testing, which involves installing the
> latest set of packages in environments and then testing that it works.
> In that workflow, the queue/async nature of using LP as our package
> builder becomes more problematic. I cannot trigger a direct test of a
> set of packages as part of the testing of a commit if part of the
> process is "upload into the LP queue and then wait for a while" -
> especially since LP still has not added any sort of callback mechanism
> to report completion. This means Jenkins is locally building packages
> anyway so that it can pass them to deployment environments to test them.
> If it's building them locally, and that's what's being tested, why
> wouldn't that be what's published?
>
> - Divergent dev process from the rest of OpenStack. The PPB just
> recently voted that all OpenStack projects must use the same set of
> tooling. Now granted, packaging is not a core project, so we could get
> legalistic and claim that we're not required to comply - but that seems
> a little disingenuous - especially since I was a strong proponent of the
> one-set-of-tooling model. Additionally, since we have a strong code
> review/testing system that's integrated in to our build and deployment
> system, it seems very odd not to take advantage of it. (having new
> packages created triggered by both approval of packaging branch changes
> AND new upstream tarballs seems kindof ideal, no?)
>
> - Not integrated with CI. PPA build errors are not reported inside of
> our CI system where all of the rest of our errors are reported.
> Additionally, those build errors are not reproducible by devs locally,
> as they cannot spin up a launchpad PPA builder.
>
> - Lack of RPM support. Turns out RPM distros are still kind of popular,
> and at least one of our major contributors (hi, NTT) rolls out on
> RHEL/CentOS, and one of our partners who is going to be contributing
> testing hardware (MSFT/Novell) is associated with a distro (SuSE) that
> is RPM based. We're a few steps away from this, but it warrants mentioning.
>
> - Murky backported deps. Some of the backported deps in the PPAs came
> from a dir on my laptop. Some are from Theirry's laptop. Some are from
> Soren's laptop. There is neither documentation on how they got there,
> how we decided to put them there, how we built them or how to continue
> to collaborate on them.
>
> - Direct injection to Ubuntu not via Debian. The publically stated
> best-practice for things entering Ubuntu is to be added to/uploaded to
> Debian and then synced, unless there is some compelling reason not to do
> this. In this case, we had Ubuntu devs and no Debian devs in the core,
> so we keep improving packages and adding packages to Ubuntu but are not
> doing the same to Debian.
>
> - PPA Fragmentation. The PPB voted that we are one project, yet we have
> a separate PPA for each project.
>
> Ok. I lied - those  might have been in a general order of importance.
> Turns out though, that in terms of addressing them, they fall in to two
> different groupings:
>
> 1) How we manage and work on the packaging code itself
> 2) How we build and publish packages
>
> The two are related, but not linked. Of the two, the most absolutely
> important is 2, so I'll address it first. I'm also going to be a bit
> more blunt about it than the other one.
>
> == How we build and publish packages ==
>
> This deals with:
>  - Lack of Debian support.
>  - PPAs are Async.
>  - Not integrated with CI.
>  - Lack of RPM support.
>  - PPA Fragmentation.
>
> I'm going to be rolling out http://packages.openstack.org, which will at
> first contain packages for lucid, maverick, natty, oneiric, squeeze and
> wheezy. There will be at the very least a trunk repo and a repo
> per-release (so a diablo one) Will probably use the debian trick of
> pointing trunk at diablo at first, and then creating an essex repo and
> pointing essex at that - but I'm up for suggestions there. We have a
> Jenkins, we have plenty of builder slaves and we have pbuilder for
> chroot clean builds - there is no compelling reason for us to wait on
> the Launchpad build queue. AND - I can hand a puppet module to a dev
> that they can run on a cloud server and they can have access to an env
> which is EXACTLY like the one that build errors that they experience
> occured in.
>
> So big wins here are:
>  - single place to get OpenStack as a cohesive whole
>  - addition of pure Debian packages
>  - possibility of adding YUM repos to packages.openstack.org
>  - synchronous publishing for better testing
>  - end to end visibility of what build actions were taken
>
> Note that there is nothing stopping PPA packages from existing - we
> could even have jenkins continue upload things to PPAs as well - it's
> just not really a primary target.
>
> == How we manage and work on the packaging code itself ==
>
> Now to the one that's a little more up in the air because it's the one
> that's going to cause people to throw their hands up in the air and
> scream and gnash their teeth. Sigh. Sorry.
>
> To address:
>  - Lack of collaboration.
>  - Divergent dev process from the rest of OpenStack.
>  - Murky backported deps.
>  - Direct injection to Ubuntu not via Debian.
>
> I'm suggesting:
>  - we move all packaging to git
>  - we manage it consistently via git-buildpackage
>  - we code review packaging branches in gerrit
>  - we publish packages only through reviewed branches
>
> Because of the last, point, I'd like to point out that this applies to
> backported packages we need in our repo as well. Building any package
> from the set of important OpenStack pacakges should go through the same
> process. Managing the packaging for our packages and for backports will
> be slightly different.
>
> What exactly does that mean?
>
> For our packages:
>
> As Thomas Goirand suggested, we will add a packaging branch to the repo
> of each of our projects and have git-buildpackage do the right thing. We
> can have gerrit set different ACLs for the packaging branch so that a
> different set of folks (the packaging team) can handle the reviewing of
> that branch. Combining this with a pristine-tar and having Jenkins push
> some tags back to the repo on package release, it means that we can
> trace any release of software back to our code repo, and that only the
> repo is needed to re-create any particular release.
>
> The jenkins job for this is triggered off of both new commits to trunk
> and to new commits to packaging.
>
> For backported packages:
>
> Having branches on top of upstream source is almost useless here, as in
> most cases we're doing one-off backports. So here we simply make a new
> repo via git-import-dsc, upload it and register it with jenkins. In the
> common case, there won't be much more work to do - but in the ongoing
> case, it means that anyone in our dev community knows where to go to get
> at packaging of specific versions of things carried in our repo.
>
> The jenkins jobs for these are triggered off of new commits to packaging.
>
> For both:
>
> When Jenkins is triggered, it will pull the branches and build packages
> for all available targets in base chroots using git-pbuilder/cowbuilder.
> Each build will have its full log in Jenkins just as any other build.
> >From a repo perspective, for each target, Jenkins will attempt to build
> using packaging in the debian branch unless there is a branch named
> debian/${target} - so if the packaging needs to be overridden for, say,
> lucid (because it's older than dirt) It's easy enough to just create a
> debian/lucid branch  and leave debian to handle the other branches.
>
> These packages are then all published to packages.openstack.org and jobs
> testing deployment are kicked. At a point in the future (if we start
> doing pre-merge deployment testing) we'll need to make a pending archive
> so that we can re-publish same versions due to failure -but we're not
> there yet.
>
> As a bonus, git-buildpackage does some nice tagging of things, which we
> can have Jenkins push back, so we can keep our repos up to date with
> meta information. Since these are on ancillary branches, that pushing
> doesn't have to interrupt normal dev work. Incidentally, since that meta
> info is in the repo, we cease to need the tarball_versions and
> ppa_versions files that we currently manage on the filesystem in jenkins.
>
> I know there are other ways to accomplish some of these things, but this
> is the one I'm proposing. The fact that it fits in to our development
> methodology is appealing, as is the possibility of getting pvo and jesse
> and thomas looped in to working directly on this set of packages.
>
> Monty
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Nachi Ueno
email:nati.ueno@xxxxxxxxx
twitter:http://twitter.com/nati