launchpad-dev team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

On Wednesday 02 December 2009 03:50:50 Michael Hudson wrote:
> >  * Should SourcePackageRecipeData have an owner or created_by?
> 
> I don't think so.  SourcePackageRecipeData is really just a shared
> implementation detail of SourcePackageBuild and SourcePackageRecipe, and
> both of those should record all the interesting stuff.

Ok, so the same recipe data can be re-used in multiple recipes.  Makes sense.

So will some code attempt to factor the recipe data as much as possible into 
as few rows as possible when the recipes are created?

> >  * Why is SourcePackageRecipe a separate table to
> > SourcePackageRecipeData? Since one references the other, the duplicated
> > columns are not needed are they?  In fact the only additional column is
> > the recipe text. Maybe I don't understand what you're trying to do with
> > this?
> 
> Ah right, I meant to remove those duplicate columns from SPRecipeData.
> Essentially the reason they're separate is because SPRecipes have access
> control and a location in the UI, but SPRecipeDatas don't as many of
> them will be read-only versions referenced from SPBuild rows.

OK.

> >  * SourcePackageRecipeDataBranch isn't really needed yet, we're not
> > creating branches for daily builds.  It's not a problem adding it though
> > I guess.
> 
> It's meant to be there to record the links between the recipe and the
> branches it references.  I guess I don't understand what you mean.

My understanding was that we're not going to be creating branches at all, for 
daily builds.  The package file tree will be built from the recipe and then 
uploaded to Soyuz.

This table will be needed when we start building branches from recipes though!

> >   * I don't think you need "archive", this is not pertinent until the
> > package is actually uploaded.  At least, I think we can share these
> > across archives. Can anyone think of a reason why we would not do that?
> 
> Well, somehow we need to record the archive(s?) to which the resulting
> source pacakge will be built.  Maybe not in this table though, indeed.
> More on this below.

OK

> >   * I see you have a column called "manifest" which references
> > SourcePackageRecipeData.  Now I'm really confused about what that is!  I
> > thought recipes and manifests were separate things?  If it needs to be
> > separate why is it not SourcePackageRecipeManifest?
> 
> I hope this makes sense by now?  If we just stored recipe/manifests as
> text, both SPRecipe and SPBuild would just have a text column, but we're
> planning on being a bit more sophisticated than that.

It sorta does, but I don't get the name "manifest".  Also, why is it 
referencing there as opposed to SourcePackageRecipe?

Sorry if I am being thick!

> >  * BuildSourcePackageFromRecipeJob - this might as well contain
> > everything in SourcePackageBuild unless you have a good reason for
> > separating them?
> 
> Why would it make sense to duplicate the columns across the tables?

I meant that we should move them, not duplicate them.

> I thought we went over why BuildSourcePackageFromRecipeJob and SPBuild
> are separate last week.

Maybe.

> > I'm not sure, and replied to the other emails about this separately.
> 
> At some point I guess we need to just decide.

Or, we make jml decide :)

> > Having said that, I would still vastly prefer a separate table linking
> 
> I object slightly to the 'still' -- it feels like I've been trying to
> get a straight answer on this point for a while!  But thanks for
> providing one (even if I still have the urge to avoid this bit of
> complexity for now).

I've never advocated anything other than to put it on a separate table, so I'm 
not sure why you're objecting.

> 
> > SourcePackageBuild and Archives.
> 
> It should be easy enough to do this.  I guess the upload_log would live
> on the SPBuildArchive link?

It should be on the SourcePackageBuildUpload as you have done in the patch.  
Which is nice :)

BTW should we s/SourcePackageBuild/SourcePackageRecipeBuild/ everywhere?  In 
future we'll be building from a branch as well.
 
> > This allows the possibility of re-uploading
> > the same source recipe build if we want.  I think that will be important
> > for 2 reasons:
> >  1. re-creating upload issues
> >  2. re-creating genuine package bugs in a different PPA environment
> 
> I guess I'm not sure about some of the mechanics of this.  In the
> standard way things will work, the buildd-manager will grab the built
> package and upload them for building into any archives that are
> specified at that time.  How will the upload happen for one of these
> after-the-fact requests?

I envisage a UI that allows a rebuild/reupload of a package in the same way 
that we can retry builds right now if there was some sort of intermittent 
failure.

> This doesn't really affect the schema though... I guess we need some way
> of noting on the SPBuildArchive link that the upload it represents has
> happened.

Yes, we do.  Right now we track this via Build.buildstate.  Each "xxxBuild" 
should have similar thing.  It would be nice to part-refactor this into some 
other table but I think it's too much hassle to change the Soyuz code for this 
versus the benefit.

Also one thing we lack with the rebuilds for Builds is some audit trail.  This 
new schema will allow that for recipe builds and will be really useful for 
tracking down problems.

> > Talking of which, I noticed that this patch doesn't have anything for
> > scheduling recipe builds.  Are you doing that separately?
> 
> I am consciously not thinking about that at the moment.  I guess it will
> involve a few more fields on SPRecipe.

I think a separate table would be better, e.g. SourcePackageRecipeSchedule.  
That's easy to leave out and do later.

> > I would expect Job rows to live forever*.  At least, we've refactored the
> > existing buildqueue stuff with that expectation.  Why would they be
> > removed?
> 
> Well, because there will be really rather a lot of them if we use a Job
> row for every branch pulled, code import updated, diff generated, daily
> recipe built, etc etc.  I don't know if I'm being overly concerned about
> this.

I think we need to be able to clean them up at some point, yes.
 
> > (* where forever == until we remove the whole set of related rows for
> > whatever reason)
> 
> Do you ever delete Build rows today?

Nope - it would destroy the link between source and binary.

We could consider doing it when we start completely blowing away distroseries 
(right now we just mark them as obsolete).

> >> I still don't know where to store it.  In the
> >> existing model I think this sort of thing is tracked in the
> >> SourcePackageRelease table (correct?)
> >
> > Sort of.  We record the source creator on the SourcePackageRelease and
> > the source uploader on the PackageUpload.
> >
> > You can see why I keep banging on about why creation and uploading are
> > orthogonal :)
> 
> Araragh, more bits of the Soyuz data model to chew on my brain.
> 
> Would it be accurate to say that in a world where all builds are done
> through branches and not dput that we would still have
> SourcePackageReleases but not PackageUploads?  

Not really, we still need to model uploads.

> But that today, the only
> way you can get a SourcePackageRelease is by starting with a PackageUpload?

I think this will always be the case unless we want to re-write Soyuz. :)

> >> which we don't have an analogue
> >> for yet in the build from recipe world -- I don't know if we need one
> >> though as SourcePackageRelease <-> Build is 1-to-many in a way that
> >> doesn't apply to recipes.
> >
> > Also, if they're automated daily builds, who is the requester?
> 
> I guess the person who set it up?  

Or a celebrity?  I dunno, I'm just throwing it out there.

> This starts to lead me into the "how
> similar are daily builds and the way we expect Ubuntu devs to use this"
> sort of thinking ...
> 
> Thanks for the comments!  (and sorry for all the typos in my first post).

Thanks for working on it!

J