launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Julian Edwards wrote:
> Finally, I get a chance to reply to this.  Sorry for the delay.  This is also 
> a long reply :)
> 
> On Thursday 26 November 2009 02:02:36 Michael Hudson wrote:
>> Michael Hudson wrote:
>>> The branch at ~mwhudson/launchpad/recipe-db-schema adds a patch to add
>>> SourcePackageRecipe, SourcePackageRecipeBranch and
>>> BuildSourcePackageFromRecipeJob tables.  You can also see it at
>>> http://pastebin.ubuntu.com/327343/.
>> I talked to Jono and Julian a bit, and have made some changes.  New
>> patch at http://pastebin.ubuntu.com/328140/ .  It's certainly a fairly
>> large patch and I'd feel a little uncomfortable about landing it without
>> some model code (at least) to back it up.
> 
> Some comments:
> 
>  * Should SourcePackageRecipeData have an owner or created_by?

I don't think so.  SourcePackageRecipeData is really just a shared
implementation detail of SourcePackageBuild and SourcePackageRecipe, and
both of those should record all the interesting stuff.

>  * Why is SourcePackageRecipe a separate table to SourcePackageRecipeData? 
> Since one references the other, the duplicated columns are not needed are 
> they?  In fact the only additional column is the recipe text. Maybe I don't 
> understand what you're trying to do with this?

Ah right, I meant to remove those duplicate columns from SPRecipeData.
Essentially the reason they're separate is because SPRecipes have access
control and a location in the UI, but SPRecipeDatas don't as many of
them will be read-only versions referenced from SPBuild rows.

>  * SourcePackageRecipeDataBranch isn't really needed yet, we're not creating 
> branches for daily builds.  It's not a problem adding it though I guess.

It's meant to be there to record the links between the recipe and the
branches it references.  I guess I don't understand what you mean.

>  * in SourcePackageBuild:
>   * you have a comment that you dropped "processor" but it's still there :)

Arglbargl.  Fixed.

>   * I don't think you need "archive", this is not pertinent until the package 
> is actually uploaded.  At least, I think we can share these across archives.  
> Can anyone think of a reason why we would not do that?

Well, somehow we need to record the archive(s?) to which the resulting
source pacakge will be built.  Maybe not in this table though, indeed.
More on this below.

>   * you don't need "dependencies"

Cool.

>   * you probably still want upload_log as on Build - this records the log of 
> when the package is uploaded back into Soyuz.  This stage can go wrong if the 
> resulting package is no good and the log says why.

OK.

>   * I see you have a column called "manifest" which references 
> SourcePackageRecipeData.  Now I'm really confused about what that is!  I 
> thought recipes and manifests were separate things?  If it needs to be 
> separate why is it not SourcePackageRecipeManifest?

I hope this makes sense by now?  If we just stored recipe/manifests as
text, both SPRecipe and SPBuild would just have a text column, but we're
planning on being a bit more sophisticated than that.

>  * BuildSourcePackageFromRecipeJob - this might as well contain everything in 
> SourcePackageBuild unless you have a good reason for separating them?

Why would it make sense to duplicate the columns across the tables?

I thought we went over why BuildSourcePackageFromRecipeJob and SPBuild
are separate last week.

>>> I think it's mostly fairly uncontroversial.  Some remaining points:
>>>
>>>  - My unique index assumes that the url will be
>>> https://launchpad.net/ubuntu/sourcepackage/+recipe/name or very similar.
>>>  Arguments welcome about whether the owner should be in there :-)
>> I think we should keep the name in the url.
> 
> Yes, definitely.
> 
> The owner is another thing though :)

Gaaar.

> I'm not sure, and replied to the other emails about this separately.

At some point I guess we need to just decide.

>>>  - The big missing part is where to store the log file and manifest that
>>> are produced by the job's execution.  We could be more like the existing
>>> Soyuz code and have a separate "Build" table that points at these
>>> things, or they could live on BuildSourcePackageFromRecipeJob
>> I think a separate SourcePackageRecipeBuild table mimicing Build (hey,
>> anyone want to rename that BinaryPackageBuild or something? ;-) makes
>> most sense, so I've added that.
>>
>>>  - This doesn't get into the mess of SourcePackageRecipe subscriptions.
>>>
>>>  - My patch assumes that a built sourcepackage will be uploaded uploaded
>>> to exactly one archive.  Julian expressed some concerns about wanting to
>>>  be able to upload to more than one, but I think we can probably survive
>>> without this for now.
>> This is still true and I'd still like someone to shout if they disagree :)
> 
> Actually, having "archive" on the build record is fine, as builds are always 
> targeted to a particular archive.  I was more worried about putting it on a 
> recipe or a sourcepackage.

Right, that point got hammered out of me a while ago now.

> Having said that, I would still vastly prefer a separate table linking 

I object slightly to the 'still' -- it feels like I've been trying to
get a straight answer on this point for a while!  But thanks for
providing one (even if I still have the urge to avoid this bit of
complexity for now).

> SourcePackageBuild and Archives.

It should be easy enough to do this.  I guess the upload_log would live
on the SPBuildArchive link?

> This allows the possibility of re-uploading 
> the same source recipe build if we want.  I think that will be important for 2 
> reasons:
>  1. re-creating upload issues
>  2. re-creating genuine package bugs in a different PPA environment

I guess I'm not sure about some of the mechanics of this.  In the
standard way things will work, the buildd-manager will grab the built
package and upload them for building into any archives that are
specified at that time.  How will the upload happen for one of these
after-the-fact requests?

This doesn't really affect the schema though... I guess we need some way
of noting on the SPBuildArchive link that the upload it represents has
happened.

> Talking of which, I noticed that this patch doesn't have anything for 
> scheduling recipe builds.  Are you doing that separately?

I am consciously not thinking about that at the moment.  I guess it will
involve a few more fields on SPRecipe.

>>>  - There was talk earlier about needing to record who requested a source
>>> package build; it turns out that Job already has a requester field, so
>>> we don't need one on BuildSourcePackageFromRecipeJob.
>> However, if the Job row goes away before the SourcePackageRecipeBuild
>> row, then we'd lose that.
> 
> I would expect Job rows to live forever*.  At least, we've refactored the 
> existing buildqueue stuff with that expectation.  Why would they be removed?

Well, because there will be really rather a lot of them if we use a Job
row for every branch pulled, code import updated, diff generated, daily
recipe built, etc etc.  I don't know if I'm being overly concerned about
this.

> (* where forever == until we remove the whole set of related rows for whatever 
> reason)

Do you ever delete Build rows today?

>> I still don't know where to store it.  In the
>> existing model I think this sort of thing is tracked in the
>> SourcePackageRelease table (correct?)
> 
> Sort of.  We record the source creator on the SourcePackageRelease and the 
> source uploader on the PackageUpload.
> 
> You can see why I keep banging on about why creation and uploading are 
> orthogonal :)

Araragh, more bits of the Soyuz data model to chew on my brain.

Would it be accurate to say that in a world where all builds are done
through branches and not dput that we would still have
SourcePackageReleases but not PackageUploads?  But that today, the only
way you can get a SourcePackageRelease is by starting with a PackageUpload?

>> which we don't have an analogue
>> for yet in the build from recipe world -- I don't know if we need one
>> though as SourcePackageRelease <-> Build is 1-to-many in a way that
>> doesn't apply to recipes.
> 
> Also, if they're automated daily builds, who is the requester?

I guess the person who set it up?  This starts to lead me into the "how
similar are daily builds and the way we expect Ubuntu devs to use this"
sort of thinking ...

>> What I think I'm trying to say is that who requested the recipe be built
>> into a source package can be stored in the SourcePackageRecipeBuild table.
> 
> That sounds perfect to me.

Oh good.

>>> Comments welcome -- it would be nice to land this this cycle so we can
>>> start working on things for realz next cycle.
>> Screw this part: we can just land it on db-devel.  Rushing is bad
> 
> Agreed.

Cool.

Thanks for the comments!  (and sorry for all the typos in my first post).

There's a new patch at http://pastebin.ubuntu.com/332821/

Cheers,
mwh