launchpad-dev team mailing list archive

[William Grant <me@xxxxxxxxxxxxxxxxxx>]

On Fri, 2009-09-18 at 10:05 +0100, Julian Edwards wrote:
> On Wednesday 16 September 2009 13:31:56 William Grant wrote:
> > Hi all,
> 
> Hi William,

Hi Julian,

> > I'm currently attempting to finish bugs #285205 (Soyuz needs to be able
> > to process and publish ddebs) and #156575 (PPA builds do not create
> > -dbgsym packages). There is a large and fairly complicated issue with
> > the current implementation: the ddebs will live a life quite separate
> > from their debs (and even their sources!), in an archive untouchable by
> > any current tools, which breaks lots of assumptions.
> 
> Thank you for working on this.  It is indeed a tough problem and your further 
> analysis has exposed a lot of areas we hadn't originally thought about.
> 
> [snip]
> 
> > (1) can probably be fixed by assuming that every ddeb has a
> > corresponding deb with the same name, but with '-dbgsym' stripped off
> > the end. I'm not sure how that would go if any of the debs themselves
> > were NEW, however.
> 
> This is probably a good assumption, it will make things easier, although I 
> worry that it is easily broken.

AFAICT there is no other way to determine the corresponding deb.
Assuming that we restrict the search to the ddeb's build, how could it
break? Remember that ddebs are automatically generated, so they should
either match or there is an Ubuntu Developer playing tricks, in which
case we have bigger problems.

> I think that ddebs should not skip NEW and should do whatever the 
> corresponding deb does, including overrides.

As we discussed on IRC, they really should skip NEW. That is, the binary
upload will only be held in NEW if the debs are NEW -- a ddeb is
insufficient to hold it.

This doesn't quite match the current implementation, which I know to be
buggy anyway. This can be fixed by just always having a ddeb inherit its
corresponding deb's overrides. If the deb is NEW, no overrides for the
ddeb will be found, so both will end up NEW. If the deb is not NEW, the
ddeb will inherit the deb's overrides, and so will not be new. This also
fixes bug #430025 (Primary archive ddebs always end up in universe).
Some trickery will be required to avoid forcing archive admins to
initially override both deb and ddeb seperately.

> > To fix (2) the scripts themselves will have to be changed in fairly
> > obvious ways (as long as we can make the naming assumption).
> 
> Yes.  Any binary manipulation must also deal with ddebs in step.  We need to 
> review the code base and try and encapsulate stuff better, probably in the 
> publishing classes, so that future code automatically benefits and does the 
> right thing.

Most of it is already fairly well encapsulated, but perhaps the whole
source/binary/children selection mechanism that several scripts use
should be factored out.

> > To fix (3), at least SPPH.getBuiltBinaries,
> > PS._getSourceBinaryJoinForSources, PS.getFilesForSources,
> > PS.getBinaryPublicationsForSources will need to include the
> > corresponding debug archive in their search if looking in a primary
> > archive. I haven't had a thorough look for any other relevant methods.
> 
> We can do something similar to what we had to do for the partner archive here, 
> and that's extend the list of MAIN_ARCHIVE_PURPOSES.  This will fix some 
> stuff, and break other stuff.  Some of those methods above assume a single 
> archive; they will need fixing.

I considered that, but I don't particularly like the idea of including
partner in the calculation. Its set of packages and builds should be
disjoint from the rest, but it still feels a bit icky. I'm somewhat
considering a new Archive attribute which contains a list of grouped
archives. But perhaps that's over-complicating things, given that it
would be used for just this single case at present.

> Again, if we can encapsulate this behaviour inside the publishing classes I'll 
> be really happy.
> 
> > For (4), I think delayed copies might already be OK. They grab entire
> > builds and run them through the queue again, so things should work fine.
> 
> For copies from PPAs it should be ok.  For copies within the primary archive 
> it will get a little hairier because you hit unwanted upload checks for 
> duplicate files.  (I'm currently adding a feature to do this, mainly for 
> copying custom files around, so I'll bear it in mind.)

Right, intra-archive delayed copies are pretty broken at the moment
(although only because they perform a now-insane sanity check that there
are no existing ACCEPTED or DONE uploads in the archive for that
version?), but I don't think there's anything special that needs to be
done make the ddeb bits work.

I verified yesterday that PPA->primary delayed copies handle the debug
archive correctly.

> > Direct copies are another matter: they currently look just at binary
> > publications in the source, and create new publications in the target
> > archive. If the target is a primary archive, ddebs will need to be
> > forced into the debug archive.
> 
> Yes, the copying should be encapsulated in the publishing classes' copyTo() so 
> call sites don't need to think about it.

It seems pretty bad to have the method override the given archive, but I
see it started doing that to the component earlier this week. I guess it
makes sense.

> We'd need to fix the CopyChecker 
> class as well (which will also benefit delayed copies).

Good point. I'd forgotten about that.

> > While direct copies *into* a primary archive are probably not supported,
> 
> Actually this is possible.

I know it's currently possible, but should it be? It doesn't seem like a
good idea, what with the queue bypassing and all that.

> > I believe that the issues raised in (4) apply equally to
> > inter-pocket/series copies within a primary archive.
> 
> Yes - see above :)
> 
> > There are probably other things that I'll find to be broken. Any
> > thoughts about any of this, Soyuz guys?
> 
> Almost certainly.  It's a hard problem and we should take every opportunity to 
> fix our code so that we don't have separate pieces of code all making the same 
> assumptions, and abstract this behaviour into a single place.
> 
> I'll try and help you a bit more with this once 3.0 is out the door.

Great!

Thanks,
-- 
William Grant