coapp-developers team mailing list archive

[Garrett Serack <garretts@xxxxxxxxxxxxx>]

LOL.

I keep having to reassure command-line folks that it will be all automateable, scriptable, etc... and I have to keep reassuring GUI folks that it will work in the GUI.

I've already done this automated with my generation 1 tools with about ~120 libraries and modules. Well, most of it. That's why I'm building new tools... ones that will be flexible and funciton well under automation.

The process I described was really a 'think-of-it-this-way' rather than a 'this-is-your-life,-get-used-to-it' :D

Oh, and FYI, I've got PLENTY of server resources for build servers. :D

G
________________________________
From: Adam Kennedy [adamkennedybackup@xxxxxxxxx]
Sent: Wednesday, April 14, 2010 6:44 PM
To: Garrett Serack
Cc: coapp-developers@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Coapp-developers] How exactly will CoApp come together?

That sounds like a lot of steps...

To what extent are these steps going to be automatable, chainable and validatable?

I have 18,000 packages I'd be feeding in (and a release stream that runs at around 30-80 per day) and of the 5-10 existing downstream repositories that are currently built against  the absolute best that script-assisted human packagers have been able to reach is about 1,500 (that's the Debian Perl packaging team, and even they automate most of the process and spend most of their time doing legal and licensing checks and spotting special cases). They also tend to fall quite a ways behind the new releases, and concentrate on strategic updates as something above it adds a critical dependency.

The best of the fully automated packaging streams (FreeBSD from memory) can build around 7,000 packages, and stays pretty much entirely up to date.

I completely agree that the initial focus should be on C libraries, because those cause the most problems for everybody else.

But please bear in mind as soon as you go multi-lingual you will run into the big exist Perl/Python/PHP/Ruby/Haskell/R/et al repositories, and mass-automation is the only way to really make them tractable.

Hearing "Start visual studio, run this, then exit and run this command line tool, then review, then run that tool, then start visual studio again, etc etc" makes me wonder if you're going to hit difficulties when you start to hit mass production.

Adam K

On 13 April 2010 04:23, Garrett Serack <garretts@xxxxxxxxxxxxx<mailto:garretts@xxxxxxxxxxxxx>> wrote:
So,

I’ve been taking questions as to how CoApp packages get built.

Lemme see if I can sketch out the vision for you, so that you get an idea of where it’s going. This isn’t set in stone, but I’ve actually validated this is a workable solution.

Let’s say I want to create a library package for zlib.

First, I’m going to import the zlib source code into a Bazaar in a new CoApp sub-project on Launchpad.

Checking out from there, I’ll first see if the project can be compiled at all using MSVC (any version).  If it has an older project file, I’ll load it up in Visual Studio 10, and let it upgrade the project, and I’ll save it.

Drop back to the command line.

The SCANTOOL file can be pointed to the source directory to scan thru all the source files and build files to generate some intelligence about the project as a whole. It gets a list of all source files (C,C++,.H, etc), potential conditional defines present in the source (#define FOO …) and identifies what additional files are present in the project (for which we’ll have to determine what to do with them (delete, include in final as resources, ???). SCANTOOL dumps all of this data into an XML intelligence file for the project.

Build the project (either by the makefile, the vcprojx file, or whatever means necessary). When doing so however,  use the TRACE tool to watch the library get built. TRACE creates an XML file with every file access, write, read, delete and every command line for the build process and all its child processes.

At this point the developer can create a hand-made intelligence file as well for things that are known about the project (what targets are desired, etc).

The intelligence files and the trace data are fed into another tool MKSPEC, which creates a set of .spec files, each of which describes a binary output desired from the project (a .LIB , .DLL, .EXE, etc) and lists the files needed, conditional #defines, and other options. (this is essentially a compiler-neutral way of representing what is needed to build a particular output)

Each .spec file is then fed into MKPROJECT which will generate a VC10 project file. Plugins for MKPROJECT can trivially build other types of project files for things like VC9, make files for MinGW or CMake files for the CMake faithful. MKProject also ties together a collection of project files into a .SLN file for Visual Studio. Outputs are normalized for naming conventions.

The .SLN file is fed into Visual Studio (or MSBuild, the command line tool) and it compiles up the binaries.  (I’ve got a plan for PGO as well, [profile guided optimization], but I’m going to ignore that right now)

The binaries are fed into a tool called SMARTMANIFEST which creates .manifest and policy files for the library and binds them to any .DLLs and .EXEs created.

The binaries (and manifest data) along with the project source code and build files are fed into MKPACKAGE which uses WiX to build MSI files for each binary, along with a source MSI with just the necessary files to rebuild the binaries (source, vcxproj, sln).

At that point the developer can identify what files can be trimmed from the source tree, and the whole thing can be updated in Bazaar.

http://twitpic.com/rqmo5 -- a flowchart of what I just described. Well, without TRACE.

(there’s a lot more detail to be found, but that’s the gist of it)


<http://fearthecowboy.com/>

Garrett Serack | Microsoft's Open Source Software Developer | Microsoft Corporation
Office:(425)706-7939                                       email/messenger: garretts@xxxxxxxxxxxxx<mailto:garretts@xxxxxxxxxxxxx>
blog: http://fearthecowboy.com<http://fearthecowboy.com/>                                      twitter: @fearthecowboy<http://twitter.com/fearthecowboy>

I don't make the software you use; I make the software you use better on Windows.







_______________________________________________
Mailing list: https://launchpad.net/~coapp-developers<https://launchpad.net/%7Ecoapp-developers>
Post to     : coapp-developers@xxxxxxxxxxxxxxxxxxx<mailto:coapp-developers@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~coapp-developers<https://launchpad.net/%7Ecoapp-developers>
More help   : https://help.launchpad.net/ListHelp