kicad-developers team mailing list archive

[Adam Wolf <adamwolf@xxxxxxxxxxxxxxxxxxxx>]

If the team isn't opposed to me migrating away from the Launchpad
build servers to my own build server, I think I'm going to do that.

It ranges from difficult to impossible to run arbitrary commands on
the Launchpad build servers, but I'm certain there'd be a way to work
around it .  On the Wayne and Layne build cluster, I can do so
trivially.  The greater issue that makes me want to switch to the WNL
build server is the date in titlebar issue.

***Ignore this paragraph if you aren't interested in the nitty gritty
of the date-in-titlebar issue.*** The way the Launchpad build servers
work, is you get only three or four commands which involve bzr, and
then you can say "build", where it'll run the Debian/Ubuntu packaging
scripts, which have their own sets of restrictions.  The packaging
scripts right now are set up to build a few packages--kicad,
kicad-common, kicad-libraries, and kicad-docs.  I might have gotten
the exact names wrong but that's the idea.  The "average"
Debian/Ubuntu rules file only creates one package, but it is supported
(and encouraged in situations like ours) to create more than one.
However, due to the current directory structure, the Debian/Ubuntu
packaging information assumes it is located in a directory at the same
level as kicad, kicad-docs, and kicad-libs.  For "single package
builds", it is usually assumed that there is a debian/ directory (even
for Ubuntu), inside the main source directory of the main software.
In order to facilitate this, the Launchpad build servers let you merge
bzrs.  For a single package, you'd check out the upstream, and then
"nest" your personal bzr branch that contains the debian directory
inside of it.  Then, when the bzr Launchpad commands finish, it runs
the debian build.  Nesting only works inside the main bzr, not
parallel or closer to root than it in the directory tree.  Due to how
our package is currently set up, we actually grab my packaging bzr,
then nest kicad, kicad libs, and kicad docs inside of it.  It works
great, we get free compilations from Launchpad, it's very very low
overhead once you have the debian/ directory--with one small issue.
Because we're merging kicad's bzr into my debian bzr, when cmake looks
for the bzr revision and date during compilation, it sees *my* debian
bzr, which only changes once or twice a year as I make changes to
match Kicad development.  There are a variety of workarounds we could
do, but almost all of them involve massive modifications to the
debian/ scripts.  However, I already have a WNL build server
replicating the launchpad stuff, because I personally want to monitor
*every* check in, not just daily.  It'll be a small effort for me to
put a "post build action" of posting the .deb files to the PPA--and
this is how PPAs were started, actually, as the Launchpad autobuilds
are actually a relatively new development.

I understand the concern against "lock-in".  I also like the fact that
the build logs on the Launchpad servers are available to everyone.
Before I "cut over" to pushing packages from my build server to the
PPA rather than from the Launchpad packages to the build server, I
will find a way to also push the build logs.  I will also document the
Jenkins work so others can reproduce.  I already backup my Jenkins
server and the jobs are in XML.  I will look into what it takes to
push those XML files onto a new Jenkins install and get the same setup
I have, otherwise I'll just document it.

I can work with Miguel to get the build logs on the kicad-pcb.org site
if folks think that's an ok place for them to live.  IMO, Kicad needs
another website like I need another hole in my head :-)

Adam Wolf
Wayne and Layne, LLC

On Thu, Jun 6, 2013 at 10:41 AM, Dick Hollenbeck <dick@xxxxxxxxxxx> wrote:
>
> Hi Adam,
>
> Although I am not a user of the PPA nor of pre-packaged downloads for KiCad, I would
> consider this offer to be a *very generous* one from you.   I think my fellow KiCad users
> would benefit from it.
>
>
> Is there any way we can have a substitute plan, on the shelf, should you ever lose
> interest in providing this service?  Maybe a copy of some scripts or current PPA work,
> whatever would make the task easiest?
>
>
>
> On the problem at hand, I download the tar.bz2 file using wget at the command line, and
> the sourceforge download started out at only 8 K/sec and went up from there.  It took me
> 14 minutes plus to do the download.  The computed hash in the PPA logs is
>
> d41d8cd98f00b204e9800998ecf8427e
>
> and I find that this is the same you get on an empty file:
>
> $ touch empty.file
> $ md5sum empty.file
>
> d41d8cd98f00b204e9800998ecf8427e
>
>
> So the launchpad site may actually be blocking the download with a firewall rule or such.
> A TIMEOUT setting change is barely worth the try.
>
>
> As an experiment, I pushed the tar.bz2 file into its own bzr repo and pushed that to kicad
> project.  (Please do not tell launchpad folks.)  This exposes then an http URL at
>
>
> http://bazaar.launchpad.net/~kicad-testing-committers/kicad/boost/download/head:/boost_1_53_0.tar.bz2-20130606145758-00fag66fz00gstq5-1/boost_1_53_0.tar.bz2
>
>
> I then did wget against that URL, suddenly my rural internet connection became the
> bottleneck, as you would expect.  So launchpad did well.  The download only took 6 plus
> minutes.
>
>
> So some choices are available to you Adam.  As I said below, if you stuff the tar.bz2 file
> into the .downloads-by-cmake dir in advance of the first build, the download step will be
> skipped.  We can also use the launchpad URL, but that seems unnatural to me, and self
> defeating relative to the original idea of ease of upgrading boost.
>
>
> To fix the PPA using the services there, rather than your build cluster, it seems
> pre-loading the boost tar.bz2 file is necessary, because we/I suspect a firewall rule
> blocking the download.
>
>
>
> Dick
>
>
>
>
> On 06/06/2013 06:23 AM, Adam Wolf wrote:
>> I can't take a look at it before the weekend.
>>
>> My personal builds on the Wayne and Layne build cluster have been
>> working fine.  The PPA builds on the Launchpad box do not.  I don't
>> have hardly any control over the Launchpad box, and I recently
>> confirmed that due to how the Launchpad box service works, the
>> longstanding issue of "build date is wrong in title bar" won't be
>> fixed without a complete rewrite of the debian/ubuntu packaging.
>>
>> However, when talking to them about that, they let me know that I can
>> set up my build cluster to push packages into the PPA, and end users
>> won't see a difference.
>>
>> If the dev team doesn't object, this would likely be the easiest way
>> for me to get this stuff working (and fix a longstanding issue with
>> the PPAs at the same time.)
>>
>> Adam Wolf
>> Wayne and Layne, LLC
>>
>> On Thu, Jun 6, 2013 at 4:35 AM, Dick Hollenbeck <dick@xxxxxxxxxxx> wrote:
>>> I checked several of the package logs.  Lucid has its own problem but all
>>> the other recipes are failing with the same bad md5sum.  Maybe somebody
>>> updated the boost tar at sourceforge?  That would not be expected.
>>>
>>> Lucid is failing because I wanted cmake 2.8.4 and it is providing only
>>> 2.8.0.
>>>
>>>
>>> On Jun 6, 2013 3:54 AM, "Dick Hollenbeck" <dick@xxxxxxxxxxx> wrote:
>>>>
>>>>
>>>> This is from the ppa machine?  Has it happened more than once?  If so, is
>>>> the computed MD5 the same each time or random?
>>>>
>>>> The download can be done using wget and md5sum program can be used to
>>>> recheck binary by hand.  If still same, then I suggest on the ppa machine:
>>>>
>>>> Put the tar in the ppa recipe, copy it into correct dir from there.  When
>>>> cmake sees file there, it will skip download and go  to md5sum step and
>>>> continue from there.
>>>>
>>>> The ppa machine may not be providing enough contiguous cpu time for the
>>>> cmake C++ code to work correctly in the download function.  We can
>>>> alternatively provide a time out argument to the cmake download step as an
>>>> experiment.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Jun 6, 2013 12:42 AM, "Hans Henry von Tresckow" <hvontres@xxxxxxxxx>
>>>> wrote:
>>>>>
>>>>> It looks like to autobuilder is having trouble with the boost download.
>>>>> Here is the snippet from the build log:
>>>>>
>>>>> make[3]: Entering directory
>>>>> `/build/buildd/kicad-0.201306052021+4192~24~raring1/build/kicad'
>>>>> [  0%] Creating directories for 'boost'
>>>>> [  0%] Performing download step (download, verify and extract) for
>>>>> 'boost'
>>>>> -- downloading...
>>>>>
>>>>> src='http://downloads.sourceforge.net/project/boost/boost/1.53.0/boost_1_53_0.tar.bz2'
>>>>>
>>>>> dst='/build/buildd/kicad-0.201306052021+4192~24~raring1/kicad/.downloads-by-cmake/boost_1_53_0.tar.bz2'
>>>>>      timeout='none'
>>>>> CMake Error at boost-stamp/download-boost.cmake:9 (file):
>>>>>   file DOWNLOAD HASH mismatch
>>>>>
>>>>>     for file:
>>>>> [/build/buildd/kicad-0.201306052021+4192~24~raring1/kicad/.downloads-by-cmake/boost_1_53_0.tar.bz2]
>>>>>       expected hash: [a00d22605d5dbcfb4c9936a9b35bc4c2]
>>>>>         actual hash: [d41d8cd98f00b204e9800998ecf8427e]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 5, 2013 at 7:31 AM, Dick Hollenbeck <dick@xxxxxxxxxxx>
>>>>> wrote:
>>>>>>
>>>>>> On 05/31/2013 04:47 PM, Dick Hollenbeck wrote:
>>>>>>> In revision 4183 the boost headers have been removed from the repo.
>>>>>>>
>>>>>>> The CMakeLists.txt build environment now downloads those one time and
>>>>>>> installs them in the
>>>>>>> source tree upon first build.
>>>>>>
>>>>>>
>>>>>> In revision 4190:
>>>>>>
>>>>>> a) I made the download directory configurable,
>>>>>>
>>>>>> b) but it defaults now to ".downloads-by-cmake" rather than
>>>>>> downloads-by-cmake.
>>>>>>
>>>>>>
>>>>>> This period lets me skip this dir when grepping in tree.  If you want
>>>>>> to avoid
>>>>>> re-downloading, then checkout version 4190 and manually rename your
>>>>>> downloads-by-cmake
>>>>>> directory to .downloads-by-cmake immediately before your first build,
>>>>>> and the build will
>>>>>> skip the download step.  Otherwise you will end up with both
>>>>>> directories.
>>>>>>
>>>>>> Sorry, but I think we've got this bridge crossed now and the
>>>>>> disruptions are behind us.
>>>>>> The configure-ability of the directory also brings the benefit of
>>>>>> putting it in some out
>>>>>> of tree global place.
>>>>>>
>>>>>> Dick
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Subsequent builds will work as before, that is after boost
>>>>>>> is downloaded, un-tarred, patched, and copied to include/boost/*.
>>>>>>>
>>>>>>>
>>>>>>> (The PPA engine will however have to download them each time I
>>>>>>> suppose since it starts
>>>>>>> with a pristine bzr checkout each time.)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----< Revision Summary >--------------------------------------------
>>>>>>>
>>>>>>> This revision makes include/boost/* files into an "external project"
>>>>>>> according to CMake's
>>>>>>> ExternalProject_Add() function. The main advantages to this strategy
>>>>>>> are:
>>>>>>>
>>>>>>> 1) it is easier to track the totality of all patches made to the
>>>>>>> particular version of
>>>>>>> boost in use.
>>>>>>>
>>>>>>> 2) The procedure for the download and patching is extremely well
>>>>>>> documented and
>>>>>>> reproducable, unlike now, so therefore *easier to upgrade to new
>>>>>>> boost*.
>>>>>>>
>>>>>>> 3) You get the full set of boost headers.
>>>>>>>
>>>>>>> 4) The KiCad repo is smaller.
>>>>>>>
>>>>>>>
>>>>>>> The mechanism uses a new directory in the source tree called
>>>>>>> downloads-by-cmake to hold
>>>>>>> the boost*.tar.bz2 file. This download happens only one time, ever.
>>>>>>> Then the tar file is
>>>>>>> expanded, and it is put into a scratch bazaar repo so that changes
>>>>>>> can be tracked. Then it
>>>>>>> is patched. Then a portion of the patched boost, namely the header
>>>>>>> portion, is copied into
>>>>>>> the KiCad source tree at include/boost just as now. So the end result
>>>>>>> is the same by the
>>>>>>> time a build is undertaken.
>>>>>>>
>>>>>>> The scratch repo remains in downloads-by-cmake, so that patches can
>>>>>>> be re-generated from
>>>>>>> there at any time in the future for the ExternalProject_Add()
>>>>>>> mechanism. You can delete
>>>>>>> the directory downloads-by-cmake to get a fresh start at any time.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~kicad-developers
>>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Henry von Tresckow (hvontres)
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~kicad-developers
>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>
>