maria-developers team mailing list archive

[worklog-noreply@xxxxxxxxxxxx]

-----------------------------------------------------------------------
                              WORKLOG TASK
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
TASK...........: Use Buildbot to populate apt/yum repositories
CREATION DATE..: Wed, 12 May 2010, 07:04
SUPERVISOR.....: Knielsen
IMPLEMENTOR....: Knielsen
COPIES TO......: 
CATEGORY.......: Other
TASK ID........: 117 (http://askmonty.org/worklog/?tid=117)
VERSION........: Server-9.x
STATUS.........: Assigned
PRIORITY.......: 60
WORKED HOURS...: 4
ESTIMATE.......: 16 (hours remain)
ORIG. ESTIMATE.: 20

PROGRESS NOTES:

-=-=(Knielsen - Wed, 12 May 2010, 21:20)=-=-
High-Level Specification modified.
--- /tmp/wklog.117.old.21448    2010-05-12 21:20:34.000000000 +0000
+++ /tmp/wklog.117.new.21448    2010-05-12 21:20:34.000000000 +0000
@@ -1,5 +1,5 @@
-As for signing, I think it may be possible/best to do the signing outside of
-buildbot, as a separate process. There are some advantages to this:
+The signing of packages can be done outside of Buildbot, as a separate
+process. There are some advantages to this:
 
  - Security: the private key can be kept less exposed when it is not included
    in the buildbot infrastructure.
@@ -9,9 +9,6 @@
 
  - Generally reducing the complexity of the buildbot setup.
 
-This of course requires that it is possible to sign the packages after the
-actual build.
-
 ----
 
 Here is how to sign the .rpms.
@@ -42,20 +39,37 @@
 
 ----
 
-For .deb, I *think* we are using secure apt, which does not actually sign the
-packages, rather it signs the "Release" file which is created when the
-repository is set up. So in this case again there is no problem doing the
-signing outside of the build itself (in fact that is the way it must be).
+For .deb, it is not the individual .deb that is signed, it is the
+repository. Here is one way to generate a signed repository, using reprepro.
 
-Found two tools that can help with building and signing apt repositories:
-reprepro (seems to be the newest, recommended) and apt-ftparchive.
+The ourdelta/bakery signing stuff needs to be copied to ~/.gnupg
 
-----
+mkdir repo # or whatever
+cd repo
+mkdir conf
+cat >conf/distributions <<END
+Origin: MariaDB
+Label: MariaDB
+Codename: hardy
+Architectures: amd64
+Components: mariadb-ourdelta
+Description: MariaDB test Repository
+SignWith: autosign@xxxxxxxxxxxx
+END
+for i in `find /home/buildbot/debs/ -name '*.deb'` ; do reprepro --basedir=.
+includedeb hardy $i ; done
+
+The corrosponding line for /etc/apt/sources.list:
 
-ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen
-suggested splitting up so we have this package ourselves, or maybe it can be
-handled with replace/provide/conflict dependencies.
+    deb file:///home/buildbot/repo hardy mariadb-ourdelta
+
+This works for multiple distributions, by adding more sections to the
+conf/distributions file.
+
+----
 
-ToDo: Figure out exactly what files/directory structure needs to be uploaded
-(asked Peter, awaiting reply).
+For the mysql-client-core-5.1 issue, the solution is to split the
+mariadb-client-5.1 (and 5.2) package similarly into
+mariadb-client-core-5.1. The mariadb-client-core-5.1 package then provides:
+mysql-client-core-5.1.
 

-=-=(Knielsen - Wed, 12 May 2010, 18:25)=-=-
High-Level Specification modified.
--- /tmp/wklog.117.old.12634    2010-05-12 18:25:58.000000000 +0000
+++ /tmp/wklog.117.new.12634    2010-05-12 18:25:58.000000000 +0000
@@ -12,9 +12,35 @@
 This of course requires that it is possible to sign the packages after the
 actual build.
 
-For .rpm this seems to be easy (from reading, didn't try yet):
+----
+
+Here is how to sign the .rpms.
+
+Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros.
+
+Run
+
+    rpm --addsign *.rpm
+
+That's all! This can be tested by creating a local yum repository:
 
-    rpm --addsign <packages>
+    createrepo <dir>
+
+(where <dir> contains the signed .rpms). Then create the file
+/etc/yum.repos.d/localmaria.repo:
+
+[localmaria]
+name=Local MariaDB repo
+baseurl=file:///home/buildbot/rpms
+gpgcheck=1
+enabled=1
+gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg
+
+Now this should work to install MariaDB:
+
+    sudo yum install MariaDB-server
+
+----
 
 For .deb, I *think* we are using secure apt, which does not actually sign the
 packages, rather it signs the "Release" file which is created when the

-=-=(Knielsen - Wed, 12 May 2010, 07:14)=-=-
High-Level Specification modified.
--- /tmp/wklog.117.old.401      2010-05-12 07:14:27.000000000 +0000
+++ /tmp/wklog.117.new.401      2010-05-12 07:14:27.000000000 +0000
@@ -1 +1,35 @@
+As for signing, I think it may be possible/best to do the signing outside of
+buildbot, as a separate process. There are some advantages to this:
+
+ - Security: the private key can be kept less exposed when it is not included
+   in the buildbot infrastructure.
+
+ - It is good to have one step of human intervention before actually signing
+   and releasing packages.
+
+ - Generally reducing the complexity of the buildbot setup.
+
+This of course requires that it is possible to sign the packages after the
+actual build.
+
+For .rpm this seems to be easy (from reading, didn't try yet):
+
+    rpm --addsign <packages>
+
+For .deb, I *think* we are using secure apt, which does not actually sign the
+packages, rather it signs the "Release" file which is created when the
+repository is set up. So in this case again there is no problem doing the
+signing outside of the build itself (in fact that is the way it must be).
+
+Found two tools that can help with building and signing apt repositories:
+reprepro (seems to be the newest, recommended) and apt-ftparchive.
+
+----
+
+ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen
+suggested splitting up so we have this package ourselves, or maybe it can be
+handled with replace/provide/conflict dependencies.
+
+ToDo: Figure out exactly what files/directory structure needs to be uploaded
+(asked Peter, awaiting reply).
 

-=-=(Knielsen - Wed, 12 May 2010, 07:06)=-=-
Upgraded lucid VMs to the official release.

Discussed with Arjen how to handle things.

Did a lot of reading on how apt repositories work.

Worked 4 hours and estimate 16 hours remain (original estimate unchanged).



DESCRIPTION:

Since the package building for MariaDB is now fully automated in Buildbot, it
has been decided to use packages from Buildbot for the OurDelta apt and yum
repositories.

This worklog is about fixing/implementing anything that is missing to achieve
this.

 - When doing a real release build, packages/repositories need to be signed,
   so that users will not get a warning about unauthenticated packages. This
   signing must only be done on official releases, not on daily builds (to
   avoid confusing one with the other).

 - Packages must be uploaded from the Buildbot host. The OurDelta
   infrastructure has a DropBox share that could be used for this, another
   option is to simply use rsync.

 - Ubuntu 10.04 "lucid" has been released, and we need to support that for
   packages, so the Buildbot VM for lucid must be upgraded to have the
   official release.

 - In Ubuntu 10.04, the official MySQL packages include a new package
   mysql-client-core, we currently have a conflict with this on install that
   we need to handle somehow.


HIGH-LEVEL SPECIFICATION:



The signing of packages can be done outside of Buildbot, as a separate
process. There are some advantages to this:

 - Security: the private key can be kept less exposed when it is not included
   in the buildbot infrastructure.

 - It is good to have one step of human intervention before actually signing
   and releasing packages.

 - Generally reducing the complexity of the buildbot setup.

----

Here is how to sign the .rpms.

Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros.

Run

    rpm --addsign *.rpm

That's all! This can be tested by creating a local yum repository:

    createrepo <dir>

(where <dir> contains the signed .rpms). Then create the file
/etc/yum.repos.d/localmaria.repo:

[localmaria]
name=Local MariaDB repo
baseurl=file:///home/buildbot/rpms
gpgcheck=1
enabled=1
gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg

Now this should work to install MariaDB:

    sudo yum install MariaDB-server

----

For .deb, it is not the individual .deb that is signed, it is the
repository. Here is one way to generate a signed repository, using reprepro.

The ourdelta/bakery signing stuff needs to be copied to ~/.gnupg

mkdir repo # or whatever
cd repo
mkdir conf
cat >conf/distributions <<END
Origin: MariaDB
Label: MariaDB
Codename: hardy
Architectures: amd64
Components: mariadb-ourdelta
Description: MariaDB test Repository
SignWith: autosign@xxxxxxxxxxxx
END
for i in `find /home/buildbot/debs/ -name '*.deb'` ; do reprepro --basedir=.
includedeb hardy $i ; done

The corrosponding line for /etc/apt/sources.list:

    deb file:///home/buildbot/repo hardy mariadb-ourdelta

This works for multiple distributions, by adding more sections to the
conf/distributions file.

----

For the mysql-client-core-5.1 issue, the solution is to split the
mariadb-client-5.1 (and 5.2) package similarly into
mariadb-client-core-5.1. The mariadb-client-core-5.1 package then provides:
mysql-client-core-5.1.


ESTIMATED WORK TIME

ESTIMATED COMPLETION DATE
-----------------------------------------------------------------------
WorkLog (v3.5.9)