canonical-ubuntu-qa team mailing list archive
-
canonical-ubuntu-qa team
-
Mailing list archive
-
Message #02140
[Merge] ~hyask/autopkgtest-cloud:skia/local_dev into autopkgtest-cloud:master
Skia has proposed merging ~hyask/autopkgtest-cloud:skia/local_dev into autopkgtest-cloud:master.
Requested reviews:
Canonical's Ubuntu QA (canonical-ubuntu-qa)
For more details, see:
https://code.launchpad.net/~hyask/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/456612
--
Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~hyask/autopkgtest-cloud:skia/local_dev into autopkgtest-cloud:master.
diff --git a/.gitignore b/.gitignore
index 6a76737..0682011 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
docs/_build
*.charm
+__pycache__
diff --git a/charms/focal/autopkgtest-web/webcontrol/publish-db b/charms/focal/autopkgtest-web/webcontrol/publish-db
index 76883bd..759cf13 100755
--- a/charms/focal/autopkgtest-web/webcontrol/publish-db
+++ b/charms/focal/autopkgtest-web/webcontrol/publish-db
@@ -74,7 +74,9 @@ def init_db(path, path_current, path_rw):
logging.debug("Old current_versions copied over")
current_version_copied = True
except sqlite3.OperationalError as e:
- if "no such column: pocket" not in str(
+ if "no such table: current" in str(e):
+ logging.debug("current database is empty, nothing to copy")
+ elif "no such column: pocket" not in str(
e
) and "no such column: component" not in str(
e
diff --git a/docs/deploying.rst b/docs/deploying.rst
index 85d853f..ddad0f9 100644
--- a/docs/deploying.rst
+++ b/docs/deploying.rst
@@ -214,7 +214,7 @@ don't worry about messing it up. For that reason it's important to keep
automated deployments working and eliminate the need for post-deploy manual
hacks.
-testing wip changes
+Testing WIP changes
^^^^^^^^^^^^^^^^^^^
The ``charm release`` command demonstrated above releases to the *stable*
@@ -226,5 +226,112 @@ merged into the main branch, you can release into *edge* with ``charm release
$ mojo run
-under the staging user as usual to test your change. Staging tracks edge by
+Under the staging user as usual to test your change. Staging tracks edge by
default.
+
+
+Deploying a local development environment
+-----------------------------------------
+
+Prerequisites
+^^^^^^^^^^^^^
+
+An OpenStack environment
+""""""""""""""""""""""""
+
+*canonistack*, despite working enough to allow you to complete the deployment,
+probably won't really work and you'll have a lot of process choking on
+``openstack`` commands returning errors.
+
+`microstack <https://microstack.run/>`_ is currently a more viable option,
+albeit is doesn't provide ``swift`` storage yet. It is quick and easy to set
+up in a bridged VM if you don't want it running on your host or don't have an
+Ubuntu host available.
+
+Running this will allow you to fully understand everything and will help you get
+more comfortable with debugging infrastructure issues.
+
+For the following guide, we'll assume you have that environment RC file named
+``devstack.rc``.
+
+Here are some quick tips if you go with your own *microstack*. Do that after you
+complete the official setup:
+
+ * You can use the `admin` user, but it is very much advised to use the `demo`
+ one instead, to avoid developing features with overprivileged user. Its
+ RC file should be created when you configure OpenStack under the name
+ ``demo-openrc``.
+
+ * Run ``sunbeam openrc`` to get the admin credentials.
+
+ * Make sure you have the correct route to OpenStack ``external-network`` on
+ your machine: ``sudo ip route add 10.20.20.0/24 via $microstack_machine_ip``
+
+ * Run ``sunbeam dashboard-url`` to know where the admin web dashboard is.
+
+ * From admin dashboard:
+
+ * Activate DHCP on ``external-subnet`` (subnet of ``external-network``), so
+ that your instances easily get an IP.
+
+ * Make ``external-network`` public, so that it can be used by the `demo` user.
+
+ * Edit default security group to allow inbound SSH.
+
+ * Upload a first image usable by `autopkgtest-cloud` (name it something like
+ ``ubuntu-$release-daily-amd64-server-20231207-disk1.img``). You
+ can grab a ``$release-server-cloudimg-amd64.img`` file from https://cloud-images.ubuntu.com/
+
+ * Create a public flavor usable for `autopkgtest-cloud`. Default name is
+ ``cpu2-ram4-disk20``, configurable in ``mojo/service-bundle``, look for
+ ``worker-default-flavor``.
+
+**NOTE**: as of 2023-12, with `snap` `openstack` version `2023.1`, you may have
+issues when rebooting the machine it's running on. See `this Mattermost thread
+<https://chat.charmhub.io/charmhub/pl/xkcuak5wrin67mrxirxobc3cte>`_ for hints on
+how to debug that.
+
+
+swift storage
+"""""""""""""
+
+*canonistack* can do if you have access to it, and it's the default *develop*
+configuration in ``mojo/service-bundle``, but any other environment should do.
+
+Once ``microstack`` support swift out of the box, you will be able to use that
+instead.
+
+A ``juju`` cloud
+""""""""""""""""
+
+The OpenStack environment will not be used to run the ``autopkgtest-cloud`` code
+itself, as it is only for the cloud worker to use. The code will run locally on
+whatever ``juju`` cloud you set up with ``juju bootstrap``. A local LXD cloud
+is fine.
+
+Please note that on a local LXD cloud, you may run into some strange network issues.
+They may be `related to Docker + LXD <https://documentation.ubuntu.com/lxd/en/latest/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-lxd-and-docker>`_.
+
+
+
+Running autopkgtest-cloud
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The procedure is mostly the same as described in this whole page, except there
+are a few steps that can gain you some time:
+
+1. To set up the ``mojo`` environment, you have a ``mojorc`` file at the root of this repo that you can source and should work out of the box for a devel environment.
+2. Not every secrets are needed in local dev, so ``~/.local/share/mojo/LOCAL/autopkgtest-cloud/devel`` should only contain the following:
+ * ``devstack.rc`` for your OpenStack access
+ * ``influx-{hostname,password}.txt`` with dummy values, the files only need to exist
+ * ``novarcs.tar`` created as described in the *Supply secrets* section (``tar cvf novarcs.tar devstack.rc`` will do)
+ * ``swift_password`` with your OpenStack password found in your ``canonistack.rc`` (or any other ``swift`` password from another environment)
+3. Edit ``mojo/service-bundle`` for the following values, each time in the ``devel`` branch of the code:
+ * ``swift-username``
+ * ``swift-project-name``
+ * ``storage_path_internal``
+ * ``charm`` for `autopkgtest-cloud-worker`, `autopkgtest-lxd-worker`, and `autopkgtest-web` applications
+4. Build the local charms, as they are used in `devel` configuration by default:
+ * ``cd charms/focal/autopkgtest-cloud-worker && charmcraft clean && charmcraft pack``
+ * ``cd charms/focal/autopkgtest-web && charmcraft clean && charmcraft pack``
+5. ``mojo run`` should now be working enough to get you a local working web UI
diff --git a/mojo/service-bundle b/mojo/service-bundle
index ee90d2f..a44c740 100644
--- a/mojo/service-bundle
+++ b/mojo/service-bundle
@@ -22,11 +22,12 @@ description: "autopkgtest-cloud"
series: {{ series }}
applications:
autopkgtest-cloud-worker:
+{%- if stage_name == "production" or stage_name == "staging" %}
charm: ubuntu-release-autopkgtest-cloud-worker
channel: {{ channel }}
-{%- if stage_name == "production" or stage_name == "staging" %}
num_units: 2
{%- else %}
+ charm: XXX/path/to/autopkgtest-cloud-git-repo/XXX/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud-worker_ubuntu-20.04-amd64.charm
num_units: 1
{%- endif %}
constraints: mem=16G cores=8 root-disk=40G
@@ -60,11 +61,14 @@ applications:
swift-tenant: stg_proposed-migration_project
swift-auth-version: 2
{%- elif stage_name == "devel" %}
- swift-auth-url: XXX
- swift-username: XXX
- swift-region: XXX
+ influxdb-username: dev_proposed_migration
+ influxdb-context: devel
+ # Most Canonistack values can be found in your canonistack novarc file
+ swift-auth-url: https://keystone.bos01.canonistack.canonical.com:5000/v3
+ swift-username: XXX # canonistack username
+ swift-region: canonistack-bos01
swift-project-domain-name: default
- swift-project-name: XXX
+ swift-project-name: XXX_project # canonistack project
swift-user-domain-name: default
swift-auth-version: 3
{%- endif %}
@@ -77,9 +81,7 @@ applications:
{%- elif stage_name == "staging" %}
net-name: net_stg-proposed-migration
{%- elif stage_name == "devel" %}
- net-name: net_instances
- mirror: http://ports.ubuntu.com/ubuntu-ports/
- worker-args: ssh -s /CHECKOUTDIR//ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname testbed-/HOSTNAME/ --mirror=/MIRROR/
+ net-name: external-network
{%- endif %}
{%- if stage_name == "production" or stage_name == "staging" %}
mirror: http://ftpmaster.internal/ubuntu/
@@ -87,6 +89,11 @@ applications:
worker-big-flavor: autopkgtest-big
worker-args: ssh -s /CHECKOUTDIR//ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$HOSTARCH-server --keyname testbed-/HOSTNAME/ --net-id=/NET_NAME/ -e TERM=linux -e 'http_proxy={{ http_proxy }}' -e 'https_proxy={{ https_proxy }}' -e 'no_proxy={{ no_proxy }}' --mirror=/MIRROR/
worker-setup-command: /AUTOPKGTEST_CLOUD_DIR//worker-config-production/setup-canonical.sh
+{% elif stage_name == "devel" %}
+ mirror: http://archive.ubuntu.com/ubuntu/
+ worker-default-flavor: cpu2-ram4-disk20
+ worker-big-flavor: cpu4-ram8-disk50
+ worker-args: ssh -s /CHECKOUTDIR//ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname testbed-/HOSTNAME/ --mirror=/MIRROR/
{% endif %}
{%- if stage_name == "production" %}
n-workers: |-
@@ -114,13 +121,18 @@ applications:
s390x: 1
{%- elif stage_name == "devel" %}
n-workers: |-
- bos01:
+ devstack:
+ amd64: 1
arm64: 0
ppc64el: 0
{%- endif %}
autopkgtest-lxd-worker:
+{%- if stage_name == "production" or stage_name == "staging" %}
charm: ubuntu-release-autopkgtest-cloud-worker
channel: {{ channel }}
+{%- else %}
+ charm: XXX/path/to/autopkgtest-cloud-git-repo/XXX/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud-worker_ubuntu-20.04-amd64.charm
+{%- endif %}
num_units: 1
constraints: mem=16G cores=8 root-disk=40G
{%- if stage_name == "production" or stage_name == "staging" %}
@@ -173,8 +185,12 @@ applications:
enable_modules: include cgi proxy proxy_http
mpm_type: prefork
autopkgtest-web:
+{%- if stage_name == "production" or stage_name == "staging" %}
charm: ubuntu-release-autopkgtest-web
channel: {{ channel }}
+{% else %}
+ charm: XXX/path/to/autopkgtest-cloud-git-repo/XXX/charms/focal/autopkgtest-web/autopkgtest-web_ubuntu-20.04-amd64.charm
+{%- endif %}
options:
hostname: {{ hostname }}
{%- if stage_name == "production" %}
@@ -184,8 +200,10 @@ applications:
{%- set storage_host_internal = "objectstorage.prodstack5.canonical.com:443" %}
{%- set storage_path_internal = "/swift/v1/AUTH_cc509e38c54f4edebda2fd17557309bb" %}
{%- elif stage_name == "devel" %}
- storage_host_internal: XXX
- storage_path_internal: XXX
+ {# canonistack objectstorage URL, find this with `swift auth` #}
+ {%- set storage_host_internal = "swift-proxy.bos01.canonistack.canonical.com:8080" %}
+ {# canonistack swift path, find this with `swift auth` #}
+ {%- set storage_path_internal = "/v1/AUTH_0123456789abcdef0123456789abcdef" %}
{%- endif %}
storage-url-internal: https://{{ storage_host_internal }}{{ storage_path_internal }}
{%- if stage_name == "production" or stage_name == "staging" %}
diff --git a/mojorc b/mojorc
new file mode 100644
index 0000000..cc81516
--- /dev/null
+++ b/mojorc
@@ -0,0 +1,14 @@
+# This file will setup your environment for a local develop `mojo run`.
+# Please have a look at the "Deploying" documentation page to know how to use this file.
+
+base_dir="$(dirname "$(realpath "$0")")"
+
+export MOJO_ROOT=~/.local/share/mojo
+export MOJO_SERIES=focal
+export MOJO_PROJECT=autopkgtest-cloud
+export MOJO_WORKSPACE=autopkgtest-cloud
+export MOJO_SPEC="$base_dir/mojo/"
+export MOJO_STAGE=devel
+
+mojo project-new $MOJO_PROJECT -s $MOJO_SERIES --container containerless
+mojo workspace-new --project $MOJO_PROJECT -s $MOJO_SERIES $MOJO_SPEC $MOJO_WORKSPACE
