cloud-init-dev team mailing list archive

[Chad Smith <chad.smith@xxxxxxxxxxxxx>]

initial doc notes. haven't gotten to the code yet. This looks good/very helpful.

Diff comments:

> diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
> index 0663811..0e470c6 100644
> --- a/doc/rtd/topics/tests.rst
> +++ b/doc/rtd/topics/tests.rst
> @@ -47,20 +47,29 @@ The test configuration is a YAML file such as *ntp_server.yaml* below:
>          cat /etc/ntp.conf | grep '^server'
>  
>  
> -There are two keys, 1 required and 1 optional, in the YAML file:
> +There are several keys, 1 required and some optional, in the YAML file:
>  
>  1. The required key is ``cloud_config``. This should be a string of valid
>     YAML that is exactly what would normally be placed in a cloud-config file,
>     including the cloud-config header. This essentially sets up the scenario
>     under test.
>  
> -2. The optional key is ``collect_scripts``. This key has one or more
> +2. One optional key is ``collect_scripts``. This key has one or more
>     sub-keys containing strings of arbitrary commands to execute (e.g.
>     ```cat /var/log/cloud-config-output.log```). In the example above the
>     output of dpkg is captured, grep for ntp, and the number of lines
>     reported. The name of the sub-key is important. The sub-key is used by
>     the verification script to recall the output of the commands ran.
>  
> +3. The optinal key ``enabled`` enables or disables the test case. By defaul

s/optinal/optional/
s/defaul/default/

> +   the test case will be enabled.
> +
> +4. The optional key ``required_features`` key may be used to specify a list

s/optional key/optional /

> +   of features flags that an image must have to be able to run the testcase.
> +   For example, if a testcase relies on an image supporting apt, then the
> +   config for the testcase should include ``required_features: [ apt ]``.
> +
> +
>  Default Collect Scripts
>  -----------------------
>  
> @@ -252,15 +316,259 @@ configuration users can run the integration tests via tox:
>  Users need to invoke the citest enviornment and then pass any additional
>  arguments.
>  
> +Setup Image
> +-----------
> +The ``run`` and ``collect`` commands have many options to setup the image
> +before running tests in addition to installing a deb in the target. Any
> +combination of the following can be used:
> +
> +* ``--deb``: install a deb into the image
> +* ``--rpm``: install a rpm into the image
> +* ``--repo``: enable a repository and update cloud-init afterwards
> +* ``--ppa``: enable a ppa and update cloud-init afterwards
> +* ``--upgrade``: upgrade cloud-init from repos
> +* ``--upgrade-full``: run a full system upgrade
> +* ``--script``: execute a script in the image. this can perform any setup
> +  required that is not covered by the other options
> +
> +Configuring the Test Suite
> +==========================
> +
> +Most of the behavior of the test suite is configurable through several yaml
> +files. These control the behavior of the test suite's platforms, images, and
> +tests. The main config files for platforms, images and testcases are
> +``platforms.yaml``, ``releases.yaml`` and ``testcases.yaml``.
> +
> +Config handling
> +---------------
> +All configurable parts of the test suite use a defaults + overrides system for
> +managing config entries. All base config items are dictionaries.
> +
> +Merging is done on a key by key basis, with all keys in the default and
> +overrides represented in the final result. If a key exists both in
> +the defaults and the overrides, then behavior depends on the type of data the
> +key refers to. If it is atomic data or a list, then the overrides will replace
> +the default. If the data is a dictionary then the value will be the result of
> +merging that dictionary from the default config and that dictionary from the
> +overrides.
> +
> +Merging is done using the function ``tests.cloud_tests.config.merge_config``,
> +which can be examined for more detail on config merging behavior.
> +
> +The following demonstrates merge behavior:
> +
> +.. code-block:: yaml
> +
> +    defaults:
> +        list_item:
> +         - list_entry_1
> +         - list_entry_2
> +        int_item_1: 123
> +        int_item_2: 234
> +        dict_item:
> +            subkey_1: 1
> +            subkey_2: 2
> +            subkey_dict:
> +                subsubkey_1: a
> +                subsubkey_2: b
> +
> +    overrides:
> +        list_item:
> +         - overridden_list_entry
> +        int_item_1: 0
> +        dict_item:
> +            subkey_2: false
> +            subkey_dict:
> +                subsubkey_2: 'new value'
> +
> +    result:
> +        list_item:
> +         - overridden_list_entry
> +        int_item_1: 0
> +        int_item_2: 234
> +        dict_item:
> +            subkey_1: 1
> +            subkey_2: false
> +            subkey_dict:
> +                subsubkey_1: a
> +                subsubkey_2: 'new value'
> +
> +
> +Image Config Structure
> +----------------------
> +Image configuration is handled in ``releases.yaml``. The image configuration
> +controls how platforms locate and acquire images, how the platforms should
> +interact with the images, how platforms should detect when an image has fully
> +booted, any options that are required to set the image up, and features that
> +the image supports.
> +
> +Since settings for locating an image and interacting with it differ from
> +platform to platform, there are 4 levels of settings available for images on
> +top of the default image settings. The structure of the image config file is:
> +
> +.. code-block:: yaml
> +
> +    default_release_config:
> +        default:
> +            ...
> +        <platform>:
> +            ...
> +        <platform>:
> +            ...
> +
> +    releases:
> +        <release name>:
> +            <default>:
> +                ...
> +            <platform>:
> +                ...
> +            <platform>:
> +                ...
> +
> +
> +The base config is created from the overall defaults and the overrides for the
> +platform. The overrides are created from the default config for the image and
> +the platform specific overrides for the image.
> +
> +Image Config for System Boot
> +----------------------------
> +The test suite must be able to test if a system has fully booted and if
> +cloud-init has finished running, so that running collect scripts does not race
> +against the target image booting. This is done using the
> +``system_ready_script`` and ``cloud_init_ready_script`` image config keys.
> +
> +Each of these keys accepts a small bash test statement as a string that must
> +return 0 or 1. Since this test statement will be added into a larger bash
> +statement it must be a single statement using the ``[`` test syntax.
> +
> +The default image config provides a system ready script that works for any
> +systemd based image. If the iamge is not systmed based, then a different test

s/iamge/image/ 
s/systmed/systemd/

> +statement must be provided. The default config also provides a test for whether
> +or not cloud-init has finished which checks for the file
> +``/run/cloud-init/result.json``. This should be sufficient for most systems, as
> +writing to this file is one of the last things cloud-init does.
> +
> +The setting ``boot_timeout`` controls how long, in seconds, the platform should
> +wait for an image to boot. If the system ready script has not indicated that
> +the system is fully booted within this time an error will be raised.
> +
> +Image Config Feature Flags
> +--------------------------
> +Not all testcases can work on all images due to features the testcase requires
> +not being present on that image. If a testcase requires features in an image
> +that are not likely to be present across all distros and platforms that the
> +test suite supports, then the test can be skipped everywhere it is not
> +supported.
> +
> +This is done through feature flags, which are names for features supported on
> +some images but not all that may be required by testcases. Configuration for
> +feature flags is provided in ``releases.yaml`` under the ``features`` top level
> +key. The features config includes a list of all currently defined feature flags
> +and their meanings, and a list of feature groups.
> +
> +Feature groups are groups of features that many images have in common. For
> +example, the ``ubuntu_specific`` feature group includes features that should be
> +present across most ubuntu releases, but may or may not be for other distros.
> +Feature groups are specified for an image as a list under the key
> +``feature_groups``.
> +
> +An image's feature flags are derived from the features groups that that image
> +has and any feature overrides provided. Feature overrides can be specified
> +under the ``features`` key which accepts a dictionary of
> +``{<feature_name>: true/false}`` mappings. If a feature is omitted from an
> +image's feature flags or set to false in the overrides then the test suite will
> +skip any tests that require that feature when using that image.
> +
> +Feature flags may be overridden at runtime using the ``--feature-override``
> +command line argument. It accepts a feature flag and value to set in the format
> +``<feature name>: true/false``. Multiple ``--feature-override`` flags can be
> +used, and will all be applied to all feature flags for images used during a
> +test.
> +
> +Image Config Setup Overrides
> +----------------------------
> +If an image requires some of the options for image setup to be used, then it
> +may specify overrides for the command line arguments passed into setup image.
> +These may be specified as a dictionary under the ``setup_overrides`` key. When
> +an image is set up, the arguments that control how it is set up will be the
> +arguments from the command line, with any entries in ``setup_overrides`` used
> +to override these arguments.
> +
> +For example, images that do not come with cloud-init already installed should
> +have ``setup_overrides: {upgrade: true}`` specified so that in the event that
> +no additional setup options are given, cloud-init will be installed from the
> +image's repos before running tests. Note that if other options such as
> +``--deb`` are passed in on the command line, these will still work as expected,
> +since apt's policy for cloud-init would prefer the locally installed deb over
> +an older version from the repos.
> +
> +Image Config Platform Specific Options
> +--------------------------------------
> +There are many platform specific options in image configuration that allow
> +platforms to locate images and that control additional setup that the platform
> +may have to do to make the image useable. For information on how these work,
> +please consult the documentation for that platform in the integration testing
> +suite and the ``releases.yaml`` file for examples.
> +
> +Error Handling Behavior
> +=======================
> +
> +The test suite makes an attempt to run as many tests as possible even in the
> +event of some failing so that automated runs collect as much data as possible.
> +In the event that something goes wrong while setting up for or running a test,
> +the test suite will attempt to continue running any tests which have not been
> +effected by the error.
> +
> +For example, if the test suite was told to run tests on one platform for two
> +releases and an error occured setting up the first image, all tests for that
> +image would be skipped, and the test suite would continue to set up the second
> +image and run tests on it. Or, if the system does not start properly for one
> +testcase out of many to run on that image, that testcase will be skipped and
> +the next one will be run.
> +
> +Note that if any errors at all occur, the test suite will record the failure

s/at all//

> +and where it occurred in the result data and write it out to the specified
> +result file.
> +
> +Exit Codes
> +----------
> +The test suite counts how many errors occur throughout a run. The exit code
> +after a run is the number of errors that occured. If the exit code is non-zero
> +than something is wrong either with the test suite, the configuration for an
> +image, a testcase, or cloud-init itself.
> +
> +Note that the exit code does not always direclty correspond to the number
> +of failed testcases, since in some cases, a single error during image setup
> +can mean that several testcases are not run. If run is used, then the exit code
> +will be the sum of the number of errors in the collect and verify stages.
> +
> +Result Data
> +-----------
> +The test suite generates result data that includes how long each stage of the
> +test suite took and which parts were and were not successful. This data is
> +dumped to the log after the collect and verify stages, and may also be written
> +out in yaml format to a file. If part of the setup failed, the traceback for
> +the failure and the error message will be included in the result file. If a
> +test verifier finds a problem with the collected data from a test run, the
> +class, test function and test will be recorded in the result data.
> +
> +Data Dir
> +--------
> +When using run, the collected data is written into a temporary directory. In
> +the even that all tests pass, this directory is deleted. In the even that a
> +test fails or an error occurs, this data will be left in place, and a message
> +will be written to the log giving the location of the data.
>  
>  Architecture
>  ============
>  
> -The following outlines the process flow during a complete end-to-end LXD-backed test.
> +The following outlines the process flow during a complete end-to-end
> +LXD-backed test.
>  
>  1. Configuration
>      * The back end and specific OS releases are verified as supported
> -    * The test or tests that need to be run are determined either by directory or by individual yaml
> +    * The test or tests that need to be run are determined either by
> +      directory or by individual yaml
>  
>  2. Image Creation
>      * Acquire the daily LXD image


-- 
https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/323588
Your team cloud init development team is requested to review the proposed merge of ~powersj/cloud-init:integration-test-revamp into cloud-init:master.