cloud-init team mailing list archive

[Scott Moser <smoser@xxxxxxxxxx>]

On Wed, 21 Dec 2016, Michael Felt wrote:

> On 20-Dec-16 22:57, Scott Moser wrote:
> > Hey,
> > some inline comments below.
> >
> > I really appreciate your interest, and want to help out. 2 things though
> > a.) i'll be out after friday for 2 weeks
> > b.) converstations lkke this would be good to have on cloud-init mailing
> > list
> >     https://lists.launchpad.net/cloud-init/
> >
> > You can subscribe at https://launchpad.net/~cloud-init
> >
> > So for further discussion would you mind sending there?
> > You can introduce yourself and your goals.
> Introduction:

Welcome.

> e) also new to python - so I may ask things you would/could/might consider
> self-evident.

Please feel free to ask.  In #cloud-init on freenode or here.

> Goals:
> a) healthy discussions (see below)
> b) a port of cloud-init for AIX - and suitable for inclusion in the main
> branch (unlike the current "source" now available based on 0.7.5)

\o/


> > > e.g., I assume CFG_BUILTIN is the default(s) for config.cfg - so what is
> > > BUILTIN_CFG - new name for same thing, or (imho) just a poor choice.
> > well, the BUILTIN_CFG that you see here is per-config module "BUILTIN".  I
> > guess your're right in that consistency would be nice, but they're
> > different things.
> > there is a cloudinit "global" builtin config and then each of the config
> > modules kind of have their own.
> I suspected as much, but for the casual reader - CFG_BUILTIN and BUILTIN_CFG
> may not be clearly remembered.
>
> >
> > I see your point about wanting a centralized default, but it makes sense
> > from a different perspective for each config module to have its own.. that
> > way the config modules are more "modules".
> >
> > It seems like probably we want to get the 'distro' object involved in more
> > things.  Also, there is the Paths object which is somewhat relevant here.
> >
> >
> > > michael@x071:[/data/prj/aixtools/cloud-init/cloudinit]grep CFG *.py */*.py
> > > helpers.py:                                CFG_ENV_NAME)
> > > helpers.py:        if CFG_ENV_NAME in os.environ:
> > > helpers.py:            e_fn = os.environ[CFG_ENV_NAME]
> > > settings.py:CFG_ENV_NAME = "CLOUD_CFG"
> > > settings.py:    from cloudinit.distros.aix.settings import CFG_BUILTIN,
> > > CLOUD_CONFIG
> > > settings.py:    CFG_BUILTIN = {
> > > ssh_util.py:DEF_SSHD_CFG = "/etc/ssh/sshd_config"
> > > ssh_util.py:            ssh_cfg = parse_ssh_config_map(DEF_SSHD_CFG)
> > > ssh_util.py:                        "%r instead", DEF_SSHD_CFG,
> > > auth_key_fn)
> > > util.py:from cloudinit.settings import (CFG_BUILTIN)
> > > util.py:    return obj_copy.deepcopy(CFG_BUILTIN)
> > > config/cc_emit_upstart.py:        cmd = ['initctl', 'emit', str(n),
> > > 'CLOUD_CFG=%s' % cfgpath]
> > > config/cc_fan.py:BUILTIN_CFG = {
> > > config/cc_fan.py:    mycfg = util.mergemanydict([cfgin, BUILTIN_CFG])
> > > config/cc_landscape.py:LSC_CLIENT_CFG_FILE = "/etc/landscape/client.conf"
> > > config/cc_landscape.py:LSC_BUILTIN_CFG = {
> > > config/cc_landscape.py:        LSC_BUILTIN_CFG,
> > > config/cc_landscape.py:        LSC_CLIENT_CFG_FILE,
> > > config/cc_landscape.py:
> > > util.ensure_dir(os.path.dirname(LSC_CLIENT_CFG_FILE))
> > > config/cc_landscape.py:    util.write_file(LSC_CLIENT_CFG_FILE,
> > > contents.getvalue())
> > > config/cc_landscape.py:    log.debug("Wrote landscape config file to %s",
> > > LSC_CLIENT_CFG_FILE)
> > > config/cc_mcollective.py:SERVER_CFG = '/etc/mcollective/server.cfg'
> > > config/cc_mcollective.py:def configure(config, server_cfg=SERVER_CFG,
> > > config/cc_set_passwords.py:        old_lines =
> > > ssh_util.parse_ssh_config(ssh_util.DEF_SSHD_CFG)
> > > config/cc_set_passwords.py: util.write_file(ssh_util.DEF_SSHD_CFG,
> > > "\n".join(lines))
> > > config/cc_snappy.py:BUILTIN_CFG = {
> > > config/cc_snappy.py:    mycfg = util.mergemanydict([cfgin, BUILTIN_CFG])
> > > sources/DataSourceAzure.py:DS_CFG_PATH = ['datasource', DS_NAME]
> > > sources/DataSourceAzure.py: util.get_cfg_by_path(sys_cfg, DS_CFG_PATH,
> > > {}),
> > > sources/DataSourceAzure.py:        user_ds_cfg =
> > > util.get_cfg_by_path(self.cfg, DS_CFG_PATH, {})
> > > sources/DataSourceSmartOS.py:DS_CFG_PATH = ['datasource', DS_NAME]
> > > sources/DataSourceSmartOS.py: util.get_cfg_by_path(sys_cfg, DS_CFG_PATH,
> > > {}),
> > >
> > > ==== B ====
> > >
> > > As much as possible I want to maintain "defaults", so I was thinking
> > > something
> > > such as: (in settings.py)
> > >
> > > # This is expected to be a yaml formatted file
> > > import sys
> > > if sys.platform[:3] == "aix":
> > >      from cloudinit.distros.aix.settings import CFG_BUILTIN, CLOUD_CONFIG
> > > else:
> > >      CLOUD_CONFIG = '/etc/cloud/cloud.cfg'
> > >
> > >      # What u get if no config is provided
> > >      CFG_BUILTIN = {
> > > ...
> > I think that we probably want to get away (as much as possible) from
> > static configuration here, but rather have things "do the right thing".
> > With the ability to statically define them via config if that doesnt work.
> There will always be defaults - and there will always be differences in
> distributions. UNIX/POSIX has those differences (Solaris, HPUX, Tru-64, AIX,
> etc) as Linux also has differences (both coming from distribution (debian,
> ubuntu, redhat, suse (my first was slackware after I left minix).
> However, having them in a "single" place, or at least in a well-defined
> location would be much better.
>
> New thought:
> One of my problems, as I work though the code is knowing what something is
> supossed to be doing. One thing I miss - a lot - is __doc__ aka DocString for
> each of the files in cloud-init. Back to "default" definitions - I would look
> to something like __init__.py for definitions that are "global" to a module,
> and one of those items, imho, should be a DocString.

Agreed.  I'm definitely willing to accept docstring improvements and sane
code doc as well.

> I know I am new to python - cloudinit.config - for example, looks to be a lot
> of files that are concerned with the config phase (hence the name), but I
> suspect that there are some very big differences in the files that are using
> some CFG variable:
> michael@x071:[/data/prj/aixtools/cloud-init/cloudinit/config]grep -c CFG *.py
> | grep -v :0
> cc_emit_upstart.py:1
> cc_fan.py:2
> cc_landscape.py:7
> cc_mcollective.py:2
> cc_set_passwords.py:2
> cc_snappy.py:2
>
> iow - I can imagine how cc_set_passwords is a "generic" config operation, but
> landscape, upstart, fan (no idea what that is), snappy, mcollective (again,
> hmm, what is that) are more closely related to a systems management approach
> for a (part of) a distro. I have to read, and guess what they do, or intend to
> do (just looked into all of them, and I am eating a few of my words - they
> have a nice DocString all -
>
> so I will just come back to my "problem" - since "defaults" are not in a
> common location, nor is there a high-level description of the code structure
> (that I could find to date - I may just be blind in one eye and cannot see out
> the other - I have to read, line by line (class by class) what is in each
> file.

You're not blind, it would definitely be helpful.

> Please do not think I am trying to flame cloud-init. It is much better than
> anything I could even imagine! If anything, see me, as someone struggling to
> see what is in front of me. Maybe the problem is all (certainly largely) mine
> - I just need to study more. I assume you know the code well and maybe have
> (already expressed) ideas on how it can be improved.
>
> So, two sub-goals in this mail:
> a) to see and understand cloud-init better (think out-loud, elicit
> feedback/hints from you)
> b) to help you know where it is confusing (one person) aka FYI and I hope it
> helps!

One way to solve this would be for you to ask and document (with merge
proposals) as you get responses.  There is definitely room for improvement
in many places in cloud-init.