cloud-init team mailing list archive

[Michael Felt <michael@xxxxxxxxxxxxx>]

On 20-Dec-16 22:57, Scott Moser wrote:
> Hey,
> some inline comments below.
>
> I really appreciate your interest, and want to help out. 2 things though
> a.) i'll be out after friday for 2 weeks
> b.) converstations lkke this would be good to have on cloud-init mailing
> list
>     https://lists.launchpad.net/cloud-init/
>
> You can subscribe at https://launchpad.net/~cloud-init
>
> So for further discussion would you mind sending there?
> You can introduce yourself and your goals.
Introduction:
a) I know a lot about AIX: I host two portals: www.rootvg.net and 
www.aixtools.net. I am on github as "aixtools".
b) I try to not talk about my employer, however, I do not hide it either 
(IBM Lab Services)
c) what I do here, and all in OSS is my own time, on my own personal 
hardware (so it is older model, like me ;))
d) I am a maverick (others might say "independent thinker", but arrogant 
works too)
e) also new to python - so I may ask things you would/could/might 
consider self-evident.

Goals:
a) healthy discussions (see below)
b) a port of cloud-init for AIX - and suitable for inclusion in the main 
branch (unlike the current "source" now available based on 0.7.5)
c) learn more python
d) find more interest aka backing for a patch I wrote for (c)python for 
AIX (Lib/ctypes).

FYI: I am still typing, but my mail program says I have received it. If 
this comes in double - please forgive the new guy.
> There wont be a ton of traffic, but community involvement like this will
> help others to see that they can help out.
>
> Scott
>
> On Tue, 20 Dec 2016, Michael Felt wrote:
>
> > On 20-Dec-16 20:14, Scott Moser wrote:
> > > Looks great.
> > > I pulled, I made some minor changes to the commit message, but just some
> > > that aren't worth discussing.
> > >
> > > Thanks.
> > Two questions:
> >
> > A - regarding a more "portable", i.e., getting defaults into a single file,
> > rather than having to look everywhere for them
> >
> > B- my "initial" approach to override (forgot the proper OO term for sub-class
> > overrides)
> >
> > ==== A ====
> >
> > Something - "smallish", but goes beyond just a change in ssh_util.py, e.g.,
> > move DEF_SSHD_CFG to settings.py
> >
> > A grep two deep shows, for someone like myself - new to cloud-init - confusing
> > naming schemes:
> >
> > e.g., I assume CFG_BUILTIN is the default(s) for config.cfg - so what is
> > BUILTIN_CFG - new name for same thing, or (imho) just a poor choice.
> well, the BUILTIN_CFG that you see here is per-config module "BUILTIN".  I
> guess your're right in that consistency would be nice, but they're
> different things.
> there is a cloudinit "global" builtin config and then each of the config
> modules kind of have their own.
I suspected as much, but for the casual reader - CFG_BUILTIN and 
BUILTIN_CFG may not be clearly remembered.

> I see your point about wanting a centralized default, but it makes sense
> from a different perspective for each config module to have its own.. that
> way the config modules are more "modules".
>
> It seems like probably we want to get the 'distro' object involved in more
> things.  Also, there is the Paths object which is somewhat relevant here.
>
>
> > michael@x071:[/data/prj/aixtools/cloud-init/cloudinit]grep CFG *.py */*.py
> > helpers.py:                                CFG_ENV_NAME)
> > helpers.py:        if CFG_ENV_NAME in os.environ:
> > helpers.py:            e_fn = os.environ[CFG_ENV_NAME]
> > settings.py:CFG_ENV_NAME = "CLOUD_CFG"
> > settings.py:    from cloudinit.distros.aix.settings import CFG_BUILTIN,
> > CLOUD_CONFIG
> > settings.py:    CFG_BUILTIN = {
> > ssh_util.py:DEF_SSHD_CFG = "/etc/ssh/sshd_config"
> > ssh_util.py:            ssh_cfg = parse_ssh_config_map(DEF_SSHD_CFG)
> > ssh_util.py:                        "%r instead", DEF_SSHD_CFG, auth_key_fn)
> > util.py:from cloudinit.settings import (CFG_BUILTIN)
> > util.py:    return obj_copy.deepcopy(CFG_BUILTIN)
> > config/cc_emit_upstart.py:        cmd = ['initctl', 'emit', str(n),
> > 'CLOUD_CFG=%s' % cfgpath]
> > config/cc_fan.py:BUILTIN_CFG = {
> > config/cc_fan.py:    mycfg = util.mergemanydict([cfgin, BUILTIN_CFG])
> > config/cc_landscape.py:LSC_CLIENT_CFG_FILE = "/etc/landscape/client.conf"
> > config/cc_landscape.py:LSC_BUILTIN_CFG = {
> > config/cc_landscape.py:        LSC_BUILTIN_CFG,
> > config/cc_landscape.py:        LSC_CLIENT_CFG_FILE,
> > config/cc_landscape.py: util.ensure_dir(os.path.dirname(LSC_CLIENT_CFG_FILE))
> > config/cc_landscape.py:    util.write_file(LSC_CLIENT_CFG_FILE,
> > contents.getvalue())
> > config/cc_landscape.py:    log.debug("Wrote landscape config file to %s",
> > LSC_CLIENT_CFG_FILE)
> > config/cc_mcollective.py:SERVER_CFG = '/etc/mcollective/server.cfg'
> > config/cc_mcollective.py:def configure(config, server_cfg=SERVER_CFG,
> > config/cc_set_passwords.py:        old_lines =
> > ssh_util.parse_ssh_config(ssh_util.DEF_SSHD_CFG)
> > config/cc_set_passwords.py: util.write_file(ssh_util.DEF_SSHD_CFG,
> > "\n".join(lines))
> > config/cc_snappy.py:BUILTIN_CFG = {
> > config/cc_snappy.py:    mycfg = util.mergemanydict([cfgin, BUILTIN_CFG])
> > sources/DataSourceAzure.py:DS_CFG_PATH = ['datasource', DS_NAME]
> > sources/DataSourceAzure.py: util.get_cfg_by_path(sys_cfg, DS_CFG_PATH, {}),
> > sources/DataSourceAzure.py:        user_ds_cfg =
> > util.get_cfg_by_path(self.cfg, DS_CFG_PATH, {})
> > sources/DataSourceSmartOS.py:DS_CFG_PATH = ['datasource', DS_NAME]
> > sources/DataSourceSmartOS.py: util.get_cfg_by_path(sys_cfg, DS_CFG_PATH, {}),
> >
> > ==== B ====
> >
> > As much as possible I want to maintain "defaults", so I was thinking something
> > such as: (in settings.py)
> >
> > # This is expected to be a yaml formatted file
> > import sys
> > if sys.platform[:3] == "aix":
> >      from cloudinit.distros.aix.settings import CFG_BUILTIN, CLOUD_CONFIG
> > else:
> >      CLOUD_CONFIG = '/etc/cloud/cloud.cfg'
> >
> >      # What u get if no config is provided
> >      CFG_BUILTIN = {
> > ...
> I think that we probably want to get away (as much as possible) from
> static configuration here, but rather have things "do the right thing".
> With the ability to statically define them via config if that doesnt work.
There will always be defaults - and there will always be differences in 
distributions. UNIX/POSIX has those differences (Solaris, HPUX, Tru-64, 
AIX, etc) as Linux also has differences (both coming from distribution 
(debian, ubuntu, redhat, suse (my first was slackware after I left minix).
However, having them in a "single" place, or at least in a well-defined 
location would be much better.

New thought:
One of my problems, as I work though the code is knowing what something 
is supossed to be doing. One thing I miss - a lot - is __doc__ aka 
DocString for each of the files in cloud-init. Back to "default" 
definitions - I would look to something like __init__.py for definitions 
that are "global" to a module, and one of those items, imho, should be a 
DocString.

I know I am new to python - cloudinit.config - for example, looks to be 
a lot of files that are concerned with the config phase (hence the 
name), but I suspect that there are some very big differences in the 
files that are using some CFG variable:
michael@x071:[/data/prj/aixtools/cloud-init/cloudinit/config]grep -c CFG 
*.py | grep -v :0
cc_emit_upstart.py:1
cc_fan.py:2
cc_landscape.py:7
cc_mcollective.py:2
cc_set_passwords.py:2
cc_snappy.py:2

iow - I can imagine how cc_set_passwords is a "generic" config 
operation, but landscape, upstart, fan (no idea what that is), snappy, 
mcollective (again, hmm, what is that) are more closely related to a 
systems management approach for a (part of) a distro. I have to read, 
and guess what they do, or intend to do (just looked into all of them, 
and I am eating a few of my words - they have a nice DocString all -

so I will just come back to my "problem" - since "defaults" are not in a 
common location, nor is there a high-level description of the code 
structure (that I could find to date - I may just be blind in one eye 
and cannot see out the other - I have to read, line by line (class by 
class) what is in each file.

Please do not think I am trying to flame cloud-init. It is much better 
than anything I could even imagine! If anything, see me, as someone 
struggling to see what is in front of me. Maybe the problem is all 
(certainly largely) mine - I just need to study more. I assume you know 
the code well and maybe have (already expressed) ideas on how it can be 
improved.

So, two sub-goals in this mail:
a) to see and understand cloud-init better (think out-loud, elicit 
feedback/hints from you)
b) to help you know where it is confusing (one person) aka FYI and I 
hope it helps!

regards,
Michael
> > The reverse could be the defaults are defined, and then "from ... import ..."
> > could be used to override selective bits,
> > or perhaps just overwrite just small bits. - One that is likely to frequently
> > be an issue is:
> >
> >          },
> >          'distro': 'ubuntu',
> >      },
> >
> > Again, I am still learning the "art of python" - it is very very different to
> > how I used C 30 years ago (I feel very "old dogish"),
> > and, if I knew how - I would replace the static text "ubuntu" with a variable
> > I could establish via sys.platform and/or
> > cloudinit.distros.somethingExtra.
> >
> > Your feedback on what you see as the correct solution path is appreciated.
> >
> > Sincerely,
> >
> > Michael