yahoo-eng-team team mailing list archive

[Carbrich <1892902@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Fore some reason cloud-init failed to install package "SL". It receives
a sigterm before trying to install package "SL" This only happens on
CentOS 8. For all other packages I have the same issue. If I try to
install packages within user-scrips "runcmd" I have the same issue. If I
do a sleep or a ping within runcmd,  I have the same issue.

If I remove /var/lib/cloud after a failed deployment and reboot the
client, everything works fine. If i run the module with python the same
way as cloud-init does, it works without exit 1.

It almost looks like some kind of race condition.

I packaged our base image with a daily build of cloud-init, to be sure
it is not resolved in a later version of cloud-init. It has been tested
with cloud-init 20.2+137.gc0450c02-1 and the behaviour is the same on
cloud-init 18.5/19.4.

Our workflow; Image is getting build with packer and installed with all
necessary items to run cloud-init.; Image is added to vCenter as a
template; Foreman clones VM on vCenter; Machine boots, Vmware does the
network configuration as described in cloud-init docs(Cloning on
vCenter). Cloud-init installs puppet and registers client to puppet
master.; In the background the image is imported in vCloud and more
stuff happens for our monitoring etc..; For our CentOS 7 deployment
street this workflow is working without any issues.

For more reading follow the link below, sorry for the mixed up data on
the forum. I tried a lot! https://community.theforeman.org/t/cloud-init-
does-not-install-puppet-but-runs-the-final-module-and-succeeds-
according-cloud-init/20155/27

I also added cloud-init-1.tar.gz with the logs. The data is anonymized.

Your guys help will be welcome!

PS. Sometimes really sometimes it just works 1 out of 100 or so.


### Cloud-config ###
--------------------


#cloud-config
hostname: test05
fqdn: test05
manage_etc_hosts: true
users: {}
packages:
- sl
runcmd:
- touch /tmp/test

#cloud.cfg
cloud_init_modules:
 - bootcmd
 - growpart

cloud_config_modules:
 - runcmd
 - timezone
 - package-update-upgrade-install

cloud_final_modules:
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - phone-home
 - power_state_change
 - final-message

system_info:
  distro: rhel
  paths:
    cloud_dir: /var/lib/cloud
    templates_dir: /etc/cloud/templates
  ssh_svcname: sshd

# vim:syntax=yaml


### /etc/cloud/cloud.cfg.d/01_network.cfg ###
---------------------------------------------


network:
  config: disabled

### 10_foreman.cfg
cat /etc/cloud/cloud.cfg.d/10_foreman.cfg
datasource_list: [NoCloud]
datasource:
  NoCloud:
    seedfrom: http://foreman/userdata/

#
- &log_syslog |
   [handler_cloudLogHandler]
   class=handlers.SysLogHandler
   level=DEBUG
   formatter=simpleFormatter
   args=("/dev/log", handlers.SysLogHandler.LOG_USER)

### /etc/cloud/cloud.cfg.d/05_logging.cfg ###
---------------------------------------------


## This yaml formated config file handles setting
## logger information.  The values that are necessary to be set
## are seen at the bottom.  The top '_log' are only used to remove
## redundency in a syslog and fallback-to-file case.
##
## The 'log_cfgs' entry defines a list of logger configs
## Each entry in the list is tried, and the first one that
## works is used.  If a log_cfg list entry is an array, it will
## be joined with '\n'.
_log:
 - &log_base |
   [loggers]
   keys=root,cloudinit

   [handlers]
   keys=consoleHandler,cloudLogHandler

   [formatters]
   keys=simpleFormatter,arg0Formatter

   [logger_root]
   level=DEBUG
   handlers=consoleHandler,cloudLogHandler

   [logger_cloudinit]
   level=DEBUG
   qualname=cloudinit
   handlers=
   propagate=1

   [handler_consoleHandler]
   class=StreamHandler
   level=WARNING
   formatter=arg0Formatter
   args=(sys.stderr,)

   [formatter_arg0Formatter]
   format=%(asctime)s - %(filename)s[%(levelname)s]: %(message)s

   [formatter_simpleFormatter]
   format=[CLOUDINIT] %(filename)s[%(levelname)s]: %(message)s
 - &log_file |
   [handler_cloudLogHandler]
   class=FileHandler
   level=DEBUG
   formatter=arg0Formatter
   args=('/var/log/cloud-init.log', 'a', 'UTF-8')
 - &log_syslog |
   [handler_cloudLogHandler]
   class=handlers.SysLogHandler
   level=DEBUG
   formatter=simpleFormatter
   args=("/dev/log", handlers.SysLogHandler.LOG_USER)

log_cfgs:
# Array entries in this list will be joined into a string
# that defines the configuration.
#
# If you want logs to go to syslog, uncomment the following line.
# - [ *log_base, *log_syslog ]
#
# The default behavior is to just log to a file.
# This mechanism that does not depend on a system service to operate.
 - [ *log_base, *log_file ]
# A file path can also be used.
# - /etc/log.conf

# This tells cloud-init to redirect its stdout and stderr to
# 'tee -a /var/log/cloud-init-output.log' so the user can see output
# there without needing to look on the console.
output: {all: '| tee -a /var/log/cloud-init-output.log'}

** Affects: cloud-init
     Importance: Undecided
         Status: New

** Attachment added: "cloud init logs"
   https://bugs.launchpad.net/bugs/1892902/+attachment/5404514/+files/cloud-init-1.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1892902

Title:
  Cloud-init  received SIGTERM and is thereby unable to install packages
  or running runcmd completely

Status in cloud-init:
  New

Bug description:
  Fore some reason cloud-init failed to install package "SL". It
  receives a sigterm before trying to install package "SL" This only
  happens on CentOS 8. For all other packages I have the same issue. If
  I try to install packages within user-scrips "runcmd" I have the same
  issue. If I do a sleep or a ping within runcmd,  I have the same
  issue.

  If I remove /var/lib/cloud after a failed deployment and reboot the
  client, everything works fine. If i run the module with python the
  same way as cloud-init does, it works without exit 1.

  It almost looks like some kind of race condition.

  I packaged our base image with a daily build of cloud-init, to be sure
  it is not resolved in a later version of cloud-init. It has been
  tested with cloud-init 20.2+137.gc0450c02-1 and the behaviour is the
  same on cloud-init 18.5/19.4.

  Our workflow; Image is getting build with packer and installed with
  all necessary items to run cloud-init.; Image is added to vCenter as a
  template; Foreman clones VM on vCenter; Machine boots, Vmware does the
  network configuration as described in cloud-init docs(Cloning on
  vCenter). Cloud-init installs puppet and registers client to puppet
  master.; In the background the image is imported in vCloud and more
  stuff happens for our monitoring etc..; For our CentOS 7 deployment
  street this workflow is working without any issues.

  For more reading follow the link below, sorry for the mixed up data on
  the forum. I tried a lot! https://community.theforeman.org/t/cloud-
  init-does-not-install-puppet-but-runs-the-final-module-and-succeeds-
  according-cloud-init/20155/27

  I also added cloud-init-1.tar.gz with the logs. The data is
  anonymized.

  Your guys help will be welcome!

  PS. Sometimes really sometimes it just works 1 out of 100 or so.

  
  ### Cloud-config ###
  --------------------

  
  #cloud-config
  hostname: test05
  fqdn: test05
  manage_etc_hosts: true
  users: {}
  packages:
  - sl
  runcmd:
  - touch /tmp/test

  #cloud.cfg
  cloud_init_modules:
   - bootcmd
   - growpart

  cloud_config_modules:
   - runcmd
   - timezone
   - package-update-upgrade-install

  cloud_final_modules:
   - scripts-per-once
   - scripts-per-boot
   - scripts-per-instance
   - scripts-user
   - phone-home
   - power_state_change
   - final-message

  system_info:
    distro: rhel
    paths:
      cloud_dir: /var/lib/cloud
      templates_dir: /etc/cloud/templates
    ssh_svcname: sshd

  # vim:syntax=yaml


  ### /etc/cloud/cloud.cfg.d/01_network.cfg ###
  ---------------------------------------------

  
  network:
    config: disabled

  ### 10_foreman.cfg
  cat /etc/cloud/cloud.cfg.d/10_foreman.cfg
  datasource_list: [NoCloud]
  datasource:
    NoCloud:
      seedfrom: http://foreman/userdata/

  #
  - &log_syslog |
     [handler_cloudLogHandler]
     class=handlers.SysLogHandler
     level=DEBUG
     formatter=simpleFormatter
     args=("/dev/log", handlers.SysLogHandler.LOG_USER)

  ### /etc/cloud/cloud.cfg.d/05_logging.cfg ###
  ---------------------------------------------

  
  ## This yaml formated config file handles setting
  ## logger information.  The values that are necessary to be set
  ## are seen at the bottom.  The top '_log' are only used to remove
  ## redundency in a syslog and fallback-to-file case.
  ##
  ## The 'log_cfgs' entry defines a list of logger configs
  ## Each entry in the list is tried, and the first one that
  ## works is used.  If a log_cfg list entry is an array, it will
  ## be joined with '\n'.
  _log:
   - &log_base |
     [loggers]
     keys=root,cloudinit

     [handlers]
     keys=consoleHandler,cloudLogHandler

     [formatters]
     keys=simpleFormatter,arg0Formatter

     [logger_root]
     level=DEBUG
     handlers=consoleHandler,cloudLogHandler

     [logger_cloudinit]
     level=DEBUG
     qualname=cloudinit
     handlers=
     propagate=1

     [handler_consoleHandler]
     class=StreamHandler
     level=WARNING
     formatter=arg0Formatter
     args=(sys.stderr,)

     [formatter_arg0Formatter]
     format=%(asctime)s - %(filename)s[%(levelname)s]: %(message)s

     [formatter_simpleFormatter]
     format=[CLOUDINIT] %(filename)s[%(levelname)s]: %(message)s
   - &log_file |
     [handler_cloudLogHandler]
     class=FileHandler
     level=DEBUG
     formatter=arg0Formatter
     args=('/var/log/cloud-init.log', 'a', 'UTF-8')
   - &log_syslog |
     [handler_cloudLogHandler]
     class=handlers.SysLogHandler
     level=DEBUG
     formatter=simpleFormatter
     args=("/dev/log", handlers.SysLogHandler.LOG_USER)

  log_cfgs:
  # Array entries in this list will be joined into a string
  # that defines the configuration.
  #
  # If you want logs to go to syslog, uncomment the following line.
  # - [ *log_base, *log_syslog ]
  #
  # The default behavior is to just log to a file.
  # This mechanism that does not depend on a system service to operate.
   - [ *log_base, *log_file ]
  # A file path can also be used.
  # - /etc/log.conf

  # This tells cloud-init to redirect its stdout and stderr to
  # 'tee -a /var/log/cloud-init-output.log' so the user can see output
  # there without needing to look on the console.
  output: {all: '| tee -a /var/log/cloud-init-output.log'}

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1892902/+subscriptions