yahoo-eng-team team mailing list archive

[Elod Illes <1882094@xxxxxxxxxxxxxxxxxx>]

** Changed in: nova/victoria
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1882094

Title:
  [queens][OSP13] An overcloud reboot will sometimes leave nova_api
  broken

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) train series:
  In Progress
Status in OpenStack Compute (nova) ussuri series:
  In Progress
Status in OpenStack Compute (nova) victoria series:
  Fix Released

Bug description:
  Description of problem:

  This is a composable HA overcloud with tls-everywhere with 2020-05-28.2 compose.
  We reboot the overcloud one node at the time, and from time to time (totally not consistent) we see that the nova_api containers is in an unhealthy state and returns the following in a loop:
  ```
  [Thu Jun 04 07:00:29.332162 2020] [:error] [pid 19] [remote 172.17.1.36:180] mod_wsgi (pid=19): Exception occurred processing WSGI script '/var/www/cgi-bin/nova/nova-api'.
  [Thu Jun 04 07:00:29.332185 2020] [:error] [pid 19] [remote 172.17.1.36:180] Traceback (most recent call last):
  [Thu Jun 04 07:00:29.332208 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/var/www/cgi-bin/nova/nova-api", line 54, in <module>
  [Thu Jun 04 07:00:29.332271 2020] [:error] [pid 19] [remote 172.17.1.36:180]     application = init_application()
  [Thu Jun 04 07:00:29.332286 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/wsgi.py", line 20, in init_pplication
  [Thu Jun 04 07:00:29.332317 2020] [:error] [pid 19] [remote 172.17.1.36:180]     return wsgi_app.init_application(NAME)
  [Thu Jun 04 07:00:29.332327 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi_app.py", line 78, in init_applcation
  [Thu Jun 04 07:00:29.332351 2020] [:error] [pid 19] [remote 172.17.1.36:180]     config.parse_args([], default_config_files=conf_files)
  [Thu Jun 04 07:00:29.332367 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/nova/config.py", line 35, in parse_args
  [Thu Jun 04 07:00:29.332385 2020] [:error] [pid 19] [remote 172.17.1.36:180]     log.register_options(CONF)
  [Thu Jun 04 07:00:29.332401 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 250, in register_options
  [Thu Jun 04 07:00:29.332433 2020] [:error] [pid 19] [remote 172.17.1.36:180]     conf.register_cli_opts(_options.common_cli_opts)
  [Thu Jun 04 07:00:29.332461 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2440, in __inner
  [Thu Jun 04 07:00:29.332490 2020] [:error] [pid 19] [remote 172.17.1.36:180]     result = f(self, *args, **kwargs)
  [Thu Jun 04 07:00:29.332503 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2662, in register_cli_opts
  [Thu Jun 04 07:00:29.332523 2020] [:error] [pid 19] [remote 172.17.1.36:180]     self.register_cli_opt(opt, group, clear_cache=False)
  [Thu Jun 04 07:00:29.332532 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2444, in __inner
  [Thu Jun 04 07:00:29.332550 2020] [:error] [pid 19] [remote 172.17.1.36:180]     return f(self, *args, **kwargs)
  [Thu Jun 04 07:00:29.332559 2020] [:error] [pid 19] [remote 172.17.1.36:180]   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2654, in register_cli_opt
  [Thu Jun 04 07:00:29.332577 2020] [:error] [pid 19] [remote 172.17.1.36:180]     raise ArgsAlreadyParsedError("cannot register CLI option")
  [Thu Jun 04 07:00:29.332603 2020] [:error] [pid 19] [remote 172.17.1.36:180] ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
  ```

  It look like we should always reset CONF when starting the wsgi app
  [1].

  Version-Release number of selected component (if applicable):
  openstack-nova-api-17.0.13-7.el7ost.noarch     
  puppet-nova-12.5.0-8.el7ost.noarch             
  python-nova-17.0.13-7.el7ost.noarch            
  openstack-nova-common-17.0.13-7.el7ost.noarch  
  python2-novaclient-10.1.1-1.el7ost.noarch

  [1]
  https://github.com/openstack/nova/blob/stable/queens/nova/api/openstack/wsgi_app.py#L78

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1882094/+subscriptions