yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

Public bug reported:

This is a truly bizarre thing going on where when apache is stopped,
then started to take the new Keystone code in grenade, it fails to start
on the other side with the following error:

2015-06-19 13:35:56.889309 ArgsAlreadyParsedError: arguments already
parsed: cannot register CLI option

This prevents all the rest of the job from completing because there is
no Keystone.

This is made more bizzare by the fact that it only seems to express on
neutron grenade jobs. It's a race, it doesn't happen all the time, but
the impact is pretty problematic.

Here is the stacktrace:

2015-06-19 13:35:56.888353 mod_wsgi (pid=28797): Target WSGI script '/var/www/keystone/admin' cannot be loaded as Python module.
2015-06-19 13:35:56.888369 mod_wsgi (pid=28797): Exception occurred processing WSGI script '/var/www/keystone/admin'.
2015-06-19 13:35:56.888387 Traceback (most recent call last):
2015-06-19 13:35:56.888402   File "/var/www/keystone/admin", line 25, in <module>
2015-06-19 13:35:56.888427     application = wsgi_server.initialize_application(name)
2015-06-19 13:35:56.888436   File "/opt/stack/new/keystone/keystone/server/wsgi.py", line 38, in initialize_application
2015-06-19 13:35:56.888450     common.configure()
2015-06-19 13:35:56.888458   File "/opt/stack/new/keystone/keystone/server/common.py", line 28, in configure
2015-06-19 13:35:56.888470     config.configure()
2015-06-19 13:35:56.888476   File "/opt/stack/new/keystone/keystone/common/config.py", line 1107, in configure
2015-06-19 13:35:56.888813     help='Do not monkey-patch threading system modules.'))
2015-06-19 13:35:56.888827   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1820, in __inner
2015-06-19 13:35:56.889264     list of default config files, or a list from find_config_files().
2015-06-19 13:35:56.889275   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1993, in register_cli_opt
2015-06-19 13:35:56.889289     
2015-06-19 13:35:56.889309 ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option

And here is a linked log for a failed job:
http://logs.openstack.org/19/193519/2/check/check-grenade-dsvm-
neutron/e0a1dbe/logs/apache/keystone.txt.gz#_2015-06-19_13_35_56_888353

** Affects: keystone
     Importance: Critical
         Status: New

** Description changed:

  This is a truly bizarre thing going on where when apache is stopped,
  then started to take the new Keystone code in grenade, it fails to start
  on the other side with the following error:
  
  2015-06-19 13:35:56.889309 ArgsAlreadyParsedError: arguments already
  parsed: cannot register CLI option
  
  This prevents all the rest of the job from completing because there is
  no Keystone.
  
  This is made more bizzare by the fact that it only seems to express on
  neutron grenade jobs. It's a race, it doesn't happen all the time, but
  the impact is pretty problematic.
+ 
+ Here is the stacktrace:
+ 
+ 2015-06-19 13:35:56.888353 mod_wsgi (pid=28797): Target WSGI script '/var/www/keystone/admin' cannot be loaded as Python module.
+ 2015-06-19 13:35:56.888369 mod_wsgi (pid=28797): Exception occurred processing WSGI script '/var/www/keystone/admin'.
+ 2015-06-19 13:35:56.888387 Traceback (most recent call last):
+ 2015-06-19 13:35:56.888402   File "/var/www/keystone/admin", line 25, in <module>
+ 2015-06-19 13:35:56.888427     application = wsgi_server.initialize_application(name)
+ 2015-06-19 13:35:56.888436   File "/opt/stack/new/keystone/keystone/server/wsgi.py", line 38, in initialize_application
+ 2015-06-19 13:35:56.888450     common.configure()
+ 2015-06-19 13:35:56.888458   File "/opt/stack/new/keystone/keystone/server/common.py", line 28, in configure
+ 2015-06-19 13:35:56.888470     config.configure()
+ 2015-06-19 13:35:56.888476   File "/opt/stack/new/keystone/keystone/common/config.py", line 1107, in configure
+ 2015-06-19 13:35:56.888813     help='Do not monkey-patch threading system modules.'))
+ 2015-06-19 13:35:56.888827   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1820, in __inner
+ 2015-06-19 13:35:56.889264     list of default config files, or a list from find_config_files().
+ 2015-06-19 13:35:56.889275   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1993, in register_cli_opt
+ 2015-06-19 13:35:56.889289     
+ 2015-06-19 13:35:56.889309 ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
+ 
+ And here is a linked log for a failed job:
+ http://logs.openstack.org/19/193519/2/check/check-grenade-dsvm-
+ neutron/e0a1dbe/logs/apache/keystone.txt.gz#_2015-06-19_13_35_56_888353

** Changed in: keystone
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1466893

Title:
  Keystone wsgi will not start after upgrade on neutron grenade jobs
  some times

Status in OpenStack Identity (Keystone):
  New

Bug description:
  This is a truly bizarre thing going on where when apache is stopped,
  then started to take the new Keystone code in grenade, it fails to
  start on the other side with the following error:

  2015-06-19 13:35:56.889309 ArgsAlreadyParsedError: arguments already
  parsed: cannot register CLI option

  This prevents all the rest of the job from completing because there is
  no Keystone.

  This is made more bizzare by the fact that it only seems to express on
  neutron grenade jobs. It's a race, it doesn't happen all the time, but
  the impact is pretty problematic.

  Here is the stacktrace:

  2015-06-19 13:35:56.888353 mod_wsgi (pid=28797): Target WSGI script '/var/www/keystone/admin' cannot be loaded as Python module.
  2015-06-19 13:35:56.888369 mod_wsgi (pid=28797): Exception occurred processing WSGI script '/var/www/keystone/admin'.
  2015-06-19 13:35:56.888387 Traceback (most recent call last):
  2015-06-19 13:35:56.888402   File "/var/www/keystone/admin", line 25, in <module>
  2015-06-19 13:35:56.888427     application = wsgi_server.initialize_application(name)
  2015-06-19 13:35:56.888436   File "/opt/stack/new/keystone/keystone/server/wsgi.py", line 38, in initialize_application
  2015-06-19 13:35:56.888450     common.configure()
  2015-06-19 13:35:56.888458   File "/opt/stack/new/keystone/keystone/server/common.py", line 28, in configure
  2015-06-19 13:35:56.888470     config.configure()
  2015-06-19 13:35:56.888476   File "/opt/stack/new/keystone/keystone/common/config.py", line 1107, in configure
  2015-06-19 13:35:56.888813     help='Do not monkey-patch threading system modules.'))
  2015-06-19 13:35:56.888827   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1820, in __inner
  2015-06-19 13:35:56.889264     list of default config files, or a list from find_config_files().
  2015-06-19 13:35:56.889275   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1993, in register_cli_opt
  2015-06-19 13:35:56.889289     
  2015-06-19 13:35:56.889309 ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option

  And here is a linked log for a failed job:
  http://logs.openstack.org/19/193519/2/check/check-grenade-dsvm-
  neutron/e0a1dbe/logs/apache/keystone.txt.gz#_2015-06-19_13_35_56_888353

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1466893/+subscriptions