yahoo-eng-team team mailing list archive

[sean mooney <sean.k.mooney@xxxxxxxxx>]

Public bug reported:

Description
===========
This is related to a downstream bug first reported here:https://bugzilla.redhat.com/show_bug.cgi?id=1630069 where in it was discovers that due to how
triplo currently deploys the placemetn api under mod_wsgi

when deployed under mod_wsgi if the wsgi application exits with an error it is reloaded
back into the same python interperter instance. As a result of this behavior module level 
variable have a longer lifetime then the lifetime of the application. when run under uwsgi
when the application is reloaded it is reloaded in a new python interperter meaning 
the lifetime of the module level variables is scoped to the life time of the application.

As a result of the life time semantics of mod_wsgi the placment api and nova-api
must assume that the wsgi applications init can be invoked multiple times on failure.
The current use of the sqlalcamey enginefacade transaction_context is not reentrant
resulting in a type error being raised on subsequet calls to configure when the wsgi
application is reloaded. 


Expected result
===============
it should be possible to reload the nova and placement api wsgi application
under mod_wsgi on failure 

Actual result
=============

46087 [Wed Oct 10 15:10:49.433284 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Target WSGI script '/var/www/cgi-bin/nova/nova-placement-api' cannot be loaded as Python module.
46088 [Wed Oct 10 15:10:49.433305 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Exception occurred processing WSGI script '/var/www/cgi-bin/nova/nova-placement-api'.
46089 [Wed Oct 10 15:10:49.433320 2018] [:error] [pid 14] [remote 172.25.0.10:208] Traceback (most recent call last):
46090 [Wed Oct 10 15:10:49.433333 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/var/www/cgi-bin/nova/nova-placement-api", line 54, in <module>
46091 [Wed Oct 10 15:10:49.433354 2018] [:error] [pid 14] [remote 172.25.0.10:208]     application = init_application()
46092 [Wed Oct 10 15:10:49.433361 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi.py", line 108, in init_application
46093 [Wed Oct 10 15:10:49.433386 2018] [:error] [pid 14] [remote 172.25.0.10:208]     db_api.configure(conf.CONF)
46094 [Wed Oct 10 15:10:49.433392 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/db_api.py", line 35, in configure
46095 [Wed Oct 10 15:10:49.433403 2018] [:error] [pid 14] [remote 172.25.0.10:208]     **_get_db_conf(conf.placement_database))
46096 [Wed Oct 10 15:10:49.433408 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 788, in configure
46097 [Wed Oct 10 15:10:49.433420 2018] [:error] [pid 14] [remote 172.25.0.10:208]     self._factory.configure(**kw)
46098 [Wed Oct 10 15:10:49.433425 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
46099 [Wed Oct 10 15:10:49.433435 2018] [:error] [pid 14] [remote 172.25.0.10:208]     return wrapped(*args, **kwargs)
46100 [Wed Oct 10 15:10:49.433440 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 312, in configure
46101 [Wed Oct 10 15:10:49.433449 2018] [:error] [pid 14] [remote 172.25.0.10:208]     self._configure(False, kw)
46102 [Wed Oct 10 15:10:49.433453 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _configure
46103 [Wed Oct 10 15:10:49.433462 2018] [:error] [pid 14] [remote 172.25.0.10:208]     raise TypeError("this TransactionFactory is already started")
46104 [Wed Oct 10 15:10:49.433473 2018] [:error] [pid 14] [remote 172.25.0.10:208] TypeError: this TransactionFactory is already started

** Affects: nova
     Importance: Medium
     Assignee: sean mooney (sean-k-mooney)
         Status: In Progress


** Tags: api placement

** Changed in: nova
     Assignee: (unassigned) => sean mooney (sean-k-mooney)

** Changed in: nova
       Status: New => In Progress

** Changed in: nova
   Importance: Undecided => Medium

** Tags added: api placement

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1799246

Title:
  module level init of db transation contexts cause failure under
  mod_uwsgi on module reload

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  This is related to a downstream bug first reported here:https://bugzilla.redhat.com/show_bug.cgi?id=1630069 where in it was discovers that due to how
  triplo currently deploys the placemetn api under mod_wsgi

  when deployed under mod_wsgi if the wsgi application exits with an error it is reloaded
  back into the same python interperter instance. As a result of this behavior module level 
  variable have a longer lifetime then the lifetime of the application. when run under uwsgi
  when the application is reloaded it is reloaded in a new python interperter meaning 
  the lifetime of the module level variables is scoped to the life time of the application.

  As a result of the life time semantics of mod_wsgi the placment api and nova-api
  must assume that the wsgi applications init can be invoked multiple times on failure.
  The current use of the sqlalcamey enginefacade transaction_context is not reentrant
  resulting in a type error being raised on subsequet calls to configure when the wsgi
  application is reloaded. 

  
  Expected result
  ===============
  it should be possible to reload the nova and placement api wsgi application
  under mod_wsgi on failure 

  Actual result
  =============

  46087 [Wed Oct 10 15:10:49.433284 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Target WSGI script '/var/www/cgi-bin/nova/nova-placement-api' cannot be loaded as Python module.
  46088 [Wed Oct 10 15:10:49.433305 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Exception occurred processing WSGI script '/var/www/cgi-bin/nova/nova-placement-api'.
  46089 [Wed Oct 10 15:10:49.433320 2018] [:error] [pid 14] [remote 172.25.0.10:208] Traceback (most recent call last):
  46090 [Wed Oct 10 15:10:49.433333 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/var/www/cgi-bin/nova/nova-placement-api", line 54, in <module>
  46091 [Wed Oct 10 15:10:49.433354 2018] [:error] [pid 14] [remote 172.25.0.10:208]     application = init_application()
  46092 [Wed Oct 10 15:10:49.433361 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi.py", line 108, in init_application
  46093 [Wed Oct 10 15:10:49.433386 2018] [:error] [pid 14] [remote 172.25.0.10:208]     db_api.configure(conf.CONF)
  46094 [Wed Oct 10 15:10:49.433392 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/db_api.py", line 35, in configure
  46095 [Wed Oct 10 15:10:49.433403 2018] [:error] [pid 14] [remote 172.25.0.10:208]     **_get_db_conf(conf.placement_database))
  46096 [Wed Oct 10 15:10:49.433408 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 788, in configure
  46097 [Wed Oct 10 15:10:49.433420 2018] [:error] [pid 14] [remote 172.25.0.10:208]     self._factory.configure(**kw)
  46098 [Wed Oct 10 15:10:49.433425 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
  46099 [Wed Oct 10 15:10:49.433435 2018] [:error] [pid 14] [remote 172.25.0.10:208]     return wrapped(*args, **kwargs)
  46100 [Wed Oct 10 15:10:49.433440 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 312, in configure
  46101 [Wed Oct 10 15:10:49.433449 2018] [:error] [pid 14] [remote 172.25.0.10:208]     self._configure(False, kw)
  46102 [Wed Oct 10 15:10:49.433453 2018] [:error] [pid 14] [remote 172.25.0.10:208]   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _configure
  46103 [Wed Oct 10 15:10:49.433462 2018] [:error] [pid 14] [remote 172.25.0.10:208]     raise TypeError("this TransactionFactory is already started")
  46104 [Wed Oct 10 15:10:49.433473 2018] [:error] [pid 14] [remote 172.25.0.10:208] TypeError: this TransactionFactory is already started

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1799246/+subscriptions