yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1864418] Re: has wrong with use apache to start neutron api in docker container

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Radosław Piliszek <1864418@xxxxxxxxxxxxxxxxxx>
Date: Wed, 26 Feb 2020 08:26:10 -0000
Reply-to: Bug 1864418 <1864418@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
yj.bai is working on that for kolla, it is not a bug in the current
state of kolla.

Indeed it is weird that the 4th request fails. Could it be the case that WSGI pool gets poisoned by the first three requests?
I believe this request is about neutron having difficulty being run behind apache/mod_wsgi.

I subscribed to this report, please mention me if you need help from
kolla side.

** Changed in: kolla
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1864418

Title:
  has wrong with use apache to start neutron api  in docker container

Status in kolla:
  Invalid
Status in neutron:
  New

Bug description:
  I started the neutron api with apache in the docker container use kolla.There was no problem with the first three requests (use openstack networking list) after startup. but the fourth started to be problematic.
  I started the RPC service:
  # /usr/bin/neutron-rpc-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini

  I refer to this article :
  https://docs.openstack.org/neutron/latest/admin/config-wsgi.html

  /etc/httpd/conf.d/wsgi-neutron.conf

  Listen [fd00:1001::101]:9696

  <Directory "/var/lib/kolla/venv/bin">
      <FilesMatch "neutron-api">
          AllowOverride None
          Options None
          Require all granted
      </FilesMatch>
  </Directory>

  <VirtualHost *:9696>
      WSGIDaemonProcess neutron-server processes=1 threads=1 user=neutron group=neutron display-name=%{GROUP}
      WSGIProcessGroup neutron-server
      WSGIScriptAlias / /var/lib/kolla/venv/bin/neutron-api
      WSGIApplicationGroup %{GLOBAL}
      WSGIPassAuthorization On
      <IfVersion >= 2.4>
        ErrorLogFormat "%{cu}t %M"
      </IfVersion>
      ErrorLog "/var/log/neutron/neutron-error.log"
      LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
      CustomLog "/var/log/neutron/neutron-error.log" logformat
  </VirtualHost>

  Alias /networking /var/lib/kolla/venv/bin/neutron-api
  <Location /networking>
      SetHandler wsgi-script
      Options +ExecCGI
      WSGIProcessGroup neutron-server
      WSGIApplicationGroup %{GLOBAL}
      WSGIPassAuthorization On
  </Location>

  WSGISocketPrefix /var/run/apache2

  The following problem occurs after the fourth request is sent

  var/log/kolla/neutron/neutron-api.log

  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors [req-dc4bf26f-7078-4857-978e-60bffa96df16 51a6efff6aa34b44926fcb95c5a01b3b e474c27f32bf462d87eecb392f373424 - default default] An error occurred during processing the request: GET /v2.0/networks HTTP/1.1^M
  Accept: application/json^M
  Accept-Encoding: gzip, deflate^M
  Connection: keep-alive^M
  Host: [fd00:1001::111]:9696^M
  User-Agent: openstacksdk/0.38.0 keystoneauth1/3.18.0 python-requests/2.22.0 CPython/2.7.5^M
  X-Auth-Token: *****
  X-Forwarded-For: fd00:1001::111^M
  X-Forwarded-Proto: https: AttributeError: 'module' object has no attribute 'poll'
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     response = req.get_response(self.application)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1314, in send
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1278, in call_application
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/osprofiler/web.py", line 112, in __call__
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return request.get_response(self.application)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1314, in send
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1278, in call_application
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 338, in __call__
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     response = self.process_request(req)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 659, in process_request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp = super(AuthProtocol, self).process_request(request)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     allow_expired=allow_expired)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     data = self.fetch_token(token, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 754, in fetch_token
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     allow_expired=allow_expired)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 180, in verify_token
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     allow_expired=allow_expired)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 75, in verify_token
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     access_rules_support=ACCESS_RULES_SUPPORT)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 120, in validate
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     access_rules_support=access_rules_support)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 93, in get_token_data
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp, body = self._client.get(url, headers=headers)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 386, in get
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return self.request(url, 'GET', **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 545, in request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return self.session.request(url, method, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return self.session.request(url, method, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 888, in request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp = send(**kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 979, in _send_request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp = self.session.request(method, url, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     resp = self.send(prep, **send_kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     r = adapter.send(request, **kwargs)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/requests/adapters.py", line 449, in send
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     timeout=timeout
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 591, in urlopen
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     conn = self._get_conn(timeout=pool_timeout)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 240, in _get_conn
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     if conn and is_connection_dropped(conn):
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/util/connection.py", line 24, in is_connection_dropped
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return wait_for_read(sock, timeout=0.0)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/util/wait.py", line 143, in wait_for_read
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     return wait_for_socket(sock, read=True, timeout=timeout)
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors   File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/util/wait.py", line 95, in poll_wait_for_socket
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors     poll_obj = select.poll()
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors AttributeError: 'module' object has no attribute 'poll'
  2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors


  
  I add httpd and mod_wsgi packages to neutron server image.
  I success to use openstack network list command three times or two times.
  but form the third or fourth times this wrong appeared.

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla/+bug/1864418/+subscriptions
References

[Bug 1864418] [NEW] has wrong with use apache to start neutron api in docker container
From: yj.bai, 2020-02-24