yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1761536] Re: Nova compute manager failed to create virtual interface

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: David Ames <david.ames@xxxxxxxxxxxxx>
Date: Wed, 25 Apr 2018 18:15:40 -0000
Reply-to: Bug 1761536 <1761536@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Root cause of the instance boot failures is app armor on the neutron-
gateway blocking neutron agents from creating temporary directories:

1vlRG/" pid=1412869 comm="neutron-dhcp-ag" requested_mask="c" denied_mask="c" fsuid=115 ouid=115
[76035.437502] audit: type=1400 audit(1524677252.781:36019): apparmor="DENIED" operation="mkdir" profile="/usr/bin/neutron-dhcp-agent" name="/tmp/tmp4AIVtB/" pid=1412869 comm="neutron-dhcp-ag" requested_mask="c" denied_mask="c" fsuid=115 ouid=115

Both the dhcp-agent and the l3-agent both show the problem.

Assigning this bug to neutron-gateway for the app armor bug


A secondary issue that is as yet not root caused is DBConnection errors
from all the API charms connecting to percona cluster. After changing
the neutron-gateway aa-profile-mode to complain we saw these errors much
less frequently but they did not go away entirely.

2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines [req-dcf87632-8b6e-4071-a336-64b1442dc7fe - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1']
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     connection.scalar(select([1]))
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 877, in scalar
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     return self.execute(object, *multiparams, **params).scalar()
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     return meth(self, multiparams, params)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     return connection._execute_clauseelement(self, multiparams, params)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     compiled_sql, distilled_params
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     context)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1398, in _handle_dbapi_exception
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     util.raise_from_cause(newraise, exc_info)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     context)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     cursor.execute(statement, parameters)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 165, in execute
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     result = self._query(query)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 321, in _query
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 860, in query
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     result.read()
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1349, in read
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     first_packet = self.connection._read_packet()
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 991, in _read_packet
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     packet_header = self._read_bytes(4)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1037, in _read_bytes
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1']
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines 


Next steps:
Please redeploy with neutron-gatway aa-profile-mode=complain while we fix the apparmor profile bug. The current deploy has been tainted by all of our debugging attempts. Though we are able to launch instances it no longer represents a valid test.

** Changed in: charm-neutron-gateway
       Status: Invalid => Triaged

** Changed in: charm-neutron-gateway
   Importance: Undecided => Critical

** Changed in: charm-neutron-gateway
     Assignee: (unassigned) => David Ames (thedac)

** Changed in: charm-neutron-gateway
    Milestone: None => 18.05

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1761536

Title:
  Nova compute manager failed to create virtual interface

Status in OpenStack neutron-gateway charm:
  Triaged
Status in OpenStack neutron-openvswitch charm:
  New
Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Rally test scenario: NovaServers.boot_server_associate_and_dissociate_floating_ip fails.
  All 5 nova-compute-kvm instances timeout:

  --------------------------------------------------------------------------------
  Task 2ccf3cf6-c252-4e0f-8fdd-ca58ad819aff has 5 error(s)
  --------------------------------------------------------------------------------

  TimeoutException: Rally tired waiting 300.00 seconds for Server
  s_rally_504bd98b_fLz3akho:23cbd6ad-67f7-4e0f-9095-390f50897b62 to
  become ('ACTIVE') current status BUILD

  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line 71, in _run_scenario_once
      getattr(scenario_inst, method_name)(**scenario_kwargs)
    File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/servers.py", line 1116, in run
      server = self._boot_server(image, flavor, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/rally/task/atomic.py", line 91, in func_atomic_actions
      f = func(self, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 86, in _boot_server
      check_interval=CONF.openstack.nova_server_boot_poll_interval
    File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 252, in wait_for_status
      timeout=timeout)
  TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_504bd98b_fLz3akho:23cbd6ad-67f7-4e0f-9095-390f50897b62 to become ('ACTIVE') current status BUILD

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-gateway/+bug/1761536/+subscriptions
References

[Bug 1761536] [NEW] Nova compute manager failed to create virtual interface
From: Eric Vasquez, 2018-04-05