yahoo-eng-team team mailing list archive

[OpenStack Infra <1541742@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/278501
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a5a7b892df46f5d48840f600f03e7be2db999a1d
Submitter: Jenkins
Branch:    master

commit a5a7b892df46f5d48840f600f03e7be2db999a1d
Author: Jakub Libosvar <libosvar@xxxxxxxxxx>
Date:   Wed Feb 10 16:35:30 2016 +0000

    fullstack: Gracefully stop neutron-server process
    
    There is possible scenario that neutron-server loses connection to AMQP
    bus and that can cause timeouts when hanging on rpc calls. On the other
    hand, we should be able to stop service gracefully even if AMQP server
    isn't running.
    
    During teardown there were still neutron-server orphaned api workers
    processes running that had open connection to database which caused
    unexpected failures.
    
    Note that this change applies only for neutron-server because previous
    investigations showed agents using oslo service can hang on rpc causing
    other failures. Next step should be making sure rabbitmq is stopped as
    last or decrease rpc timeouts in agents.
    
    Change-Id: I3d793ba924dc3fd229d8588f4be1e943614f22bb
    Related-bug: 1494363
    Closes-bug: 1541742


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1541742

Title:
  fullstack tests break when tearing down database

Status in neutron:
  Fix Released

Bug description:
  http://logs.openstack.org/41/265041/7/check/gate-neutron-dsvm-
  fullstack/8ac64cd/testr_results.html.gz

  Late runs fail with the same errors:

  Traceback (most recent call last):
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/fixture.py", line 125, in cleanUp
      return self._cleanups(raise_errors=raise_first)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/callmany.py", line 88, in __call__
      reraise(error[0], error[1], error[2])
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/callmany.py", line 82, in __call__
      cleanup(*args, **kwargs)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/testresources/__init__.py", line 797, in tearDownResources
      resource[1].finishedWith(getattr(test, resource[0]), result)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/testresources/__init__.py", line 509, in finishedWith
      self._clean_all(resource, result)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/testresources/__init__.py", line 478, in _clean_all
      self.clean(resource)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/provision.py", line 127, in clean
      resource.database.engine)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/provision.py", line 263, in drop_all_objects
      self.impl.drop_all_objects(engine)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/provision.py", line 415, in drop_all_objects
      conn.execute(schema.DropConstraint(fkc))
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
      return meth(self, multiparams, params)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
      return connection._execute_ddl(self, multiparams, params)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 968, in _execute_ddl
      compiled
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
      context)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
      util.raise_from_cause(newraise, exc_info)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
      reraise(type(exception), exception, tb=exc_tb)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
      context)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
      cursor.execute(statement, parameters)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
      result = self._query(query)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
      conn.query(q)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 819, in query
      self._affected_rows = self._read_query_result(unbuffered=unbuffered)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 1001, in _read_query_result
      result.read()
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 1285, in read
      first_packet = self.connection._read_packet()
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 945, in _read_packet
      packet_header = self._read_bytes(4)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/connections.py", line 971, in _read_bytes
      data = self._rfile.read(num_bytes)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/pymysql/_socketio.py", line 59, in readinto
      return self._sock.recv_into(b)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 353, in recv_into
      return self._recv_loop(self.fd.recv_into, buffer, nbytes, flags)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 341, in _recv_loop
      self._read_trampoline()
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 312, in _read_trampoline
      timeout_exc=socket.timeout("timed out"))
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 201, in _trampoline
      mark_as_closed=self._mark_as_closed)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline
      return hub.switch()
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
      return self.greenlet.switch()
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
      self.wait(sleep_time)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
      presult = self.do_poll(seconds)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
      return self.poll.poll(seconds)
    File "/opt/stack/new/neutron/.tox/dsvm-fullstack-constraints/local/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
      raise TimeoutException()
  IndexError: tuple index out of range

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1541742/+subscriptions