yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1851871] [NEW] Unit tests intermittently fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" during service destroy

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Fri, 08 Nov 2019 17:57:54 -0000
Reply-to: Bug 1851871 <1851871@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen here:

https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e8/691900/3/check
/openstack-tox-py27/6e8b215/testr_results.html.gz

ft1.34: nova.tests.unit.conductor.test_conductor.ConductorTaskAPITestCase.test_unshelve_instance_schedule_and_rebuild_novalid_hosttesttools.testresult.real._StringException: pythonlogging:'': {{{
2019-11-08 16:07:22,725 INFO [nova.virt.driver] Loading compute driver 'fake.SmallFakeDriver'
2019-11-08 16:07:22,735 INFO [nova.compute.manager] Deleting orphan compute node 2 hypervisor host is fake_phyp1, nodes are set(['fakenode1', 'fakenode2'])
2019-11-08 16:07:22,739 ERROR [nova.compute.manager] Failed to delete compute node resource provider for compute node 906973a2-d73c-4254-8f1b-dfce1ae55359: An auth plugin is required to determine endpoint URL
2019-11-08 16:07:22,753 WARNING [nova.compute.resource_tracker] No compute node record for fake-mini:fakenode1
2019-11-08 16:07:22,760 INFO [nova.compute.resource_tracker] Compute node record created for fake-mini:fakenode1 with uuid: f4a97ca1-8388-4005-a3e0-d714beebcd69
2019-11-08 16:07:22,803 WARNING [nova.compute.resource_tracker] No compute node record for fake-mini:fakenode2
2019-11-08 16:07:22,807 INFO [nova.compute.resource_tracker] Compute node record created for fake-mini:fakenode2 with uuid: bbd1fd88-6b11-4de7-9d0e-a909d51df43d
2019-11-08 16:07:22,851 INFO [nova.service] Starting conductor node (version 20.1.0)
2019-11-08 16:07:22,970 WARNING [nova.conductor.manager] No valid host found for unshelve instance
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/fixture.py", line 125, in cleanUp
    return self._cleanups(raise_errors=raise_first)
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py", line 89, in __call__
    reraise(error[0], error[1], error[2])
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py", line 83, in __call__
    cleanup(*args, **kwargs)
  File "nova/service.py", line 271, in kill
    self.service_ref.destroy()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
    return fn(self, *args, **kwargs)
  File "nova/objects/service.py", line 446, in destroy
    db.service_destroy(self._context, self.id)
  File "nova/db/api.py", line 99, in service_destroy
    return IMPL.service_destroy(context, service_id)
  File "nova/db/sqlalchemy/api.py", line 222, in wrapped
    with ctxt_mgr.writer.using(context):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
    context=context) as resource:
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 710, in _produce_block
    self._writer()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 725, in _writer
    "Can't upgrade a READER transaction "
TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction

But there are a few hits for this in other tests/jobs for unrelated
changes:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TypeError%3A%20Can't%20upgrade%20a%20READER%20transaction%20to%20a%20WRITER
%20mid-
transaction%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d

12 fails in 7 days.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure testing

** Summary changed:

- Functional tests intermittently fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" during service destroy
+ Unit tests intermittently fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" during service destroy

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1851871

Title:
  Unit tests intermittently fail with "TypeError: Can't upgrade a READER
  transaction to a WRITER mid-transaction" during service destroy

Status in OpenStack Compute (nova):
  New

Bug description:
  Seen here:

  https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e8/691900/3/check
  /openstack-tox-py27/6e8b215/testr_results.html.gz

  ft1.34: nova.tests.unit.conductor.test_conductor.ConductorTaskAPITestCase.test_unshelve_instance_schedule_and_rebuild_novalid_hosttesttools.testresult.real._StringException: pythonlogging:'': {{{
  2019-11-08 16:07:22,725 INFO [nova.virt.driver] Loading compute driver 'fake.SmallFakeDriver'
  2019-11-08 16:07:22,735 INFO [nova.compute.manager] Deleting orphan compute node 2 hypervisor host is fake_phyp1, nodes are set(['fakenode1', 'fakenode2'])
  2019-11-08 16:07:22,739 ERROR [nova.compute.manager] Failed to delete compute node resource provider for compute node 906973a2-d73c-4254-8f1b-dfce1ae55359: An auth plugin is required to determine endpoint URL
  2019-11-08 16:07:22,753 WARNING [nova.compute.resource_tracker] No compute node record for fake-mini:fakenode1
  2019-11-08 16:07:22,760 INFO [nova.compute.resource_tracker] Compute node record created for fake-mini:fakenode1 with uuid: f4a97ca1-8388-4005-a3e0-d714beebcd69
  2019-11-08 16:07:22,803 WARNING [nova.compute.resource_tracker] No compute node record for fake-mini:fakenode2
  2019-11-08 16:07:22,807 INFO [nova.compute.resource_tracker] Compute node record created for fake-mini:fakenode2 with uuid: bbd1fd88-6b11-4de7-9d0e-a909d51df43d
  2019-11-08 16:07:22,851 INFO [nova.service] Starting conductor node (version 20.1.0)
  2019-11-08 16:07:22,970 WARNING [nova.conductor.manager] No valid host found for unshelve instance
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/fixture.py", line 125, in cleanUp
      return self._cleanups(raise_errors=raise_first)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py", line 89, in __call__
      reraise(error[0], error[1], error[2])
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py", line 83, in __call__
      cleanup(*args, **kwargs)
    File "nova/service.py", line 271, in kill
      self.service_ref.destroy()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
      return fn(self, *args, **kwargs)
    File "nova/objects/service.py", line 446, in destroy
      db.service_destroy(self._context, self.id)
    File "nova/db/api.py", line 99, in service_destroy
      return IMPL.service_destroy(context, service_id)
    File "nova/db/sqlalchemy/api.py", line 222, in wrapped
      with ctxt_mgr.writer.using(context):
    File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
      return self.gen.next()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
      context=context) as resource:
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 710, in _produce_block
      self._writer()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 725, in _writer
      "Can't upgrade a READER transaction "
  TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction

  But there are a few hits for this in other tests/jobs for unrelated
  changes:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TypeError%3A%20Can't%20upgrade%20a%20READER%20transaction%20to%20a%20WRITER
  %20mid-
  transaction%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d

  12 fails in 7 days.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1851871/+subscriptions
Follow ups

[Bug 1851871] Re: Unit tests intermittently fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" during service destroy
From: Balazs Gibizer, 2020-04-21