yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #71945
[Bug 1746509] Re: TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
Reviewed: https://review.openstack.org/555093
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4
Submitter: Zuul
Branch: master
commit b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4
Author: melanie witt <melwittt@xxxxxxxxx>
Date: Wed Mar 21 22:57:50 2018 +0000
Move _make_instance_list call outside of DB transaction context
The _make_instance_list method is used to make an InstanceList object
out of database dict-like instance objects. It's possible while making
the list that the various _from_db_object methods that are called might
do their own database writes.
Currently, we're calling _make_instance_list nested inside of a 'reader'
database transaction context and we hit the error:
TypeError: Can't upgrade a READER transaction to a WRITER
mid-transaction
during the _make_instance_list call if anything tries to do a database
write. The scenario encountered was after an upgrade to Pike, older
service records without UUIDs were attempted to be updated with UUIDs
upon access, and that access happened to be during an instance list,
so it failed when trying to write the service UUID while nested inside
the 'reader' database transaction context.
This simply moves the _make_instance_list method call out from the
@db.select_db_reader_mode decorated _get_by_filters_impl method to the
get_by_filters method to remove the nesting.
Closes-Bug: #1746509
Change-Id: Ifadf408802cc15eb9769d2dc1fc920426bb7fc20
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1746509
Title:
TypeError: Can't upgrade a READER transaction to a WRITER mid-
transaction
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) pike series:
In Progress
Status in OpenStack Compute (nova) queens series:
In Progress
Bug description:
Hi, I was running OPenstack Newton with no nova_cell0 database and placement-api setup . After migrate to Openstack Pike and correctly setup the nova_cell0 and placement-api everything is working fine except the openstack server list on tenant that already exist .
For example :
1. For a new tenant created after the migration at Pike.
nova --os-project-name="New Project" list
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| c41c7e8d-4bc0-4a0f-a9d3-dc719ae2aff0 | SAMPLE_VM | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.8 |
| 3d3d3e10-f326-4a92-9253-1511e738d1cc | SECOND_INSTANCE | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.5 |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
2. For a old tenant created before the migration at Newton .
nova --os-project-name="InterCon" list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-0dd4ef4d-54c2-4cfd-b8d9-636c5736ef5f)
And here the log related to this error .
2018-01-31 07:45:35.832 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Searching by: {'deleted': False} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2311
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Skipping already-collected cell0 list _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2503
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Listing 1000 instances in cell ea0a32fb-c148-4d4f-a42d-4e46356ff926 _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2506
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.934 2340 DEBUG nova.objects.service [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Generated UUID f9ae1e59-21be-4fc3-9b02-ebb4c35783ec for service 19 _from_db_object /usr/lib/python2.7/site-packages/nova/objects/service.py:245
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Unexpected exception in API method: TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 336, in wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 168, in detail
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 311, in _get_servers
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2415, in get_all
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2512, in _get_instances_by_filters_all_cells
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2543, in _get_instances_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1245, in get_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1235, in _get_by_filters_impl
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1198, in _make_instance_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 448, in _from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions objects.Service, db_inst['services'])
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 1121, in obj_make_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **extra_args)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 246, in _from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions service.save()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 363, in save
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions db_service = db.service_update(self._context, self.id, updates)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return IMPL.service_update(context, service_id, values)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self.force_reraise()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions with ctxt_mgr.writer.using(context):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return self.gen.next()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1034, in _transaction_scope
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions context=context) as resource:
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 684, in _produce_block
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self._writer()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 702, in _writer
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions "Can't upgrade a READER transaction "
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
2018-01-31 07:45:35.938 2340 INFO nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2018-01-31 07:45:35.938 2340 DEBUG nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1029
2018-01-31 07:45:35.939 2340 INFO nova.osapi_compute.wsgi.server [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] 192.168.0.180 "GET /v2.1/servers/detail?all_tenants=True HTTP/1.1" status: 500 len: 626 time: 0.2633660
but at same tenant I can show a instance information .
nova --os-project-name="Intercon" show 5249185f-0a8e-4b24-a777-214ed967695f
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| INTERNAL-NETWORK network | 192.168.120.14 |
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | XXXXXXXXXXX |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-10-19T15:30:57.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2017-10-19T15:30:47Z |
| description | zabbix-intercon |
| flavor:disk | 40 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.medium |
| flavor:ram | 4096 |
| flavor:swap | 0 |
| flavor:vcpus | 2 |
| hostId | 723f077d53d4182fcb76bb87c7fc16efd7ad5fea5d939a9d1a61cb07 |
| id | 5249185f-0a8e-4b24-a777-214ed967695f |
| image | Centos-7 (d3f18d34-bb00-44e1-85cd-71bfae7ca24a) |
| key_name | saulo-fcm360_id_rsa |
| locked | False |
| metadata | {} |
| name | zabbix-intercon |
| os-extended-volumes:volumes_attached | [{"id": "4a30e8a4-86c3-4cef-9f89-5bccefd9df22", "delete_on_termination": false}] |
| progress | 0 |
| security_groups | Zabbix Access, default |
| status | ACTIVE |
| tags | [] |
| tenant_id | e082484fc4004da78407a0c0857ae8a9 |
| updated | 2018-01-31T00:06:36Z |
| user_id | ff9a3af4473c480cacd0c4e18d68edb4 |
+--------------------------------------+----------------------------------------------------------------------------------+
Ant help ?
Saulo Augusto Silva
sauloaugustosilva at gmail.com
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1746509/+subscriptions
References
