yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #71920
[Bug 1746509] Re: TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
** Also affects: nova/pike
Importance: Undecided
Status: New
** Also affects: nova/queens
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1746509
Title:
TypeError: Can't upgrade a READER transaction to a WRITER mid-
transaction
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) pike series:
New
Status in OpenStack Compute (nova) queens series:
New
Bug description:
Hi, I was running OPenstack Newton with no nova_cell0 database and placement-api setup . After migrate to Openstack Pike and correctly setup the nova_cell0 and placement-api everything is working fine except the openstack server list on tenant that already exist .
For example :
1. For a new tenant created after the migration at Pike.
nova --os-project-name="New Project" list
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| c41c7e8d-4bc0-4a0f-a9d3-dc719ae2aff0 | SAMPLE_VM | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.8 |
| 3d3d3e10-f326-4a92-9253-1511e738d1cc | SECOND_INSTANCE | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.5 |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
2. For a old tenant created before the migration at Newton .
nova --os-project-name="InterCon" list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-0dd4ef4d-54c2-4cfd-b8d9-636c5736ef5f)
And here the log related to this error .
2018-01-31 07:45:35.832 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Searching by: {'deleted': False} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2311
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Skipping already-collected cell0 list _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2503
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Listing 1000 instances in cell ea0a32fb-c148-4d4f-a42d-4e46356ff926 _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2506
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.934 2340 DEBUG nova.objects.service [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Generated UUID f9ae1e59-21be-4fc3-9b02-ebb4c35783ec for service 19 _from_db_object /usr/lib/python2.7/site-packages/nova/objects/service.py:245
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Unexpected exception in API method: TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 336, in wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 168, in detail
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 311, in _get_servers
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2415, in get_all
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2512, in _get_instances_by_filters_all_cells
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2543, in _get_instances_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1245, in get_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1235, in _get_by_filters_impl
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1198, in _make_instance_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 448, in _from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions objects.Service, db_inst['services'])
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 1121, in obj_make_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **extra_args)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 246, in _from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions service.save()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 363, in save
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions db_service = db.service_update(self._context, self.id, updates)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return IMPL.service_update(context, service_id, values)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self.force_reraise()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions with ctxt_mgr.writer.using(context):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return self.gen.next()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1034, in _transaction_scope
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions context=context) as resource:
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 684, in _produce_block
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self._writer()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 702, in _writer
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions "Can't upgrade a READER transaction "
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
2018-01-31 07:45:35.938 2340 INFO nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2018-01-31 07:45:35.938 2340 DEBUG nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1029
2018-01-31 07:45:35.939 2340 INFO nova.osapi_compute.wsgi.server [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] 192.168.0.180 "GET /v2.1/servers/detail?all_tenants=True HTTP/1.1" status: 500 len: 626 time: 0.2633660
but at same tenant I can show a instance information .
nova --os-project-name="Intercon" show 5249185f-0a8e-4b24-a777-214ed967695f
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| INTERNAL-NETWORK network | 192.168.120.14 |
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | XXXXXXXXXXX |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-10-19T15:30:57.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2017-10-19T15:30:47Z |
| description | zabbix-intercon |
| flavor:disk | 40 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.medium |
| flavor:ram | 4096 |
| flavor:swap | 0 |
| flavor:vcpus | 2 |
| hostId | 723f077d53d4182fcb76bb87c7fc16efd7ad5fea5d939a9d1a61cb07 |
| id | 5249185f-0a8e-4b24-a777-214ed967695f |
| image | Centos-7 (d3f18d34-bb00-44e1-85cd-71bfae7ca24a) |
| key_name | saulo-fcm360_id_rsa |
| locked | False |
| metadata | {} |
| name | zabbix-intercon |
| os-extended-volumes:volumes_attached | [{"id": "4a30e8a4-86c3-4cef-9f89-5bccefd9df22", "delete_on_termination": false}] |
| progress | 0 |
| security_groups | Zabbix Access, default |
| status | ACTIVE |
| tags | [] |
| tenant_id | e082484fc4004da78407a0c0857ae8a9 |
| updated | 2018-01-31T00:06:36Z |
| user_id | ff9a3af4473c480cacd0c4e18d68edb4 |
+--------------------------------------+----------------------------------------------------------------------------------+
Ant help ?
Saulo Augusto Silva
sauloaugustosilva at gmail.com
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1746509/+subscriptions
References
