yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1879950] [NEW] Unable to update router name with ovn

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jakub Libosvar <1879950@xxxxxxxxxxxxxxxxxx>
Date: Thu, 21 May 2020 13:30:24 -0000
Reply-to: Bug 1879950 <1879950@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Neutron return 500 when attempting to change name of an existing router:

$ openstack router set --name router-flat 29ead671-3957-4bf2-9dbf-e1bff0a12ab2
HttpException: 500: Server Error for url: http://192.168.121.189:9696/v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2, Request Failed: internal server error while processing your request.

May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:liveness_check_at': '2020-05-21T13:29:47.048155+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:metadata_liveness_check_at': '2020-05-21T13:29:47.055462+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: DEBUG neutron.db.agents_db [None req-29c44001-08f7-4ff1-9885-2a8fc4dc7e61 None None] Agent healthcheck: found 2 active agents {{(pid=3400) agent_health_check /opt/stack/neutron/neutron/db/agents_db.py:327}}
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-38aed43c-4b9a-4521-9e69-349701e72f62 service neutron] 192.168.121.189 "GET /v2.0/ports?device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97&fields=binding%3Ahost_id&fields=binding%3Avif_type HTTP/1.1" status: 200  len: 265 time: 0.0434918
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-d77dcd0a-bac0-49ec-ba88-73e2a220cb6c service neutron] 192.168.121.189 "GET /v2.0/ports?tenant_id=9635d9bacb954ae78b28222e6eba4070&device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97 HTTP/1.1" status: 200  len: 1456 time: 0.0404735
May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] Attributes excluded by policy engine: ['vlan_transparent'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] 192.168.121.189 "GET /v2.0/networks?id=336d1bc4-69c6-4b70-b48c-36b190d979db HTTP/1.1" status: 200  len: 930 time: 0.0695074
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-48e27f2d-f333-4b8a-b992-fa46d6a849ba service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=10.0.0.2&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200  len: 1032 time: 0.0478008
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-2354eb84-e80c-4abb-91f3-25701844207b service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=fd4f%3A707c%3A5223%3A0%3Af816%3A3eff%3Afeac%3Aa720&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200  len: 217 time: 0.0307662
May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] 192.168.121.189 "GET /v2.0/subnets?id=5cce10c1-b3c3-4e3f-b709-2066dcc2c785&id=28dfb2ed-2662-4638-a2d4-9369c64b2f3c HTTP/1.1" status: 200  len: 1599 time: 0.0369835
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-a6779a14-962e-461e-b806-be974da65e35 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200  len: 1441 time: 0.0365410
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-7db1d18d-7e32-4a44-a9e8-e855fd16a079 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200  len: 1441 time: 0.0359824
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8a3492de-0578-4c0b-bab6-4bf86f630196 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=segments HTTP/1.1" status: 200  len: 212 time: 0.0682638
May 21 13:29:49 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-211dc48a-64e4-4e49-b630-88d45c543861 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200  len: 277 time: 0.0675299
May 21 13:29:56 devstack neutron-server[3378]: DEBUG neutron.wsgi [-] (3395) accepted ('192.168.121.189', 54086) {{(pid=3395) server /usr/local/lib/python3.6/site-packages/eventlet/wsgi.py:985}}
May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.api.v2.resource [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] show failed (client error): The resource could not be found.
May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.wsgi [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] 192.168.121.189 "GET /v2.0/routers/router1 HTTP/1.1" status: 404  len: 312 time: 0.0663443
May 21 13:29:57 devstack neutron-server[3378]: INFO neutron.wsgi [None req-82148186-c9ef-4e98-977c-0efbfeafbf84 demo admin] 192.168.121.189 "GET /v2.0/routers?name=router1 HTTP/1.1" status: 200  len: 859 time: 0.1023729
May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron.api.v2.base [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Request body: {'router': {'name': 'router-flat'}} {{(pid=3395) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:719}}
May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, after_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Chassis candidates with external connectivity: ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) get_candidates_for_scheduling /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1412}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.scheduler.l3_ovn_scheduler [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Gateway lrp-d7c38c52-9350-47bf-9275-118a24538a9f scheduled on chassis ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) _schedule_gateway /opt/stack/neutron/neutron/scheduler/l3_ovn_scheduler.py:85}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, resource={'id': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2', 'name': 'router-flat', 'tenant_id': '9635d9bacb954ae78b28222e6eba4070', 'admin_state_up': True, 'status': 'ACTIVE', 'external_gateway_info': {'network_id': 'c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'external_fixed_ips': [{'subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f', 'ip_address': '172.24.4.51'}, {'subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'ip_address': '2001:db8::1'}], 'enable_snat': True}, 'gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'description': '', 'routes': [], 'tags': [], 'created_at': '2020-05-20T11:27:32Z', 'updated_at': '2020-05-21T13:29:57Z', 'revision_number': 9, 'project_id': '9635d9bacb954ae78b28222e6eba4070'}, resource_type=routers, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DeleteLRouterExtGwCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DeleteNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, type=snat, logical_ip=10.0.0.0/26, external_ip=172.24.4.51, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): AddLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, may_exist=True, columns={'mac': 'fa:16:3e:54:cf:1a', 'networks': ['2001:db8::1/64', '172.24.4.51/24'], 'external_ids': {'neutron:revision_number': '6', 'neutron:subnet_ids': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'neutron:network_name': 'neutron-c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'neutron:router_name': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2'}, 'options': {}, 'gateway_chassis': ['30a0cb52-356d-4f2d-be32-821652201fc7']}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): SetLRouterPortInLSwitchPortCommand(lswitch_port=d7c38c52-9350-47bf-9275-118a24538a9f, lrouter_port=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '0.0.0.0/0', 'nexthop': '172.24.4.1', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '::/0', 'nexthop': '2001:db8::2', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): AddNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'type': 'snat', 'logical_ip': '10.0.0.0/26', 'external_ip': '172.24.4.51'}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): UpdateLRouterCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'external_ids': {'neutron:router_name': 'router-flat', 'neutron:gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'neutron:revision_number': '9'}, 'enabled': True}, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]:   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
May 21 13:29:58 devstack neutron-server[3378]:     txn.results.put(txn.do_commit())
May 21 13:29:58 devstack neutron-server[3378]:   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
May 21 13:29:58 devstack neutron-server[3378]:     raise RuntimeError(msg)
May 21 13:29:58 devstack neutron-server[3378]: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]:
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update router 29ead671-3957-4bf2-9dbf-e1bff0a12ab2. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update lrouter for 29ead671-3957-4bf2-9dbf-e1bff0a12ab2: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     original_router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     'error': e})
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     context, router_id, added, removed, txn=txn)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     yield t
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     del self._nested_txns_map[cur_thread_id]
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     self.result = self.commit()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     raise result.ex
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     txn.results.put(txn.do_commit())
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     raise RuntimeError(msg)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Added routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:113}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Removed routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:121}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: ERROR root [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router\n    original_router)\n', '  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router\n    \'error\': e})\n', '  File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n    raise value\n', '  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router\n    context, router_id, added, removed, txn=txn)\n', '  File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n    next(self.gen)\n', '  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction\n    yield t\n', '  File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n    next(self.gen)\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction\n    del self._nested_txns_map[cur_thread_id]\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__\n    self.result = self.commit()\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit\n    raise result.ex\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run\n    txn.results.put(txn.do_commit())\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit\n    raise RuntimeError(msg)\n', 'RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}\n']: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] update failed: No details.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     original_router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     'error': e})
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     context, router_id, added, removed, txn=txn)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     yield t
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     del self._nested_txns_map[cur_thread_id]
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.result = self.commit()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise result.ex
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     txn.results.put(txn.do_commit())
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise RuntimeError(msg)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred:
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     result = method(request=request, **args)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 625, in update
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return self._update(request, id, body, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 685, in _update
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     obj = obj_updater(request.context, id, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 148, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     revert_router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/extraroute_db.py", line 72, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     context, id, router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1963, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     id, router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return method(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_db.py", line 287, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     updated = self._make_router_dict(router_db)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_db.py", line 198, in _make_router_dict
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     'network_id': router.gw_port['network_id'],
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 287, in __get__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return self.impl.get(instance_state(instance), dict_)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 723, in get
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     value = self.callable_(state, passive)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 710, in _load_for_state
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     % (orm_util.state_str(state), self.key)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
May 21 13:29:58 devstack neutron-server[3378]: INFO neutron.wsgi [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] 192.168.121.189 "PUT /v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2 HTTP/1.1" status: 500  len: 368 time: 1.8027527

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1879950

Title:
  Unable to update router name with ovn

Status in neutron:
  New

Bug description:
  Neutron return 500 when attempting to change name of an existing
  router:

  $ openstack router set --name router-flat 29ead671-3957-4bf2-9dbf-e1bff0a12ab2
  HttpException: 500: Server Error for url: http://192.168.121.189:9696/v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2, Request Failed: internal server error while processing your request.

  May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:liveness_check_at': '2020-05-21T13:29:47.048155+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:metadata_liveness_check_at': '2020-05-21T13:29:47.055462+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
  May 21 13:29:47 devstack neutron-server[3378]: DEBUG neutron.db.agents_db [None req-29c44001-08f7-4ff1-9885-2a8fc4dc7e61 None None] Agent healthcheck: found 2 active agents {{(pid=3400) agent_health_check /opt/stack/neutron/neutron/db/agents_db.py:327}}
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-38aed43c-4b9a-4521-9e69-349701e72f62 service neutron] 192.168.121.189 "GET /v2.0/ports?device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97&fields=binding%3Ahost_id&fields=binding%3Avif_type HTTP/1.1" status: 200  len: 265 time: 0.0434918
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-d77dcd0a-bac0-49ec-ba88-73e2a220cb6c service neutron] 192.168.121.189 "GET /v2.0/ports?tenant_id=9635d9bacb954ae78b28222e6eba4070&device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97 HTTP/1.1" status: 200  len: 1456 time: 0.0404735
  May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] Attributes excluded by policy engine: ['vlan_transparent'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] 192.168.121.189 "GET /v2.0/networks?id=336d1bc4-69c6-4b70-b48c-36b190d979db HTTP/1.1" status: 200  len: 930 time: 0.0695074
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-48e27f2d-f333-4b8a-b992-fa46d6a849ba service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=10.0.0.2&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200  len: 1032 time: 0.0478008
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-2354eb84-e80c-4abb-91f3-25701844207b service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=fd4f%3A707c%3A5223%3A0%3Af816%3A3eff%3Afeac%3Aa720&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200  len: 217 time: 0.0307662
  May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
  May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] 192.168.121.189 "GET /v2.0/subnets?id=5cce10c1-b3c3-4e3f-b709-2066dcc2c785&id=28dfb2ed-2662-4638-a2d4-9369c64b2f3c HTTP/1.1" status: 200  len: 1599 time: 0.0369835
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-a6779a14-962e-461e-b806-be974da65e35 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200  len: 1441 time: 0.0365410
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-7db1d18d-7e32-4a44-a9e8-e855fd16a079 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200  len: 1441 time: 0.0359824
  May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8a3492de-0578-4c0b-bab6-4bf86f630196 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=segments HTTP/1.1" status: 200  len: 212 time: 0.0682638
  May 21 13:29:49 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-211dc48a-64e4-4e49-b630-88d45c543861 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200  len: 277 time: 0.0675299
  May 21 13:29:56 devstack neutron-server[3378]: DEBUG neutron.wsgi [-] (3395) accepted ('192.168.121.189', 54086) {{(pid=3395) server /usr/local/lib/python3.6/site-packages/eventlet/wsgi.py:985}}
  May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.api.v2.resource [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] show failed (client error): The resource could not be found.
  May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.wsgi [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] 192.168.121.189 "GET /v2.0/routers/router1 HTTP/1.1" status: 404  len: 312 time: 0.0663443
  May 21 13:29:57 devstack neutron-server[3378]: INFO neutron.wsgi [None req-82148186-c9ef-4e98-977c-0efbfeafbf84 demo admin] 192.168.121.189 "GET /v2.0/routers?name=router1 HTTP/1.1" status: 200  len: 859 time: 0.1023729
  May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron.api.v2.base [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Request body: {'router': {'name': 'router-flat'}} {{(pid=3395) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:719}}
  May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
  May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, after_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Chassis candidates with external connectivity: ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) get_candidates_for_scheduling /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1412}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.scheduler.l3_ovn_scheduler [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Gateway lrp-d7c38c52-9350-47bf-9275-118a24538a9f scheduled on chassis ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) _schedule_gateway /opt/stack/neutron/neutron/scheduler/l3_ovn_scheduler.py:85}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, resource={'id': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2', 'name': 'router-flat', 'tenant_id': '9635d9bacb954ae78b28222e6eba4070', 'admin_state_up': True, 'status': 'ACTIVE', 'external_gateway_info': {'network_id': 'c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'external_fixed_ips': [{'subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f', 'ip_address': '172.24.4.51'}, {'subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'ip_address': '2001:db8::1'}], 'enable_snat': True}, 'gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'description': '', 'routes': [], 'tags': [], 'created_at': '2020-05-20T11:27:32Z', 'updated_at': '2020-05-21T13:29:57Z', 'revision_number': 9, 'project_id': '9635d9bacb954ae78b28222e6eba4070'}, resource_type=routers, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DeleteLRouterExtGwCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DeleteNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, type=snat, logical_ip=10.0.0.0/26, external_ip=172.24.4.51, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): AddLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, may_exist=True, columns={'mac': 'fa:16:3e:54:cf:1a', 'networks': ['2001:db8::1/64', '172.24.4.51/24'], 'external_ids': {'neutron:revision_number': '6', 'neutron:subnet_ids': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'neutron:network_name': 'neutron-c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'neutron:router_name': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2'}, 'options': {}, 'gateway_chassis': ['30a0cb52-356d-4f2d-be32-821652201fc7']}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): SetLRouterPortInLSwitchPortCommand(lswitch_port=d7c38c52-9350-47bf-9275-118a24538a9f, lrouter_port=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '0.0.0.0/0', 'nexthop': '172.24.4.1', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '::/0', 'nexthop': '2001:db8::2', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): AddNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'type': 'snat', 'logical_ip': '10.0.0.0/26', 'external_ip': '172.24.4.51'}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): UpdateLRouterCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'external_ids': {'neutron:router_name': 'router-flat', 'neutron:gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'neutron:revision_number': '9'}, 'enabled': True}, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
  May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Traceback (most recent call last):
  May 21 13:29:58 devstack neutron-server[3378]:   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
  May 21 13:29:58 devstack neutron-server[3378]:     txn.results.put(txn.do_commit())
  May 21 13:29:58 devstack neutron-server[3378]:   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
  May 21 13:29:58 devstack neutron-server[3378]:     raise RuntimeError(msg)
  May 21 13:29:58 devstack neutron-server[3378]: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  May 21 13:29:58 devstack neutron-server[3378]:
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update router 29ead671-3957-4bf2-9dbf-e1bff0a12ab2. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update lrouter for 29ead671-3957-4bf2-9dbf-e1bff0a12ab2: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin Traceback (most recent call last):
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     original_router)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     'error': e})
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     context, router_id, added, removed, txn=txn)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     next(self.gen)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     yield t
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     next(self.gen)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     del self._nested_txns_map[cur_thread_id]
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     self.result = self.commit()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     raise result.ex
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     txn.results.put(txn.do_commit())
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin     raise RuntimeError(msg)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Added routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:113}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Removed routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:121}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
  May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
  May 21 13:29:58 devstack neutron-server[3378]: ERROR root [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router\n    original_router)\n', '  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router\n    \'error\': e})\n', '  File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n    raise value\n', '  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router\n    context, router_id, added, removed, txn=txn)\n', '  File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n    next(self.gen)\n', '  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction\n    yield t\n', '  File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n    next(self.gen)\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction\n    del self._nested_txns_map[cur_thread_id]\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__\n    self.result = self.commit()\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit\n    raise result.ex\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run\n    txn.results.put(txn.do_commit())\n', '  File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit\n    raise RuntimeError(msg)\n', 'RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}\n']: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] update failed: No details.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     original_router)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     'error': e})
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     context, router_id, added, removed, txn=txn)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     next(self.gen)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     yield t
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     next(self.gen)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     del self._nested_txns_map[cur_thread_id]
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.result = self.commit()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise result.ex
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     txn.results.put(txn.do_commit())
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise RuntimeError(msg)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred:
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     result = method(request=request, **args)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 625, in update
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return self._update(request, id, body, **kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 685, in _update
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     obj = obj_updater(request.context, id, **kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 148, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     revert_router)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/extraroute_db.py", line 72, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     context, id, router)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1963, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     id, router)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     self.force_reraise()
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     raise value
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_db.py", line 287, in update_router
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     updated = self._make_router_dict(router_db)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_db.py", line 198, in _make_router_dict
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     'network_id': router.gw_port['network_id'],
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 287, in __get__
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     return self.impl.get(instance_state(instance), dict_)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 723, in get
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     value = self.callable_(state, passive)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource   File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 710, in _load_for_state
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource     % (orm_util.state_str(state), self.key)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
  May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
  May 21 13:29:58 devstack neutron-server[3378]: INFO neutron.wsgi [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] 192.168.121.189 "PUT /v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2 HTTP/1.1" status: 500  len: 368 time: 1.8027527

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1879950/+subscriptions
Follow ups

[Bug 1879950] Re: [OVN] neutron-ovn-db-sync-util wipes out hash ring nodes
From: OpenStack Infra, 2020-05-22