yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2028037] [NEW] Error 500 on create network when PostgreSQL is used

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <2028037@xxxxxxxxxxxxxxxxxx>
Date: Tue, 18 Jul 2023 07:50:32 -0000
Reply-to: Bug 2028037 <2028037@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Error seen in the periodic job neutron-ovn-tempest-postgres-full every
day since 5.07.2023: https://zuul.openstack.org/builds?job_name=neutron-
ovn-tempest-postgres-
full&project=openstack%2Fneutron&branch=master&skip=0

Error from neutron server log:
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: WARNING oslo_db.sqlalchemy.exc_filters [None req-eb023ace-8737-4e89-9557-e058931a7892 demo demo] DBAPIError exception wrapped.: psycopg2.errors.GroupingError: column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]:                                                              ^
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters     self.dialect.do_execute(
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters psycopg2.errors.GroupingError: column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters                                                              ^
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters 
Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters 
Jul 18 03:38:15.533633 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers [None req-eb023ace-8737-4e89-9557-e058931a7892 demo demo] Mechanism driver 'ovn' failed in create_network_postcommit: oslo_db.exception.DBError: (psycopg2.errors.GroupingError) column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
Jul 18 03:38:15.533633 np0034701773 neutron-server[55620]: LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
Jul 18 03:38:15.533633 np0034701773 neutron-server[55620]:                                                              ^
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: [SQL: SELECT anon_1.project_id AS anon_1_project_id, anon_1.id AS anon_1_id, anon_1.name AS anon_1_name, anon_1.network_id AS anon_1_network_id, anon_1.segment_id AS anon_1_segment_id, anon_1.subnetpool_id AS anon_1_subnetpool_id, anon_1.ip_version AS anon_1_ip_version, anon_1.cidr AS anon_1_cidr, anon_1.gateway_ip AS anon_1_gateway_ip, anon_1.enable_dhcp AS anon_1_enable_dhcp, anon_1.ipv6_ra_mode AS anon_1_ipv6_ra_mode, anon_1.ipv6_address_mode AS anon_1_ipv6_address_mode, anon_1.standard_attr_id AS anon_1_standard_attr_id, subnetpools_1.shared AS subnetpools_1_shared, subnetpoolrbacs_1.project_id AS subnetpoolrbacs_1_project_id, subnetpoolrbacs_1.id AS subnetpoolrbacs_1_id, subnetpoolrbacs_1.target_project AS subnetpoolrbacs_1_target_project, subnetpoolrbacs_1.action AS subnetpoolrbacs_1_action, subnetpoolrbacs_1.object_id AS subnetpoolrbacs_1_object_id, standardattributes_1.id AS standardattributes_1_id, standardattributes_1.resource_type AS standardattributes_1_resource_type, standardattributes_1.description AS standardattributes_1_description, standardattributes_1.revision_number AS standardattributes_1_revision_number, standardattributes_1.created_at AS standardattributes_1_created_at, standardattributes_1.updated_at AS standardattributes_1_updated_at, subnetpools_1.project_id AS subnetpools_1_project_id, subnetpools_1.id AS subnetpools_1_id, subnetpools_1.name AS subnetpools_1_name, subnetpools_1.ip_version AS subnetpools_1_ip_version, subnetpools_1.default_prefixlen AS subnetpools_1_default_prefixlen, subnetpools_1.min_prefixlen AS subnetpools_1_min_prefixlen, subnetpools_1.max_prefixlen AS subnetpools_1_max_prefixlen, subnetpools_1.is_default AS subnetpools_1_is_default, subnetpools_1.default_quota AS subnetpools_1_default_quota, subnetpools_1.hash AS subnetpools_1_hash, subnetpools_1.address_scope_id AS subnetpools_1_address_scope_id, subnetpools_1.standard_attr_id AS subnetpools_1_standard_attr_id, networkrbacs_1.project_id AS networkrbacs_1_project_id, networkrbacs_1.id AS networkrbacs_1_id, networkrbacs_1.target_project AS networkrbacs_1_target_project, networkrbacs_1.action AS networkrbacs_1_action, networkrbacs_1.object_id AS networkrbacs_1_object_id, standardattributes_2.id AS standardattributes_2_id, standardattributes_2.resource_type AS standardattributes_2_resource_type, standardattributes_2.description AS standardattributes_2_description, standardattributes_2.revision_number AS standardattributes_2_revision_number, standardattributes_2.created_at AS standardattributes_2_created_at, standardattributes_2.updated_at AS standardattributes_2_updated_at, subnet_dns_publish_fixed_ips_1.subnet_id AS subnet_dns_publish_fixed_ips_1_subnet_id, subnet_dns_publish_fixed_ips_1.dns_publish_fixed_ip AS subnet_dns_publish_fixed_ips_1_dns_publish_fixed_ip
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: FROM (SELECT subnets.project_id AS project_id, subnets.id AS id, subnets.name AS name, subnets.network_id AS network_id, subnets.segment_id AS segment_id, subnets.subnetpool_id AS subnetpool_id, subnets.ip_version AS ip_version, subnets.cidr AS cidr, subnets.gateway_ip AS gateway_ip, subnets.enable_dhcp AS enable_dhcp, subnets.ipv6_ra_mode AS ipv6_ra_mode, subnets.ipv6_address_mode AS ipv6_address_mode, subnets.standard_attr_id AS standard_attr_id, subnet_service_types.subnet_id AS subnet_id, subnet_service_types.service_type AS service_type
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: FROM subnets LEFT OUTER JOIN networkrbacs ON subnets.network_id = networkrbacs.object_id LEFT OUTER JOIN subnet_service_types ON subnets.id = subnet_service_types.subnet_id
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: WHERE (subnets.project_id = %(project_id_1)s OR networkrbacs.action IN (%(action_1_1)s, %(action_1_2)s) AND (networkrbacs.target_project = %(target_project_1)s OR networkrbacs.target_project = %(target_project_2)s)) AND subnets.network_id = %(network_id_1)s AND (subnet_service_types.service_type IS NULL OR subnet_service_types.service_type = %(service_type_1)s OR false) GROUP BY subnets.id) AS anon_1 LEFT OUTER JOIN subnetpools AS subnetpools_1 ON anon_1.subnetpool_id = subnetpools_1.id LEFT OUTER JOIN subnetpoolrbacs AS subnetpoolrbacs_1 ON subnetpools_1.id = subnetpoolrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_1 ON standardattributes_1.id = subnetpools_1.standard_attr_id LEFT OUTER JOIN networkrbacs AS networkrbacs_1 ON anon_1.network_id = networkrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_2 ON standardattributes_2.id = anon_1.standard_attr_id LEFT OUTER JOIN subnet_dns_publish_fixed_ips AS subnet_dns_publish_fixed_ips_1 ON anon_1.id = subnet_dns_publish_fixed_ips_1.subnet_id]
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: [parameters: {'project_id_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_2': '*', 'network_id_1': 'ea013097-aa12-4c81-b094-353e6881714e', 'service_type_1': 'network:distributed', 'action_1_1': 'access_as_shared', 'action_1_2': 'access_as_readonly'}]
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: (Background on this error at: https://sqlalche.me/e/14/f405)
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers psycopg2.errors.GroupingError: column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers                                                              ^
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers The above exception was the direct cause of the following exception:
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 575, in create_network_postcommit
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self._ovn_client.create_network(context.plugin_context, network)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1886, in create_network
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.create_metadata_port(context, network)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2372, in create_metadata_port
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return p_utils.create_port(self._plugin, context, port)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/plugins/utils.py", line 338, in create_port
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return core_plugin.create_port(
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/common/utils.py", line 731, in inner
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 223, in wrapped
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f_with_retry(*args, **kwargs,
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 137, in wrapped
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     with excutils.save_and_reraise_exception():
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.force_reraise()
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise self.value
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     with excutils.save_and_reraise_exception() as ectxt:
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.force_reraise()
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise self.value
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     with excutils.save_and_reraise_exception():
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.force_reraise()
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise self.value
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 181, in wrapped
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*dup_args, **dup_kwargs)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1586, in create_port
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     result, mech_context = self._create_port_db(context, port)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1551, in _create_port_db
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     port_db = self.create_port_db(context, port)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1538, in create_port_db
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.ipam.allocate_ips_for_port_and_store(
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 219, in allocate_ips_for_port_and_store
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     ips = self.allocate_ips_for_port(context, port_copy)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 221, in wrapped
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1044, in wrapper
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return fn(*args, **kwargs)
Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 226, in allocate_ips_for_port
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return self._allocate_ips_for_port(context, port)
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 258, in _allocate_ips_for_port
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     subnets = self._ipam_get_subnets(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_backend_mixin.py", line 679, in _ipam_get_subnets
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     subnets = subnet_obj.Subnet.find_candidate_subnets(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/objects/subnet.py", line 344, in find_candidate_subnets
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return cls._query_filter_by_fixed_ips_segment(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/objects/subnet.py", line 379, in _query_filter_by_fixed_ips_segment
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     subnets = query.all()
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2772, in all
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return self._iter().all()
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2907, in _iter
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     result = self.session.execute(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 1712, in execute
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     result = conn._execute_20(statement, params or {}, execution_options)
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return meth(self, args_10style, kwargs_10style, execution_options)
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return connection._execute_clauseelement(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     ret = self._execute_context(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self._handle_dbapi_exception(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise exception
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers oslo_db.exception.DBError: (psycopg2.errors.GroupingError) column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers                                                              ^
Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers [SQL: SELECT anon_1.project_id AS anon_1_project_id, anon_1.id AS anon_1_id, anon_1.name AS anon_1_name, anon_1.network_id AS anon_1_network_id, anon_1.segment_id AS anon_1_segment_id, anon_1.subnetpool_id AS anon_1_subnetpool_id, anon_1.ip_version AS anon_1_ip_version, anon_1.cidr AS anon_1_cidr, anon_1.gateway_ip AS anon_1_gateway_ip, anon_1.enable_dhcp AS anon_1_enable_dhcp, anon_1.ipv6_ra_mode AS anon_1_ipv6_ra_mode, anon_1.ipv6_address_mode AS anon_1_ipv6_address_mode, anon_1.standard_attr_id AS anon_1_standard_attr_id, subnetpools_1.shared AS subnetpools_1_shared, subnetpoolrbacs_1.project_id AS subnetpoolrbacs_1_project_id, subnetpoolrbacs_1.id AS subnetpoolrbacs_1_id, subnetpoolrbacs_1.target_project AS subnetpoolrbacs_1_target_project, subnetpoolrbacs_1.action AS subnetpoolrbacs_1_action, subnetpoolrbacs_1.object_id AS subnetpoolrbacs_1_object_id, standardattributes_1.id AS standardattributes_1_id, standardattributes_1.resource_type AS standardattributes_1_resource_type, standardattributes_1.description AS standardattributes_1_description, standardattributes_1.revision_number AS standardattributes_1_revision_number, standardattributes_1.created_at AS standardattributes_1_created_at, standardattributes_1.updated_at AS standardattributes_1_updated_at, subnetpools_1.project_id AS subnetpools_1_project_id, subnetpools_1.id AS subnetpools_1_id, subnetpools_1.name AS subnetpools_1_name, subnetpools_1.ip_version AS subnetpools_1_ip_version, subnetpools_1.default_prefixlen AS subnetpools_1_default_prefixlen, subnetpools_1.min_prefixlen AS subnetpools_1_min_prefixlen, subnetpools_1.max_prefixlen AS subnetpools_1_max_prefixlen, subnetpools_1.is_default AS subnetpools_1_is_default, subnetpools_1.default_quota AS subnetpools_1_default_quota, subnetpools_1.hash AS subnetpools_1_hash, subnetpools_1.address_scope_id AS subnetpools_1_address_scope_id, subnetpools_1.standard_attr_id AS subnetpools_1_standard_attr_id, networkrbacs_1.project_id AS networkrbacs_1_project_id, networkrbacs_1.id AS networkrbacs_1_id, networkrbacs_1.target_project AS networkrbacs_1_target_project, networkrbacs_1.action AS networkrbacs_1_action, networkrbacs_1.object_id AS networkrbacs_1_object_id, standardattributes_2.id AS standardattributes_2_id, standardattributes_2.resource_type AS standardattributes_2_resource_type, standardattributes_2.description AS standardattributes_2_description, standardattributes_2.revision_number AS standardattributes_2_revision_number, standardattributes_2.created_at AS standardattributes_2_created_at, standardattributes_2.updated_at AS standardattributes_2_updated_at, subnet_dns_publish_fixed_ips_1.subnet_id AS subnet_dns_publish_fixed_ips_1_subnet_id, subnet_dns_publish_fixed_ips_1.dns_publish_fixed_ip AS subnet_dns_publish_fixed_ips_1_dns_publish_fixed_ip
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers FROM (SELECT subnets.project_id AS project_id, subnets.id AS id, subnets.name AS name, subnets.network_id AS network_id, subnets.segment_id AS segment_id, subnets.subnetpool_id AS subnetpool_id, subnets.ip_version AS ip_version, subnets.cidr AS cidr, subnets.gateway_ip AS gateway_ip, subnets.enable_dhcp AS enable_dhcp, subnets.ipv6_ra_mode AS ipv6_ra_mode, subnets.ipv6_address_mode AS ipv6_address_mode, subnets.standard_attr_id AS standard_attr_id, subnet_service_types.subnet_id AS subnet_id, subnet_service_types.service_type AS service_type
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers FROM subnets LEFT OUTER JOIN networkrbacs ON subnets.network_id = networkrbacs.object_id LEFT OUTER JOIN subnet_service_types ON subnets.id = subnet_service_types.subnet_id
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers WHERE (subnets.project_id = %(project_id_1)s OR networkrbacs.action IN (%(action_1_1)s, %(action_1_2)s) AND (networkrbacs.target_project = %(target_project_1)s OR networkrbacs.target_project = %(target_project_2)s)) AND subnets.network_id = %(network_id_1)s AND (subnet_service_types.service_type IS NULL OR subnet_service_types.service_type = %(service_type_1)s OR false) GROUP BY subnets.id) AS anon_1 LEFT OUTER JOIN subnetpools AS subnetpools_1 ON anon_1.subnetpool_id = subnetpools_1.id LEFT OUTER JOIN subnetpoolrbacs AS subnetpoolrbacs_1 ON subnetpools_1.id = subnetpoolrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_1 ON standardattributes_1.id = subnetpools_1.standard_attr_id LEFT OUTER JOIN networkrbacs AS networkrbacs_1 ON anon_1.network_id = networkrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_2 ON standardattributes_2.id = anon_1.standard_attr_id LEFT OUTER JOIN subnet_dns_publish_fixed_ips AS subnet_dns_publish_fixed_ips_1 ON anon_1.id = subnet_dns_publish_fixed_ips_1.subnet_id]
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers [parameters: {'project_id_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_2': '*', 'network_id_1': 'ea013097-aa12-4c81-b094-353e6881714e', 'service_type_1': 'network:distributed', 'action_1_1': 'access_as_shared', 'action_1_2': 'access_as_readonly'}]
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers (Background on this error at: https://sqlalche.me/e/14/f405)
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.plugin [None req-eb023ace-8737-4e89-9557-e058931a7892 demo demo] mechanism_manager.create_network_postcommit failed, deleting network 'ea013097-aa12-4c81-b094-353e6881714e': neutron.plugins.ml2.common.exceptions.MechanismDriverError

** Affects: neutron
     Importance: Critical
         Status: Confirmed


** Tags: db gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2028037

Title:
  Error 500 on create network when PostgreSQL is used

Status in neutron:
  Confirmed

Bug description:
  Error seen in the periodic job neutron-ovn-tempest-postgres-full every
  day since 5.07.2023:
  https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-
  postgres-full&project=openstack%2Fneutron&branch=master&skip=0

  Error from neutron server log:
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: WARNING oslo_db.sqlalchemy.exc_filters [None req-eb023ace-8737-4e89-9557-e058931a7892 demo demo] DBAPIError exception wrapped.: psycopg2.errors.GroupingError: column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]:                                                              ^
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters     self.dialect.do_execute(
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters psycopg2.errors.GroupingError: column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters                                                              ^
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters 
  Jul 18 03:38:15.524757 np0034701773 neutron-server[55620]: ERROR oslo_db.sqlalchemy.exc_filters 
  Jul 18 03:38:15.533633 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers [None req-eb023ace-8737-4e89-9557-e058931a7892 demo demo] Mechanism driver 'ovn' failed in create_network_postcommit: oslo_db.exception.DBError: (psycopg2.errors.GroupingError) column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
  Jul 18 03:38:15.533633 np0034701773 neutron-server[55620]: LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
  Jul 18 03:38:15.533633 np0034701773 neutron-server[55620]:                                                              ^
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: [SQL: SELECT anon_1.project_id AS anon_1_project_id, anon_1.id AS anon_1_id, anon_1.name AS anon_1_name, anon_1.network_id AS anon_1_network_id, anon_1.segment_id AS anon_1_segment_id, anon_1.subnetpool_id AS anon_1_subnetpool_id, anon_1.ip_version AS anon_1_ip_version, anon_1.cidr AS anon_1_cidr, anon_1.gateway_ip AS anon_1_gateway_ip, anon_1.enable_dhcp AS anon_1_enable_dhcp, anon_1.ipv6_ra_mode AS anon_1_ipv6_ra_mode, anon_1.ipv6_address_mode AS anon_1_ipv6_address_mode, anon_1.standard_attr_id AS anon_1_standard_attr_id, subnetpools_1.shared AS subnetpools_1_shared, subnetpoolrbacs_1.project_id AS subnetpoolrbacs_1_project_id, subnetpoolrbacs_1.id AS subnetpoolrbacs_1_id, subnetpoolrbacs_1.target_project AS subnetpoolrbacs_1_target_project, subnetpoolrbacs_1.action AS subnetpoolrbacs_1_action, subnetpoolrbacs_1.object_id AS subnetpoolrbacs_1_object_id, standardattributes_1.id AS standardattributes_1_id, standardattributes_1.resource_type AS standardattributes_1_resource_type, standardattributes_1.description AS standardattributes_1_description, standardattributes_1.revision_number AS standardattributes_1_revision_number, standardattributes_1.created_at AS standardattributes_1_created_at, standardattributes_1.updated_at AS standardattributes_1_updated_at, subnetpools_1.project_id AS subnetpools_1_project_id, subnetpools_1.id AS subnetpools_1_id, subnetpools_1.name AS subnetpools_1_name, subnetpools_1.ip_version AS subnetpools_1_ip_version, subnetpools_1.default_prefixlen AS subnetpools_1_default_prefixlen, subnetpools_1.min_prefixlen AS subnetpools_1_min_prefixlen, subnetpools_1.max_prefixlen AS subnetpools_1_max_prefixlen, subnetpools_1.is_default AS subnetpools_1_is_default, subnetpools_1.default_quota AS subnetpools_1_default_quota, subnetpools_1.hash AS subnetpools_1_hash, subnetpools_1.address_scope_id AS subnetpools_1_address_scope_id, subnetpools_1.standard_attr_id AS subnetpools_1_standard_attr_id, networkrbacs_1.project_id AS networkrbacs_1_project_id, networkrbacs_1.id AS networkrbacs_1_id, networkrbacs_1.target_project AS networkrbacs_1_target_project, networkrbacs_1.action AS networkrbacs_1_action, networkrbacs_1.object_id AS networkrbacs_1_object_id, standardattributes_2.id AS standardattributes_2_id, standardattributes_2.resource_type AS standardattributes_2_resource_type, standardattributes_2.description AS standardattributes_2_description, standardattributes_2.revision_number AS standardattributes_2_revision_number, standardattributes_2.created_at AS standardattributes_2_created_at, standardattributes_2.updated_at AS standardattributes_2_updated_at, subnet_dns_publish_fixed_ips_1.subnet_id AS subnet_dns_publish_fixed_ips_1_subnet_id, subnet_dns_publish_fixed_ips_1.dns_publish_fixed_ip AS subnet_dns_publish_fixed_ips_1_dns_publish_fixed_ip
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: FROM (SELECT subnets.project_id AS project_id, subnets.id AS id, subnets.name AS name, subnets.network_id AS network_id, subnets.segment_id AS segment_id, subnets.subnetpool_id AS subnetpool_id, subnets.ip_version AS ip_version, subnets.cidr AS cidr, subnets.gateway_ip AS gateway_ip, subnets.enable_dhcp AS enable_dhcp, subnets.ipv6_ra_mode AS ipv6_ra_mode, subnets.ipv6_address_mode AS ipv6_address_mode, subnets.standard_attr_id AS standard_attr_id, subnet_service_types.subnet_id AS subnet_id, subnet_service_types.service_type AS service_type
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: FROM subnets LEFT OUTER JOIN networkrbacs ON subnets.network_id = networkrbacs.object_id LEFT OUTER JOIN subnet_service_types ON subnets.id = subnet_service_types.subnet_id
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: WHERE (subnets.project_id = %(project_id_1)s OR networkrbacs.action IN (%(action_1_1)s, %(action_1_2)s) AND (networkrbacs.target_project = %(target_project_1)s OR networkrbacs.target_project = %(target_project_2)s)) AND subnets.network_id = %(network_id_1)s AND (subnet_service_types.service_type IS NULL OR subnet_service_types.service_type = %(service_type_1)s OR false) GROUP BY subnets.id) AS anon_1 LEFT OUTER JOIN subnetpools AS subnetpools_1 ON anon_1.subnetpool_id = subnetpools_1.id LEFT OUTER JOIN subnetpoolrbacs AS subnetpoolrbacs_1 ON subnetpools_1.id = subnetpoolrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_1 ON standardattributes_1.id = subnetpools_1.standard_attr_id LEFT OUTER JOIN networkrbacs AS networkrbacs_1 ON anon_1.network_id = networkrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_2 ON standardattributes_2.id = anon_1.standard_attr_id LEFT OUTER JOIN subnet_dns_publish_fixed_ips AS subnet_dns_publish_fixed_ips_1 ON anon_1.id = subnet_dns_publish_fixed_ips_1.subnet_id]
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: [parameters: {'project_id_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_2': '*', 'network_id_1': 'ea013097-aa12-4c81-b094-353e6881714e', 'service_type_1': 'network:distributed', 'action_1_1': 'access_as_shared', 'action_1_2': 'access_as_readonly'}]
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: (Background on this error at: https://sqlalche.me/e/14/f405)
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers psycopg2.errors.GroupingError: column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers                                                              ^
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
  Jul 18 03:38:15.533860 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers The above exception was the direct cause of the following exception:
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 575, in create_network_postcommit
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self._ovn_client.create_network(context.plugin_context, network)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1886, in create_network
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.create_metadata_port(context, network)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2372, in create_metadata_port
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return p_utils.create_port(self._plugin, context, port)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/plugins/utils.py", line 338, in create_port
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return core_plugin.create_port(
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/common/utils.py", line 731, in inner
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 223, in wrapped
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f_with_retry(*args, **kwargs,
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 137, in wrapped
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     with excutils.save_and_reraise_exception():
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.force_reraise()
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise self.value
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     with excutils.save_and_reraise_exception() as ectxt:
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.force_reraise()
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise self.value
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     with excutils.save_and_reraise_exception():
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.force_reraise()
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise self.value
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 181, in wrapped
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*dup_args, **dup_kwargs)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1586, in create_port
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     result, mech_context = self._create_port_db(context, port)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1551, in _create_port_db
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     port_db = self.create_port_db(context, port)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1538, in create_port_db
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.ipam.allocate_ips_for_port_and_store(
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 219, in allocate_ips_for_port_and_store
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     ips = self.allocate_ips_for_port(context, port_copy)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 221, in wrapped
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return f(*args, **kwargs)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1044, in wrapper
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return fn(*args, **kwargs)
  Jul 18 03:38:15.534954 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 226, in allocate_ips_for_port
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return self._allocate_ips_for_port(context, port)
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 258, in _allocate_ips_for_port
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     subnets = self._ipam_get_subnets(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/db/ipam_backend_mixin.py", line 679, in _ipam_get_subnets
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     subnets = subnet_obj.Subnet.find_candidate_subnets(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/objects/subnet.py", line 344, in find_candidate_subnets
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return cls._query_filter_by_fixed_ips_segment(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/opt/stack/neutron/neutron/objects/subnet.py", line 379, in _query_filter_by_fixed_ips_segment
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     subnets = query.all()
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2772, in all
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return self._iter().all()
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2907, in _iter
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     result = self.session.execute(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 1712, in execute
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     result = conn._execute_20(statement, params or {}, execution_options)
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return meth(self, args_10style, kwargs_10style, execution_options)
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     return connection._execute_clauseelement(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     ret = self._execute_context(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self._handle_dbapi_exception(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     raise exception
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     self.dialect.do_execute(
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers     cursor.execute(statement, parameters)
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers oslo_db.exception.DBError: (psycopg2.errors.GroupingError) column "subnet_service_types.subnet_id" must appear in the GROUP BY clause or be used in an aggregate function
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers LINE 2: ...de, subnets.standard_attr_id AS standard_attr_id, subnet_ser...
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers                                                              ^
  Jul 18 03:38:15.539238 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers [SQL: SELECT anon_1.project_id AS anon_1_project_id, anon_1.id AS anon_1_id, anon_1.name AS anon_1_name, anon_1.network_id AS anon_1_network_id, anon_1.segment_id AS anon_1_segment_id, anon_1.subnetpool_id AS anon_1_subnetpool_id, anon_1.ip_version AS anon_1_ip_version, anon_1.cidr AS anon_1_cidr, anon_1.gateway_ip AS anon_1_gateway_ip, anon_1.enable_dhcp AS anon_1_enable_dhcp, anon_1.ipv6_ra_mode AS anon_1_ipv6_ra_mode, anon_1.ipv6_address_mode AS anon_1_ipv6_address_mode, anon_1.standard_attr_id AS anon_1_standard_attr_id, subnetpools_1.shared AS subnetpools_1_shared, subnetpoolrbacs_1.project_id AS subnetpoolrbacs_1_project_id, subnetpoolrbacs_1.id AS subnetpoolrbacs_1_id, subnetpoolrbacs_1.target_project AS subnetpoolrbacs_1_target_project, subnetpoolrbacs_1.action AS subnetpoolrbacs_1_action, subnetpoolrbacs_1.object_id AS subnetpoolrbacs_1_object_id, standardattributes_1.id AS standardattributes_1_id, standardattributes_1.resource_type AS standardattributes_1_resource_type, standardattributes_1.description AS standardattributes_1_description, standardattributes_1.revision_number AS standardattributes_1_revision_number, standardattributes_1.created_at AS standardattributes_1_created_at, standardattributes_1.updated_at AS standardattributes_1_updated_at, subnetpools_1.project_id AS subnetpools_1_project_id, subnetpools_1.id AS subnetpools_1_id, subnetpools_1.name AS subnetpools_1_name, subnetpools_1.ip_version AS subnetpools_1_ip_version, subnetpools_1.default_prefixlen AS subnetpools_1_default_prefixlen, subnetpools_1.min_prefixlen AS subnetpools_1_min_prefixlen, subnetpools_1.max_prefixlen AS subnetpools_1_max_prefixlen, subnetpools_1.is_default AS subnetpools_1_is_default, subnetpools_1.default_quota AS subnetpools_1_default_quota, subnetpools_1.hash AS subnetpools_1_hash, subnetpools_1.address_scope_id AS subnetpools_1_address_scope_id, subnetpools_1.standard_attr_id AS subnetpools_1_standard_attr_id, networkrbacs_1.project_id AS networkrbacs_1_project_id, networkrbacs_1.id AS networkrbacs_1_id, networkrbacs_1.target_project AS networkrbacs_1_target_project, networkrbacs_1.action AS networkrbacs_1_action, networkrbacs_1.object_id AS networkrbacs_1_object_id, standardattributes_2.id AS standardattributes_2_id, standardattributes_2.resource_type AS standardattributes_2_resource_type, standardattributes_2.description AS standardattributes_2_description, standardattributes_2.revision_number AS standardattributes_2_revision_number, standardattributes_2.created_at AS standardattributes_2_created_at, standardattributes_2.updated_at AS standardattributes_2_updated_at, subnet_dns_publish_fixed_ips_1.subnet_id AS subnet_dns_publish_fixed_ips_1_subnet_id, subnet_dns_publish_fixed_ips_1.dns_publish_fixed_ip AS subnet_dns_publish_fixed_ips_1_dns_publish_fixed_ip
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers FROM (SELECT subnets.project_id AS project_id, subnets.id AS id, subnets.name AS name, subnets.network_id AS network_id, subnets.segment_id AS segment_id, subnets.subnetpool_id AS subnetpool_id, subnets.ip_version AS ip_version, subnets.cidr AS cidr, subnets.gateway_ip AS gateway_ip, subnets.enable_dhcp AS enable_dhcp, subnets.ipv6_ra_mode AS ipv6_ra_mode, subnets.ipv6_address_mode AS ipv6_address_mode, subnets.standard_attr_id AS standard_attr_id, subnet_service_types.subnet_id AS subnet_id, subnet_service_types.service_type AS service_type
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers FROM subnets LEFT OUTER JOIN networkrbacs ON subnets.network_id = networkrbacs.object_id LEFT OUTER JOIN subnet_service_types ON subnets.id = subnet_service_types.subnet_id
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers WHERE (subnets.project_id = %(project_id_1)s OR networkrbacs.action IN (%(action_1_1)s, %(action_1_2)s) AND (networkrbacs.target_project = %(target_project_1)s OR networkrbacs.target_project = %(target_project_2)s)) AND subnets.network_id = %(network_id_1)s AND (subnet_service_types.service_type IS NULL OR subnet_service_types.service_type = %(service_type_1)s OR false) GROUP BY subnets.id) AS anon_1 LEFT OUTER JOIN subnetpools AS subnetpools_1 ON anon_1.subnetpool_id = subnetpools_1.id LEFT OUTER JOIN subnetpoolrbacs AS subnetpoolrbacs_1 ON subnetpools_1.id = subnetpoolrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_1 ON standardattributes_1.id = subnetpools_1.standard_attr_id LEFT OUTER JOIN networkrbacs AS networkrbacs_1 ON anon_1.network_id = networkrbacs_1.object_id LEFT OUTER JOIN standardattributes AS standardattributes_2 ON standardattributes_2.id = anon_1.standard_attr_id LEFT OUTER JOIN subnet_dns_publish_fixed_ips AS subnet_dns_publish_fixed_ips_1 ON anon_1.id = subnet_dns_publish_fixed_ips_1.subnet_id]
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers [parameters: {'project_id_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_1': '05623a81e93c4eaaa8b5c0574076d9ef', 'target_project_2': '*', 'network_id_1': 'ea013097-aa12-4c81-b094-353e6881714e', 'service_type_1': 'network:distributed', 'action_1_1': 'access_as_shared', 'action_1_2': 'access_as_readonly'}]
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers (Background on this error at: https://sqlalche.me/e/14/f405)
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.managers 
  Jul 18 03:38:15.543505 np0034701773 neutron-server[55620]: ERROR neutron.plugins.ml2.plugin [None req-eb023ace-8737-4e89-9557-e058931a7892 demo demo] mechanism_manager.create_network_postcommit failed, deleting network 'ea013097-aa12-4c81-b094-353e6881714e': neutron.plugins.ml2.common.exceptions.MechanismDriverError

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2028037/+subscriptions