yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #67202
[Bug 1714208] Re: Router/network creation in HA mode fails in Pike
The issue was fixed with MariaDB 10.1.X.
I think we can close this issue, unless it pops up again.
** Changed in: neutron
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1714208
Title:
Router/network creation in HA mode fails in Pike
Status in neutron:
Invalid
Bug description:
I have set up OpenStack Pike on Ubuntu 16.04 in HA mode (2
controllers, 3 compute node). In the current Pike release, router and
network creation in HA mode fails. Whereas, creating routers in non-HA
mode succeeds.
The neutron-server.log gives me the following:
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters [req-5e467987-dce5-4379-a5e9-2192a3a43702 3a5eebf84f7543fc832ef095a581c9bf e02e5f2794154037b756aaf366a4f80d - default default] DBAPIError exception wrapped from (pymysql.err.InternalError) (4025, u'CONSTRAINT `CONSTRAINT_3` failed for `neutron`.`networks`') [SQL: u'INSERT INTO networks (project_id, id, name, status, admin_state_up, vlan_transparent, availability_zone_hints, standard_attr_id) VALUES (%(project_id)s, %(id)s, %(name)s, %(status)s, %(admin_state_up)s, %(vlan_transparent)s, %(availability_zone_hints)s, %(standard_attr_id)s)'] [parameters: {'status': 'ACTIVE', 'availability_zone_hints': None, 'name': u'HA network tenant e02e5f2794154037b756aaf366a4f80d', 'admin_state_up': 1, 'vlan_transparent': None, 'standard_attr_id': 43, 'project_id': '', 'id': 'fb3f515d-26de-4872-aa77-28f9aebecedb'}]: InternalError: (4025, u'CONSTRAINT `CONSTRAINT_3` failed for `neutron`.`networks`')
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters context)
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 166, in execute
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 322, in _query
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 852, in query
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1053, in _read_query_result
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters result.read()
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1336, in read
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1010, in _read_packet
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in check_error
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval)
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (4025, u'CONSTRAINT `CONSTRAINT_3` failed for `neutron`.`networks`')
2017-08-31 11:05:38.736 4158 ERROR oslo_db.sqlalchemy.exc_filters
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource [req-5e467987-dce5-4379-a5e9-2192a3a43702 3a5eebf84f7543fc832ef095a581c9bf e02e5f2794154037b756aaf366a4f80d - default default] create failed: No details.: CallbackFailure: Callback neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._before_router_create--9223372036853416976 failed with "(pymysql.err.InternalError) (4025, u'CONSTRAINT `CONSTRAINT_3` failed for `neutron`.`networks`') [SQL: u'INSERT INTO networks (project_id, id, name, status, admin_state_up, vlan_transparent, availability_zone_hints, standard_attr_id) VALUES (%(project_id)s, %(id)s, %(name)s, %(status)s, %(admin_state_up)s, %(vlan_transparent)s, %(availability_zone_hints)s, %(standard_attr_id)s)'] [parameters: {'status': 'ACTIVE', 'availability_zone_hints': None, 'name': u'HA network tenant e02e5f2794154037b756aaf366a4f80d', 'admin_state_up': 1, 'vlan_transparent': None, 'standard_attr_id': 43, 'project_id': '', 'id': 'fb3f515d-26de-4872-aa77-28f9aebecedb'}]"
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 93, in resource
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource result = method(request=request, **args)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 435, in create
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 94, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 90, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 129, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 125, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 548, in _create
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource obj = do_create(body)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 530, in do_create
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 523, in do_create
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/l3_db.py", line 1833, in create_router
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource router)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 164, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return method(*args, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 94, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 90, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 129, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 125, in wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/l3_db.py", line 232, in create_router
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource transaction=False)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/_utils.py", line 62, in safe_creation
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource obj = create_fn()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/l3_db.py", line 198, in _create_router_db
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self, context=context, router=router)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lib/callbacks/registry.py", line 53, in notify
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource _get_callback_manager().notify(resource, event, trigger, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lib/db/utils.py", line 105, in _wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource raise db_exc.RetryRequest(e)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lib/db/utils.py", line 100, in _wrapped
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource return function(*args, **kwargs)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 152, in notify
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource raise exceptions.CallbackFailure(errors=errors)
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource CallbackFailure: Callback neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._before_router_create--9223372036853416976 failed with "(pymysql.err.InternalError) (4025, u'CONSTRAINT `CONSTRAINT_3` failed for `neutron`.`networks`') [SQL: u'INSERT INTO networks (project_id, id, name, status, admin_state_up, vlan_transparent, availability_zone_hints, standard_attr_id) VALUES (%(project_id)s, %(id)s, %(name)s, %(status)s, %(admin_state_up)s, %(vlan_transparent)s, %(availability_zone_hints)s, %(standard_attr_id)s)'] [parameters: {'status': 'ACTIVE', 'availability_zone_hints': None, 'name': u'HA network tenant e02e5f2794154037b756aaf366a4f80d', 'admin_state_up': 1, 'vlan_transparent': None, 'standard_attr_id': 43, 'project_id': '', 'id': 'fb3f515d-26de-4872-aa77-28f9aebecedb'}]"
2017-08-31 11:05:38.800 4158 ERROR neutron.api.v2.resource
I think there is a foreign key violation because the table
"standardattributes" does not contain a vaild entry with the
corresponding id (in this case "43").
This is my neutron-conf:
[DEFAULT]
allow_overlapping_ips = true
auth_strategy = keystone
base_mac = 02:05:69:00:00:00
bind_host = 10.30.200.101
bind_port = 9696
core_plugin = ml2
debug = false
default_log_levels=amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=WARN,oslo.messaging=WARN,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=WARN,dogpile.core.dogpile=WARN,oslo_service=WARN,neutron=WARN
dhcp_agents_per_network = 2
dns_domain = openstack.mycompany.com.
dvr_base_mac = 0A:05:69:00:00:00
endpoint_type = internalURL
host = os-network01
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
l3_ha = true
l3_ha_net_cidr = 169.254.192.0/18
log_dir = /var/log/neutron
max_l3_agents_per_router = 2
min_l3_agents_per_router = 2
notify_nova_on_port_data_changes = true
notify_nova_on_port_status_changes = true
router_distributed = true
service_plugins = router,firewall,qos,lbaasv2
state_path = /var/lib/neutron
transport_url = rabbit://neutron:neutronpass@os-rabbit01:5672,neutron:neutronpass@os-rabbit02:5672/openstack
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[database]
connection = mysql+pymysql://neutron:neutronDBpass@os-controller/neutron
max_retries = -1
[keystone_authtoken]
auth_type = password
auth_uri = http://os-identity:5000
auth_url = http://os-identity:35357
memcached_servers = os-memcache:11211
password = neutronpass
project_domain_name = default
project_name = service
user_domain_name = default
username = neutron
[nova]
auth_type = password
auth_url = http://os-identity:35357
endpoint_type = internal
password = novapass
project_domain_name = default
project_name = service
region_name = RegionOne
user_domain_name = default
username = nova
[oslo_concurrency]
lock_path = /var/lock/neutron
[oslo_messaging_notifications]
driver = messagingv2
[oslo_messaging_rabbit]
amqp_durable_queues = true
rabbit_ha_queues = true
rabbit_retry_backoff = 2
rabbit_retry_interval = 1
[oslo_middleware]
enable_proxy_headers_parsing = true
[service_providers]
service_provider = FIREWALL:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default
service_provider = LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
I think there is problem in router and network creation in combination with HA. But even with
l3_ha = false
dhcp_agents_per_network = 1
max_l3_agents_per_router = 1
min_l3_agents_per_router = 1
network creation still fails.
May someone please have a look?
Regards,
Jens
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1714208/+subscriptions
References