yahoo-eng-team team mailing list archive

[Michal Nasiadka <mnasiadka@xxxxxxxxx>]

Public bug reported:

Neutron fails with the following error:
2020-02-14 15:08:21.663 29 CRITICAL neutron [req-b7615e16-e470-4444-8342-86fca030273c - - - - -] Unhandled error: oslo_db.exception.DBError: (pymysql.err.InternalError) (1047, 'WSREP has not yet prepared node for application use')
[SQL: SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents.`binary` AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.availability_zone AS agents_availability_zone, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations, agents.resource_versions AS agents_resource_versions, agents.`load` AS agents_load, agents.resources_synced AS agents_resources_synced 
FROM agents 
WHERE agents.admin_state_up IN (%(admin_state_up_1)s)]
[parameters: {'admin_state_up_1': 1}]
(Background on this error at: http://sqlalche.me/e/2j85)

We see that mainly in the kolla-ansible upgrade CI - we upgrade the
single node Galera instance - and it seems Oslo.db is stuck in claiming
the 1047 error on the connection. Once neutron-server container gets
restarted - it works properly.

CI logs of neutron-server:
https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron
/neutron-server.txt#4361

Critical event related to DB:
https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron/neutron-server.txt#4175

After neutron-server restart as part of the upgrade job - everything starts working fine:
https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron/neutron-server.txt#4373

** Affects: neutron
     Importance: Undecided
         Status: Confirmed

** Affects: oslo.db
     Importance: Undecided
         Status: Confirmed

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1863579

Title:
  Unhandled error - WSREP has not yet prepared node for application use

Status in neutron:
  Confirmed
Status in oslo.db:
  Confirmed

Bug description:
  Neutron fails with the following error:
  2020-02-14 15:08:21.663 29 CRITICAL neutron [req-b7615e16-e470-4444-8342-86fca030273c - - - - -] Unhandled error: oslo_db.exception.DBError: (pymysql.err.InternalError) (1047, 'WSREP has not yet prepared node for application use')
  [SQL: SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents.`binary` AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.availability_zone AS agents_availability_zone, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations, agents.resource_versions AS agents_resource_versions, agents.`load` AS agents_load, agents.resources_synced AS agents_resources_synced 
  FROM agents 
  WHERE agents.admin_state_up IN (%(admin_state_up_1)s)]
  [parameters: {'admin_state_up_1': 1}]
  (Background on this error at: http://sqlalche.me/e/2j85)

  We see that mainly in the kolla-ansible upgrade CI - we upgrade the
  single node Galera instance - and it seems Oslo.db is stuck in
  claiming the 1047 error on the connection. Once neutron-server
  container gets restarted - it works properly.

  CI logs of neutron-server:
  https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron
  /neutron-server.txt#4361

  Critical event related to DB:
  https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron/neutron-server.txt#4175

  After neutron-server restart as part of the upgrade job - everything starts working fine:
  https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron/neutron-server.txt#4373

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1863579/+subscriptions