yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1954816] Re: Tempest Server Test Fail: tempest.lib.exceptions.ServerFault: Got server fault

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1954816@xxxxxxxxxxxxxxxxxx>
Date: Wed, 12 Apr 2023 09:36:57 -0000
Reply-to: Bug 1954816 <1954816@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Both the RabbitMQ reconnect and the nova->neutron Proxy Error points to
some intermittent network failure in the environment. I think HTTP 500
is an acceptable response from nova in such case. I'm closing this as
this does not seem to be a bug in nova per se.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1954816

Title:
  Tempest Server Test Fail:  tempest.lib.exceptions.ServerFault: Got
  server fault

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  In the tempest tests
  tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_lock_unlock_server
  tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server	
  tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server

  We got the following error:
  -------------------
  Traceback (most recent call last):
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/test.py", line 181, in setUpClass
      raise value.with_traceback(trace)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/test.py", line 174, in setUpClass
      cls.resource_setup()
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/api/compute/servers/test_server_actions.py", line 84, in resource_setup
      cls.server_id = cls.recreate_server(None, validatable=True)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/api/compute/base.py", line 433, in recreate_server
      **kwargs)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/api/compute/base.py", line 272, in create_test_server
      **kwargs)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/common/compute.py", line 199, in create_test_server
      **kwargs)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/lib/services/compute/servers_client.py", line 105, in create_server
      resp, body = self.post('servers', post_body)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/lib/common/rest_client.py", line 299, in post
      return self.request('POST', url, extra_headers, headers, body, chunked)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request
      method, url, extra_headers, headers, body, chunked)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/lib/common/rest_client.py", line 703, in request
      self._error_checker(resp, resp_body)
    File "/home/ubuntu/snap/fcbtest/19/.rally/verification/verifier-ff413e17-4b82-44cd-9717-3650864ec16c/repo/tempest/lib/common/rest_client.py", line 880, in _error_checker
      message=message)
  tempest.lib.exceptions.ServerFault: Got server fault
  Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'neutronclient.common.exceptions.NeutronClientException'>
  -------------------

  Logs indicate a rabbitmqserver failure:
  -------------------
  2021-12-14 11:30:25.545 184592 INFO nova.api.openstack.wsgi [req-8dc14f65-f915-4f1e-92d5-92fcdaa6cefc fe04dc3f609f4b969a0ec09cc608379c 2d218fa5071448ecace6f6837084d739 - 529c49c5853c4c1892560be9a6601bdc 529c49c5853c4c1892560be9a6601bdc] HTTP exception thrown: Flavor m1.small could not be found.
  2021-12-14 11:30:25.562 184593 ERROR oslo.messaging._drivers.impl_rabbit [-] [df1779d8-5671-48bd-95ea-6630fade3477] AMQP server on 192.168.33.231:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
  2021-12-14 11:30:25.578 184593 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection
  2021-12-14 11:30:25.587 184593 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
  2021-12-14 11:30:25.594 184593 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection
  2021-12-14 11:30:25.668 184594 INFO nova.api.openstack.wsgi [req-71f85d94-14f7-433e-9664-95b72c33057e fe04dc3f609f4b969a0ec09cc608379c 2d218fa5071448ecace6f6837084d739 - 529c49c5853c4c1892560be9a6601bdc 529c49c5853c4c1892560be9a6601bdc] HTTP exception thrown: Flavor m1.medium could not be found.
  2021-12-14 11:32:08.417 184592 INFO oslo.messaging._drivers.impl_rabbit [-] [8a2db959-87d5-4bdc-83b9-f5ad29e647de] Reconnected to AMQP server on 192.168.33.201:5672 via [amqp] client with port 52838.
  2021-12-14 11:32:12.847 184593 INFO oslo.messaging._drivers.impl_rabbit [-] [df1779d8-5671-48bd-95ea-6630fade3477] Reconnected to AMQP server on 192.168.33.231:5672 via [amqp] client with port 57392.
  2021-12-14 11:32:13.697 184594 INFO oslo.messaging._drivers.impl_rabbit [-] [600c8f9a-f55d-4e38-8d38-5877249f4b51] Reconnected to AMQP server on 192.168.33.195:5672 via [amqp] client with port 50156.
  2021-12-14 11:32:17.045 184591 INFO oslo.messaging._drivers.impl_rabbit [-] [b3ab0f0d-4c7c-4f55-bdf1-518f5bbb6b05] Reconnected to AMQP server on 192.168.33.231:5672 via [amqp] client with port 44666.
  2021-12-14 11:32:25.051 184594 INFO nova.api.openstack.compute.server_external_events [req-44b22f72-7fca-443d-b36a-88ac05ef4b97 20924cca8caf4750b1a9d3b5f5cc8600 325635c3bfb945a68728988bcc29ec4d - a6f2df99d2c44e759cd76505a389cdf3 a6f2df99d2c44e759cd76505a389cdf3] Creating event network-changed:207dbefc-3498-4ea9-a7dd-2806666172a4 for instance cbb79baf-9efc-45a5-950d-c0f25f85b4fe on azurill.prodymcprodface.solutionsqa
  2021-12-14 11:32:55.246 184593 INFO nova.api.openstack.wsgi [req-1189d9e2-b8c1-4a4d-b310-422b5bdeff9b 981bcfc847984bb687356ca814e587ec a17e695c5988492198d84af48ab7fd5a - 529c49c5853c4c1892560be9a6601bdc 529c49c5853c4c1892560be9a6601bdc] HTTP exception thrown: Instance cbb79baf-9efc-45a5-950d-c0f25f85b4fe could not be found.

  ...

  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.NeutronClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi <html><head>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi <title>502 Proxy Error</title>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi </head><body>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi <h1>Proxy Error</h1>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi <p>The proxy server received an invalid
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi response from an upstream server.<br />
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi <hr>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi <address>Apache/2.4.41 (Ubuntu) Server at neutron-internal.prodymcprodface.solutionsqa Port 9696</address>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi </body></html>
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi
  2021-12-14 11:33:10.267 184594 ERROR nova.api.openstack.wsgi
  -------------------

  Full collection of collected logs/artifacts/crashdumps:
  https://oil-jenkins.canonical.com/artifacts/6ba06c69-5641-47a3-a6d8-d5e915071724/index.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1954816/+subscriptions
References

[Bug 1954816] [NEW] Tempest Server Test Fail: tempest.lib.exceptions.ServerFault: Got server fault
From: Bas de Bruijne, 2021-12-14