yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2079802] [NEW] Unable to retrieve instance "..."

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Guillem <2079802@xxxxxxxxxxxxxxxxxx>
Date: Fri, 06 Sep 2024 08:43:31 -0000
Reply-to: Bug 2079802 <2079802@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

We have a deployment of OpenStack in our cluster with a provider network
configured. We launched an Instance a while ago although since last week
we are not able to retrieve instance data from Horizon or the OpenStack
API.

Going through the logs on both Controller and Compute node, we collected
the following errors:

CONTROLLER:

nova-api.log:

- 2024-09-02 10:52:59.851 54640 INFO nova.api.openstack.requestlog [...]
10.0.0.1 "GET /v2.1/servers/detail
limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_k>

- [pid: 54640|app: 0|req: 1/3] 10.0.0.1 () {34 vars in 988 bytes} [Mon
Sep  2 10:52:49 2024] GET
/v2.1/servers/detail/limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_key=display_name&sort_key=uuid
=> generated 104 bytes in 10551 msecs (HTTP/1.1 400) 7 headers in 343
bytes (1 switches on core 0)

- 2024-09-02 10:52:59.842 54640 ERROR nova.network.neutron [None
req-2c374342-0aef-484f-8878-4bf57b34d7cb
7d929138651a4a5aa75517fecdf4b49c 332f0702092741ff8e12ee4658a33c09 - -
default default] Neutron client was not able to generate a valid admin
token, please verify Neutron admin credential located in nova.conf:
neutronclient.common.exceptions. Unauthorized: 401-{'error': {'code':
401, 'title': 'Unauthorized', 'message': 'The request you have made
requires authentication.'}}

- 2024-09-02 10:49:05.064 54636 WARNING oslo_config.cfg [-] Deprecated:
Option "auth_strategy" from group "api" is deprecated for removal
(2024-09-02 10:49:05.346 54636 WARNING oslo_config.cfg [-] Deprecated:
Option "api_servers" from group "glance" is deprecated for removal (


apache2/error.log:

- [Mon Sep 02 10:52:59.861119 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] WARNING horizon.exceptions
Recoverable error: Networking client is experiencing an unauthorized
exception. (HTTP 400) (Request-ID:
req-2c374342-0aef-484f-8878-4bf57b34d7cb), referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/

- [Mon Sep 02 14:00:48.396856 2024] [wsgi:error] [pid 54434:tid 274315788576] [client 10.100.4.94:52630] DEBUG novaclient.v2.client RESP BODY: {"computeFault": {"code": 500, "message": "Unexpected API Error.
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n<class 'nova.exception.NeutronAdminCredentialConfigurationInvalid'>"}}, referer: https://licheenostrum.int.bsc.es/horizon/project/

- [Mon Sep 02 10:53:02.640401 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning:
identity:create_identity_provider deprecated without deprecated_reason
or deprecated_since. This will be an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/

- [Mon Sep 02 10:53:11.440085 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning: update_trunk
deprecated without deprecated_reason or deprecated_since. This will be
an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/

- [Mon Sep 02 10:53:11.401337 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning:
delete_security_group deprecated without deprecated_reason or
deprecated_since. This will be an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/


neutron-api.log:

- 2024-09-02 10:46:45.400 54307 CRITICAL neutron [-] Unhandled error:
KeyError: 'formatters'

- 2024-09-02 10:46:45.513 54309 CRITICAL neutron [-] Unhandled error:
LookupError: No section 'main' (prefixed by 'app' or 'application' or
'composite' or 'composit' or 'pipeline' or 'filter-app') found in config
/etc/neutron/api-paste.ini

- [pid: 54539|app: 0|req: 53/277] 10.0.0.14 () {34 vars in 1038 bytes} [Mon Sep  2 13:06:29 2024] GET /v2.0/ports?device_id=0ae813eb-8d8d-4eac-aada-efa6e8d89597&fields=
 binding%3Ahost_id&fields=binding%3Avif_type => generated 114 bytes in 1976 msecs (HTTP/1.1 401) 4 headers in 204 bytes (1 switches on core 5)

- 2024-09-02 10:49:24.428 54540 WARNING keystonemiddleware.auth_token
[None req-f3d90c96-d832-463a-adf3-4506574eca38 - - - - - -] Configuring
www_authenticate_uri to point to the public identity endpoint is
required; clients may not be able to authenticate against an admin
endpoint


COMPUTE:

nova-compute.log:

- 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager
nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img
failed to execute on /var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-
efa6e8d89597/disk : Unexpected error while running command.

- 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager Stderr:
"qemu-img: Could not open '/var/lib/nova/instances/0ae813eb-8d8d-4eac-
aada-efa6e8d89597/disk': Could not open
'/var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-efa6e8d89597/disk':
Permission denied\n"

- 2024-09-05 13:24:27.741 3592379 ERROR nova.network.neutron [None
req-6b2ff8da-eb8f-4bfb-bf86-6617588c243a - - - - - -] Neutron client was
not able to generate a valid admin token, please verify Neutron admin
credential located in nova.conf:
neutronclient.common.exceptions.Unauthorized: 401-{'error': {'code':
401, 'title': 'Unauthorized', 'message': 'The request you have made
requires authentication.'}}


I tried:

- Making sure my configuration files (nova, neutron, keystone,
openvswitch, ml2_plugin,...) match OpenStack's documentation and fit our
specific setup.

- Different configuration drivers, types, credentials and firewall drivers (openvswitch, iptables_hybrid)
- Checking file and directories permissions (in /var/lib/nova/instances) to make sure qemu is able to access vms.
- Checked that endpoints match our configuration.
- Looked for duplicates or inconsistencies in MariaDB tables.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: neutron nova-network

** Tags added: neutron nova-network

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2079802

Title:
  Unable to retrieve instance "..."

Status in OpenStack Compute (nova):
  New

Bug description:
  We have a deployment of OpenStack in our cluster with a provider
  network configured. We launched an Instance a while ago although since
  last week we are not able to retrieve instance data from Horizon or
  the OpenStack API.

  Going through the logs on both Controller and Compute node, we
  collected the following errors:

  CONTROLLER:

  nova-api.log:

  - 2024-09-02 10:52:59.851 54640 INFO nova.api.openstack.requestlog
  [...] 10.0.0.1 "GET /v2.1/servers/detail
  limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_k>

  - [pid: 54640|app: 0|req: 1/3] 10.0.0.1 () {34 vars in 988 bytes} [Mon
  Sep  2 10:52:49 2024] GET
  /v2.1/servers/detail/limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_key=display_name&sort_key=uuid
  => generated 104 bytes in 10551 msecs (HTTP/1.1 400) 7 headers in 343
  bytes (1 switches on core 0)

  - 2024-09-02 10:52:59.842 54640 ERROR nova.network.neutron [None
  req-2c374342-0aef-484f-8878-4bf57b34d7cb
  7d929138651a4a5aa75517fecdf4b49c 332f0702092741ff8e12ee4658a33c09 - -
  default default] Neutron client was not able to generate a valid admin
  token, please verify Neutron admin credential located in nova.conf:
  neutronclient.common.exceptions. Unauthorized: 401-{'error': {'code':
  401, 'title': 'Unauthorized', 'message': 'The request you have made
  requires authentication.'}}

  - 2024-09-02 10:49:05.064 54636 WARNING oslo_config.cfg [-]
  Deprecated: Option "auth_strategy" from group "api" is deprecated for
  removal (2024-09-02 10:49:05.346 54636 WARNING oslo_config.cfg [-]
  Deprecated: Option "api_servers" from group "glance" is deprecated for
  removal (

  
  apache2/error.log:

  - [Mon Sep 02 10:52:59.861119 2024] [wsgi:error] [pid 54433:tid
  274466857248] [client 10.100.4.35:55124] WARNING horizon.exceptions
  Recoverable error: Networking client is experiencing an unauthorized
  exception. (HTTP 400) (Request-ID:
  req-2c374342-0aef-484f-8878-4bf57b34d7cb), referer:
  https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/

  - [Mon Sep 02 14:00:48.396856 2024] [wsgi:error] [pid 54434:tid 274315788576] [client 10.100.4.94:52630] DEBUG novaclient.v2.client RESP BODY: {"computeFault": {"code": 500, "message": "Unexpected API Error.
  Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n<class 'nova.exception.NeutronAdminCredentialConfigurationInvalid'>"}}, referer: https://licheenostrum.int.bsc.es/horizon/project/

  - [Mon Sep 02 10:53:02.640401 2024] [wsgi:error] [pid 54433:tid
  274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
  packages/oslo_policy/policy.py:1588: DeprecationWarning:
  identity:create_identity_provider deprecated without deprecated_reason
  or deprecated_since. This will be an error in a future release,
  referer:
  https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/

  - [Mon Sep 02 10:53:11.440085 2024] [wsgi:error] [pid 54433:tid
  274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
  packages/oslo_policy/policy.py:1588: DeprecationWarning: update_trunk
  deprecated without deprecated_reason or deprecated_since. This will be
  an error in a future release, referer:
  https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/

  - [Mon Sep 02 10:53:11.401337 2024] [wsgi:error] [pid 54433:tid
  274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
  packages/oslo_policy/policy.py:1588: DeprecationWarning:
  delete_security_group deprecated without deprecated_reason or
  deprecated_since. This will be an error in a future release, referer:
  https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/

  
  neutron-api.log:

  - 2024-09-02 10:46:45.400 54307 CRITICAL neutron [-] Unhandled error:
  KeyError: 'formatters'

  - 2024-09-02 10:46:45.513 54309 CRITICAL neutron [-] Unhandled error:
  LookupError: No section 'main' (prefixed by 'app' or 'application' or
  'composite' or 'composit' or 'pipeline' or 'filter-app') found in
  config /etc/neutron/api-paste.ini

  - [pid: 54539|app: 0|req: 53/277] 10.0.0.14 () {34 vars in 1038 bytes} [Mon Sep  2 13:06:29 2024] GET /v2.0/ports?device_id=0ae813eb-8d8d-4eac-aada-efa6e8d89597&fields=
   binding%3Ahost_id&fields=binding%3Avif_type => generated 114 bytes in 1976 msecs (HTTP/1.1 401) 4 headers in 204 bytes (1 switches on core 5)

  - 2024-09-02 10:49:24.428 54540 WARNING keystonemiddleware.auth_token
  [None req-f3d90c96-d832-463a-adf3-4506574eca38 - - - - - -]
  Configuring www_authenticate_uri to point to the public identity
  endpoint is required; clients may not be able to authenticate against
  an admin endpoint


  COMPUTE:

  nova-compute.log:

  - 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager
  nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img
  failed to execute on /var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-
  efa6e8d89597/disk : Unexpected error while running command.

  - 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager Stderr:
  "qemu-img: Could not open '/var/lib/nova/instances/0ae813eb-8d8d-4eac-
  aada-efa6e8d89597/disk': Could not open
  '/var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-efa6e8d89597/disk':
  Permission denied\n"

  - 2024-09-05 13:24:27.741 3592379 ERROR nova.network.neutron [None
  req-6b2ff8da-eb8f-4bfb-bf86-6617588c243a - - - - - -] Neutron client
  was not able to generate a valid admin token, please verify Neutron
  admin credential located in nova.conf:
  neutronclient.common.exceptions.Unauthorized: 401-{'error': {'code':
  401, 'title': 'Unauthorized', 'message': 'The request you have made
  requires authentication.'}}

  
  I tried:

  - Making sure my configuration files (nova, neutron, keystone,
  openvswitch, ml2_plugin,...) match OpenStack's documentation and fit
  our specific setup.

  - Different configuration drivers, types, credentials and firewall drivers (openvswitch, iptables_hybrid)
  - Checking file and directories permissions (in /var/lib/nova/instances) to make sure qemu is able to access vms.
  - Checked that endpoints match our configuration.
  - Looked for duplicates or inconsistencies in MariaDB tables.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2079802/+subscriptions