← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93845

[Bug 2061368] [NEW] The placement response exceeded the gateway timeout when moving allocations

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
When executing the move_allocations method,
request the POST /allocations endpoint of the placement.

If the processing time exceeds the gateway timeout value,
A 504 Gateway Timeout HTTP response will be returned.

The response in move_allocations is unexpected and
cannot be deserialized as a JSON string.

In fact, all 5xx responses are not returned by placement,
so they do not have the correct data structure and should
not be processed as JSON strings.

Steps to reproduce
==================
Make POST /allocations endpoint(placement) timed out.

Expected result
===============
Nova handles the 5xx exception.

Actual result
=============
JSONDecodeError

Environment
===========
1. Exact version of OpenStack you are running. See the following
   Openstack Nova Wallaby.

2. Which hypervisor did you use?
   Libvirt + KVM

2. Which storage type did you use?
   Ceph

3. Which networking type did you use?
   Neutron

Logs & Configs
==============

nova
```
2023-12-08 11:25:07.155 +0800 ¦ node-2 ¦ nova-compute-g8r6v ¦ nova-compute ¦ 2023-12-08T11:25:07.155062613+08:00 stdout F 2023-12-08 11:25:07.154 56430 ERROR root [-] Original exception being dropped: 
['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 202, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 9249, in _rollback_live_migration\n    self._revert_allocation(context, instance, migration)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 5181, in _revert_allocation\n    instance.uuid)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 73, in wrapper\n    return f(self, *a, **k)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 115, in wrapper\n    return f(self, *a, **k)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 1990, in move_allocations\n    err = r.json()[\'errors\'][0]\n', '  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 900, in json\n    return complexjson.loads(self.text, **kwargs)\n', '  File "/usr/local/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads\n    return _default_decoder.decode(s)\n', '  File "/usr/local/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode\n    obj, end = self.raw_decode(s)\n', '  File "/usr/local/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode\n    return self.scan_once(s, idx=_w(s, idx).end())\n', 'simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n']: 
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3854d318dd64b09b41d1e4cf59b7087
```

placement
```
2023-12-08 11:24:07.147 +0800 ¦ node-1 ¦ nova-placement-api-767f46cd6f-2m574 ¦ nova-placement-api ¦ 2023-12-08T11:24:07.147043922+08:00 stdout F 2023-12-08 11:24:07.146930 Timeout when reading response headers from daemon process 'placement-api': /var/www/cgi-bin/placement/placement-api
2023-12-08 11:24:07.147 +0800 ¦ node-1 ¦ nova-placement-api-767f46cd6f-2m574 ¦ nova-placement-api ¦ 2023-12-08T11:24:07.147081599+08:00 stdout F 35.168.10.4 - - [08/Dec/2023:11:23:07 +0800] "POST /allocations HTTP/1.1" 504 247 "-" "nova-compute keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8"
```

** Affects: nova
     Importance: Undecided
     Assignee: Haidong Pang (haidong-pang)
         Status: In Progress

** Changed in: nova
       Status: New => In Progress

** Changed in: nova
     Assignee: (unassigned) => Haidong Pang (haidong-pang)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2061368

Title:
  The placement response exceeded the gateway timeout when moving
  allocations

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  When executing the move_allocations method,
  request the POST /allocations endpoint of the placement.

  If the processing time exceeds the gateway timeout value,
  A 504 Gateway Timeout HTTP response will be returned.

  The response in move_allocations is unexpected and
  cannot be deserialized as a JSON string.

  In fact, all 5xx responses are not returned by placement,
  so they do not have the correct data structure and should
  not be processed as JSON strings.

  Steps to reproduce
  ==================
  Make POST /allocations endpoint(placement) timed out.

  Expected result
  ===============
  Nova handles the 5xx exception.

  Actual result
  =============
  JSONDecodeError

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
     Openstack Nova Wallaby.

  2. Which hypervisor did you use?
     Libvirt + KVM

  2. Which storage type did you use?
     Ceph

  3. Which networking type did you use?
     Neutron

  Logs & Configs
  ==============

  nova
  ```
  2023-12-08 11:25:07.155 +0800 ¦ node-2 ¦ nova-compute-g8r6v ¦ nova-compute ¦ 2023-12-08T11:25:07.155062613+08:00 stdout F 2023-12-08 11:25:07.154 56430 ERROR root [-] Original exception being dropped: 
  ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 202, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 9249, in _rollback_live_migration\n    self._revert_allocation(context, instance, migration)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 5181, in _revert_allocation\n    instance.uuid)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 73, in wrapper\n    return f(self, *a, **k)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 115, in wrapper\n    return f(self, *a, **k)\n', '  File "/usr/local/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 1990, in move_allocations\n    err = r.json()[\'errors\'][0]\n', '  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 900, in json\n    return complexjson.loads(self.text, **kwargs)\n', '  File "/usr/local/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads\n    return _default_decoder.decode(s)\n', '  File "/usr/local/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode\n    obj, end = self.raw_decode(s)\n', '  File "/usr/local/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode\n    return self.scan_once(s, idx=_w(s, idx).end())\n', 'simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n']: 
  oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID b3854d318dd64b09b41d1e4cf59b7087
  ```

  placement
  ```
  2023-12-08 11:24:07.147 +0800 ¦ node-1 ¦ nova-placement-api-767f46cd6f-2m574 ¦ nova-placement-api ¦ 2023-12-08T11:24:07.147043922+08:00 stdout F 2023-12-08 11:24:07.146930 Timeout when reading response headers from daemon process 'placement-api': /var/www/cgi-bin/placement/placement-api
  2023-12-08 11:24:07.147 +0800 ¦ node-1 ¦ nova-placement-api-767f46cd6f-2m574 ¦ nova-placement-api ¦ 2023-12-08T11:24:07.147081599+08:00 stdout F 35.168.10.4 - - [08/Dec/2023:11:23:07 +0800] "POST /allocations HTTP/1.1" 504 247 "-" "nova-compute keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8"
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2061368/+subscriptions
  Thread Previous • Date Previous • • Thread Next