yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1597848] [NEW] Resource has ERROR status: Exceeded maximum number of retries

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrew Kalach <akndex@xxxxxxxxx>
Date: Thu, 30 Jun 2016 17:04:30 -0000
Reply-to: Bug 1597848 <1597848@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
During 9.0RC2 Certification Testing Sutie run, the following error log appeared when
executing "NovaSecGroup.boot_and_delete_server_with_secgroups" task:

2016-06-28 01:58:17.619 30929 DEBUG novaclient.v2.client [-] GET call to compute for http://172.16.184.3:8774/v2.1/servers/e3a9f1fd-d7a9-43af-a911-903297545f4e used request id req-33956e34-3b21-4f97-a0ae-fc975f9cc6b5 _log_request_id /opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py:85
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner [-] Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status.
Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'}
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Traceback (most recent call last):
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     deprecated_output = getattr(scenario_inst, method_name)(**kwargs)
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/security_group.py", line 137, in boot_and_delete_server_with_secgroups
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     **kwargs)
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     f = func(self, *args, **kwargs)
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 147, in _boot_server
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     check_interval=CONF.benchmark.nova_server_boot_poll_interval
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/common/logging.py", line 236, in wrapper
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     return f(*args, **kwargs)
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 147, in wait_for
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     check_interval=check_interval)
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 211, in wait_for_status
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     resource = update_resource(resource)
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 90, in _get_from_manager
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     fault=getattr(res, "fault", "n/a"))
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner GetResourceErrorStatus: Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status.
2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'}

Steps to reproduce
==================
Install Rally on masternode and run task "NovaSecGroup.boot_and_delete_server_with_secgroups":

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start /opt/stack/rally-scenarios/nova/boot_and_delete_server_with_secgroups.yaml

Expected result
===============
Task run without errors

Actual result
=============
Errors described above in logs

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  199 compute nodes
  20 ceph nodes
  DVR enabled
  XVLAN
Image RC2 build:
  fuel-9.0-mos-495-2016-06-16_18-18-00.iso

Diagnostic snapshot:
  http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "Logs and YAML scenario"
   https://bugs.launchpad.net/bugs/1597848/+attachment/4692924/+files/attachment.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1597848

Title:
  Resource has ERROR status: Exceeded maximum number of retries

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  During 9.0RC2 Certification Testing Sutie run, the following error log appeared when
  executing "NovaSecGroup.boot_and_delete_server_with_secgroups" task:

  2016-06-28 01:58:17.619 30929 DEBUG novaclient.v2.client [-] GET call to compute for http://172.16.184.3:8774/v2.1/servers/e3a9f1fd-d7a9-43af-a911-903297545f4e used request id req-33956e34-3b21-4f97-a0ae-fc975f9cc6b5 _log_request_id /opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py:85
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner [-] Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status.
  Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'}
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Traceback (most recent call last):
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     deprecated_output = getattr(scenario_inst, method_name)(**kwargs)
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/security_group.py", line 137, in boot_and_delete_server_with_secgroups
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     **kwargs)
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     f = func(self, *args, **kwargs)
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 147, in _boot_server
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     check_interval=CONF.benchmark.nova_server_boot_poll_interval
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/common/logging.py", line 236, in wrapper
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     return f(*args, **kwargs)
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 147, in wait_for
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     check_interval=check_interval)
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 211, in wait_for_status
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     resource = update_resource(resource)
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner   File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 90, in _get_from_manager
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner     fault=getattr(res, "fault", "n/a"))
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner GetResourceErrorStatus: Resource <Server: s_rally_02cfda0f_nOe66YVK> has ERROR status.
  2016-06-28 01:58:17.623 30929 ERROR rally.task.runner Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance e3a9f1fd-d7a9-43af-a911-903297545f4e. Last exception: Binding failed for port c3c6e915-3395-4d6f-90bf-02a565da4146, please check neutron logs for more information.', u'code': 500, u'created': u'2016-06-28T01:58:16Z'}

  Steps to reproduce
  ==================
  Install Rally on masternode and run task "NovaSecGroup.boot_and_delete_server_with_secgroups":

  1. cd /opt/stack
  2. source .venv/bin/activate
  3. rally task start /opt/stack/rally-scenarios/nova/boot_and_delete_server_with_secgroups.yaml

  Expected result
  ===============
  Task run without errors

  Actual result
  =============
  Errors described above in logs

  Environment
  ===========
  Scale 200-node lab (ENV-10):
    3 controllers
    199 compute nodes
    20 ceph nodes
    DVR enabled
    XVLAN
  Image RC2 build:
    fuel-9.0-mos-495-2016-06-16_18-18-00.iso

  Diagnostic snapshot:
    http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1597848/+subscriptions
Follow ups

[Bug 1597848] Re: Resource has ERROR status: Exceeded maximum number of retries
From: Michael Semenov, 2016-06-30