yahoo-eng-team team mailing list archive

[OpenStack Infra <1525002@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/256375
Committed: https://git.openstack.org/cgit/openstack/python-ironicclient/commit/?id=8224859c02f525fa3b7dbe917d56b889b0dbb073
Submitter: Jenkins
Branch:    master

commit 8224859c02f525fa3b7dbe917d56b889b0dbb073
Author: Dmitry Tantsur <dtantsur@xxxxxxxxxx>
Date:   Fri Dec 11 12:38:39 2015 +0100

    Scale back on how many warnings we issue
    
    * Remove "Request returned failure status" warning, it does not carry
      any helpful information for an operator (or anyone).
    * Demote retry warnings to debug level. Retrying is essentially a normal
      thing in the real world, and currently we're spamming logs with warnings.
      http://logs.openstack.org/93/255793/3/gate/gate-tempest-dsvm-ironic-agent_ssh/25175ed/logs/screen-n-cpu.txt.gz?level=TRACE
    
    Change-Id: I6c40581b9dc5ab1fe3af4b882ede17a13edacc66
    Closes-Bug: #1525002


** Changed in: python-ironicclient
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1525002

Title:
  The ironic driver needs to scale back how many errors it traces out

Status in Ironic:
  In Progress
Status in OpenStack Compute (nova):
  Confirmed
Status in python-ironicclient:
  Fix Released

Bug description:
  The amount of tracing in this n-cpu log is a bit much:

  http://logs.openstack.org/93/255793/3/gate/gate-tempest-dsvm-ironic-
  agent_ssh/25175ed/logs/screen-n-cpu.txt.gz?level=TRACE

  Like these warnings:

  http://logs.openstack.org/93/255793/3/gate/gate-tempest-dsvm-ironic-
  agent_ssh/25175ed/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-12-10_16_11_48_799

  2015-12-10 16:11:48.798 WARNING ironicclient.common.http [req-94077ab9-5adf-4720-9fb8-2e027dde9b72 tempest-BaremetalBasicOps-1285004585 tempest-BaremetalBasicOps-1966762451] Request returned failure status.
  2015-12-10 16:11:48.799 WARNING ironicclient.common.http [req-94077ab9-5adf-4720-9fb8-2e027dde9b72 tempest-BaremetalBasicOps-1285004585 tempest-BaremetalBasicOps-1966762451] Error contacting Ironic server: Node 1 is locked by host localhost, please retry after the current operation is completed.
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 150, in inner
      return func(*args, **kwargs)

    File "/opt/stack/new/ironic/ironic/conductor/manager.py", line 1519, in update_port
      purpose='port update') as task:

    File "/opt/stack/new/ironic/ironic/conductor/task_manager.py", line 152, in acquire
      driver_name=driver_name, purpose=purpose)

    File "/opt/stack/new/ironic/ironic/conductor/task_manager.py", line 222, in __init__
      self.release_resources()

    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/opt/stack/new/ironic/ironic/conductor/task_manager.py", line 203, in __init__
      self._lock()

    File "/opt/stack/new/ironic/ironic/conductor/task_manager.py", line 243, in _lock
      reserve_node()

    File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
      return Retrying(*dargs, **dkw).call(f, *args, **kw)

    File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
      raise attempt.get()

    File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
      six.reraise(self.value[0], self.value[1], self.value[2])

    File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
      attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

    File "/opt/stack/new/ironic/ironic/conductor/task_manager.py", line 236, in reserve_node
      self.node_id)

    File "/opt/stack/new/ironic/ironic/objects/node.py", line 260, in reserve
      db_node = cls.dbapi.reserve_node(tag, node_id)

    File "/opt/stack/new/ironic/ironic/db/sqlalchemy/api.py", line 226, in reserve_node
      host=node['reservation'])

  NodeLocked: Node 1 is locked by host localhost, please retry after the current operation is completed.
   (HTTP 409). Attempt 1 of 61

  UPD from dtantsur:
  The traceback has nothing to do with ironic client or driver. It gets added somewhere between ironic conductor and API levels.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ironic/+bug/1525002/+subscriptions