yahoo-eng-team team mailing list archive

[Dan Emmons <dan.emmons@xxxxxxxxxxxxx>]

Public bug reported:

When the XenAPI Glance plugin encounters a retryable error, it logs a
message stating that it is retrying the action. This was added when
retries were only implemented on uploads, but the retry logic was later
moved up a level to be usable for all glance plugin functions. I've
noticed that the messages logged indicate they will retry vhd_upload
regardless of the Glance plugin function actually being used, most
obvious during a vhd_download:

Dec 30 19:22:42  DEBUG nova.virt.xenapi.client.session Got exception:
['XENAPI_PLUGIN_FAILURE', 'signal: SIGTERM', '', '']
_unwrap_plugin_exceptions /nova/virt/xenapi/client/session.py:249

Dec 30 19:22:42  DEBUG nova.virt.xenapi.client.session  Error due to a
signal, retrying upload_vhd _is_retryable_exception
/nova/virt/xenapi/client/session.py:232

Dec 30 19:22:42  WARNING nova.virt.xenapi.client.session
glance.download_vhd failed. Retrying call.

Dec 30 19:22:42  INFO nova.virt.xenapi.client.session
glance.download_vhd attempt 2/6

This example was in a Rackspace environment, so I have redacted a few
things from the logs, but the issue is clear and easily reproducible.
For this test I built a new instance, logged into the host that the
build started on, and killed the glance process with a SIGTERM so it
would retry and log the message. I've confirmed that the issue in the
code is still present in the current trunk in GitHub.

This will be trivial to fix and I plan to assign this bug to myself and
submit the code patch. Essentially, my proposed solution is to add an
optional argument to _is_retryable_exception() to specify the function,
and then reference that if it is set or use a generic message otherwise.

** Affects: nova
     Importance: Undecided
     Assignee: Dan Emmons (dan-emmons)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1265054

Title:
  XenAPI Glance Plugin logs incorrect message when retrying download

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  When the XenAPI Glance plugin encounters a retryable error, it logs a
  message stating that it is retrying the action. This was added when
  retries were only implemented on uploads, but the retry logic was
  later moved up a level to be usable for all glance plugin functions.
  I've noticed that the messages logged indicate they will retry
  vhd_upload regardless of the Glance plugin function actually being
  used, most obvious during a vhd_download:

  Dec 30 19:22:42  DEBUG nova.virt.xenapi.client.session Got exception:
  ['XENAPI_PLUGIN_FAILURE', 'signal: SIGTERM', '', '']
  _unwrap_plugin_exceptions /nova/virt/xenapi/client/session.py:249

  Dec 30 19:22:42  DEBUG nova.virt.xenapi.client.session  Error due to a
  signal, retrying upload_vhd _is_retryable_exception
  /nova/virt/xenapi/client/session.py:232

  Dec 30 19:22:42  WARNING nova.virt.xenapi.client.session
  glance.download_vhd failed. Retrying call.

  Dec 30 19:22:42  INFO nova.virt.xenapi.client.session
  glance.download_vhd attempt 2/6

  This example was in a Rackspace environment, so I have redacted a few
  things from the logs, but the issue is clear and easily reproducible.
  For this test I built a new instance, logged into the host that the
  build started on, and killed the glance process with a SIGTERM so it
  would retry and log the message. I've confirmed that the issue in the
  code is still present in the current trunk in GitHub.

  This will be trivial to fix and I plan to assign this bug to myself
  and submit the code patch. Essentially, my proposed solution is to add
  an optional argument to _is_retryable_exception() to specify the
  function, and then reference that if it is set or use a generic
  message otherwise.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1265054/+subscriptions