yahoo-eng-team team mailing list archive

[OpenStack Infra <1535616@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/269536
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a40ee2ce1b887751a91cc373cdd6c3415fb3e2d7
Submitter: Jenkins
Branch:    master

commit a40ee2ce1b887751a91cc373cdd6c3415fb3e2d7
Author: Bob Ball <bob.ball@xxxxxxxxxx>
Date:   Tue Jan 19 10:11:10 2016 +0000

    XenAPI: Fix race on rotate_xen_guest_logs
    
    There is a race between where Xen creates the domain and starts logging
    and when XAPI returns to say that the domain has started.
    
    If the cronjob to rotate the guest logs runs in this period, it can
    delete the console logs before we set the last_dom_id which is used to
    preserve logs.
    
    This change adds a fixed delay before deleting any log files, allowing
    the VM start operation to return after the domain has been successfully
    created.
    
    Change-Id: I82ae3e008974f33b60256366f171abf1f0369e60
    Closes-Bug: 1535616


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535616

Title:
  XenAPI: Console logging race condition

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Unexpected API Error in XenServer CI
  test_get_console_output_with_unlimited_size

  http://dd6b71949550285df7dc-
  dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/22/267822/1/2904/index.html
  failed with a race condition getting the guest console logs:

  VM started:
  Jan 14 22:03:58 localhost xapi: [debug|localhost.localdomain|1158 INET 0.0.0.0:80|VM.start_on R:98346b008094|audit] VM.start_on: VM = '9bbcbe52-ca94-37df-2791-e49d8dfae4c7 (instance-00000042)'; host 'd77c538b-afe9-4e0c-b5f1-223fed0f9728 (localhost.localdomain)'

  Cron job kicks in, identifies that domain 88 isn't "valid" and deletes the log file
  Jan 14 22:04:01 localhost rotate_xen_guest_logs: Valid dom IDs: 81 87 2 85 86 83 0 82
  Jan 14 22:04:01 localhost rotate_xen_guest_logs: Deleting: /var/log/xen/guest/console.88

  VM start completed (XAPI returns):
  Jan 14 22:04:03 localhost xapi: [debug|localhost.localdomain|1158 INET 0.0.0.0:80|VM.start_on R:98346b008094|xapi] Message.create VM_STARTED 5 VM 9bbcbe52-ca94-37df-2791-e49d8dfae4c7

  Attempt to preserve the log file for domain 88 made:
  Jan 14 22:05:01 localhost xapi: [ info|localhost.localdomain|55940 UNIX /var/xapi/xapi||cli] xe vm-param-set uuid=9bbcbe52-ca94-37df-2791-e49d8dfae4c7 other-config:last_dom_id=88 username=root password=(omitted)

  The console log exists _before_ XAPI has completed starting the
  domain, so we can't preserve the log file using the last_dom_id flag.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1535616/+subscriptions