yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

Public bug reported:

We're now running neutron by default in CI jobs for ocata if they aren't
explicitly specified otherwise to run nova-network.

That might be unrelated to this, but I saw the citrix xenserver neutron
job fail today with a vif plugging timeout:

http://dd6b71949550285df7dc-
dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/47/395747/8/check
/dsvm-tempest-neutron-network/e3832fe/logs/screen-n-cpu.txt.gz

2016-11-11 00:55:41.191 3175 WARNING nova.virt.xenapi.vmops [req-
5f5774e3-92ee-4e49-947d-ce1c879bc1ab tempest-
ServersTestManualDisk-1338550150 tempest-
ServersTestManualDisk-1338550150] [instance: 79c00d5c-b285-44e5-b8db-
9fc9a8e31478] Timeout waiting for vif plugging callback

nova.conf seems to be configured correctly:

http://dd6b71949550285df7dc-
dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/47/395747/8/check
/dsvm-tempest-neutron-network/e3832fe/logs/etc/nova/nova.conf.txt.gz

vif_plugging_timeout = 300
vif_plugging_is_fatal = True

So I'm wondering if there is a problem in the vif plugging callback
logic in the xenapi driver code.

I see this earlier before the timeout:

2016-11-11 00:50:29.526 3175 DEBUG nova.compute.manager [req-5f5774e3
-92ee-4e49-947d-ce1c879bc1ab tempest-ServersTestManualDisk-1338550150
tempest-ServersTestManualDisk-1338550150] [instance: 79c00d5c-b285-44e5
-b8db-9fc9a8e31478] Preparing to wait for external event network-vif-
plugged-b6c3af4c-98cc-4077-8d4d-7009835c0c5c prepare_for_instance_event
/opt/stack/new/nova/nova/compute/manager.py:324

2016-11-11 00:50:29.527 3175 DEBUG nova.compute.manager [req-5f5774e3
-92ee-4e49-947d-ce1c879bc1ab tempest-ServersTestManualDisk-1338550150
tempest-ServersTestManualDisk-1338550150] [instance: 79c00d5c-b285-44e5
-b8db-9fc9a8e31478] Preparing to wait for external event network-vif-
plugged-f5d2ac5a-36bf-4562-b2d0-18c40f640a3c prepare_for_instance_event
/opt/stack/new/nova/nova/compute/manager.py:324

2016-11-11 00:50:29.528 3175 DEBUG nova.virt.xenapi.vmops [req-5f5774e3
-92ee-4e49-947d-ce1c879bc1ab tempest-ServersTestManualDisk-1338550150
tempest-ServersTestManualDisk-1338550150] wait for instance event
:[('network-vif-plugged', u'b6c3af4c-98cc-4077-8d4d-7009835c0c5c'),
('network-vif-plugged', u'f5d2ac5a-36bf-4562-b2d0-18c40f640a3c')] _spawn
/opt/stack/new/nova/nova/virt/xenapi/vmops.py:599

Then it starts doing the vif plugging. The odd thing is it logs twice
that it's preparing to wait for external events, but only logs 'waiting
for instance event' once. So is it waiting for another even that doesn't
happen?

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: neutron xenserver

** Tags added: neutron xenserver

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1640993

Title:
  xenserver hits vif plugging timeout with neutron CI job

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  We're now running neutron by default in CI jobs for ocata if they
  aren't explicitly specified otherwise to run nova-network.

  That might be unrelated to this, but I saw the citrix xenserver
  neutron job fail today with a vif plugging timeout:

  http://dd6b71949550285df7dc-
  dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/47/395747/8/check
  /dsvm-tempest-neutron-network/e3832fe/logs/screen-n-cpu.txt.gz

  2016-11-11 00:55:41.191 3175 WARNING nova.virt.xenapi.vmops [req-
  5f5774e3-92ee-4e49-947d-ce1c879bc1ab tempest-
  ServersTestManualDisk-1338550150 tempest-
  ServersTestManualDisk-1338550150] [instance: 79c00d5c-b285-44e5-b8db-
  9fc9a8e31478] Timeout waiting for vif plugging callback

  nova.conf seems to be configured correctly:

  http://dd6b71949550285df7dc-
  dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/47/395747/8/check
  /dsvm-tempest-neutron-network/e3832fe/logs/etc/nova/nova.conf.txt.gz

  vif_plugging_timeout = 300
  vif_plugging_is_fatal = True

  So I'm wondering if there is a problem in the vif plugging callback
  logic in the xenapi driver code.

  I see this earlier before the timeout:

  2016-11-11 00:50:29.526 3175 DEBUG nova.compute.manager [req-5f5774e3
  -92ee-4e49-947d-ce1c879bc1ab tempest-ServersTestManualDisk-1338550150
  tempest-ServersTestManualDisk-1338550150] [instance:
  79c00d5c-b285-44e5-b8db-9fc9a8e31478] Preparing to wait for external
  event network-vif-plugged-b6c3af4c-98cc-4077-8d4d-7009835c0c5c
  prepare_for_instance_event
  /opt/stack/new/nova/nova/compute/manager.py:324

  2016-11-11 00:50:29.527 3175 DEBUG nova.compute.manager [req-5f5774e3
  -92ee-4e49-947d-ce1c879bc1ab tempest-ServersTestManualDisk-1338550150
  tempest-ServersTestManualDisk-1338550150] [instance:
  79c00d5c-b285-44e5-b8db-9fc9a8e31478] Preparing to wait for external
  event network-vif-plugged-f5d2ac5a-36bf-4562-b2d0-18c40f640a3c
  prepare_for_instance_event
  /opt/stack/new/nova/nova/compute/manager.py:324

  2016-11-11 00:50:29.528 3175 DEBUG nova.virt.xenapi.vmops [req-
  5f5774e3-92ee-4e49-947d-ce1c879bc1ab tempest-
  ServersTestManualDisk-1338550150 tempest-
  ServersTestManualDisk-1338550150] wait for instance event:[('network-
  vif-plugged', u'b6c3af4c-98cc-4077-8d4d-7009835c0c5c'), ('network-vif-
  plugged', u'f5d2ac5a-36bf-4562-b2d0-18c40f640a3c')] _spawn
  /opt/stack/new/nova/nova/virt/xenapi/vmops.py:599

  Then it starts doing the vif plugging. The odd thing is it logs twice
  that it's preparing to wait for external events, but only logs
  'waiting for instance event' once. So is it waiting for another even
  that doesn't happen?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1640993/+subscriptions