yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1793606] [NEW] Setting vendordata_providers to DynamicJSON causes config drive to stop working

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bob Taylor <robert.taylor@xxxxxxxxxxx>
Date: Thu, 20 Sep 2018 21:39:11 -0000
Reply-to: Bug 1793606 <1793606@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

When the vendordata_providers option is set to DynamicJSON, the config
drive ceases to function and instances fail to spawn with a
'InvalidMetadataPath: /openstack/2013-10-17/vendor_data.json' error.
Specifically, this issue occurred when spawning amphora for Octavia.
This issue can be worked around by adding StaticJSON (the default
value)into the vendordata_providers option.

Steps to reproduce
==================
After an initial deployment and validation of Octavia on Queens, I set the nova vendordata_providers option to DynamicJSON.  After setting this value, Octavia was no longer able to spawn new amphora instances. This was isolated to the config drive not being available (see log below). 

I was able to work around the issue by adding the default StaticJSON
value back into the nova.conf along with DynamicJSON (i.e.,
vendordata_providers: "DynamicJSON,StaticJSON").

Expected result
===============
The vendordata_providers flag should be able to be set to DynamicJSON only without requiring StaticJSON to be specified as well.


Environment
===========
OpenStack Queens (nova-r17.1.0)

Logs & Configs
==============

api section from nova.conf (not functional)

[api]
auth_strategy = keystone
enable_instance_password = True
use_forwarded_for = False
vendordata_dynamic_targets = xxxx@http://my-vendordata-service
vendordata_jsonfile_path = /etc/nova/vendor_data.json
vendordata_providers = DynamicJSON

api section from nova.conf (functional)

[api]
auth_strategy = keystone
enable_instance_password = True
use_forwarded_for = False
vendordata_dynamic_targets = xxxx@http://my-vendordata-service
vendordata_jsonfile_path = /etc/nova/vendor_data.json
vendordata_providers = DynamicJSON,StaticJSON

nova.log errors from the relevant hypervisor:

2018-09-10 14:14:38.106 6889 INFO nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Using config drive
2018-09-10 14:14:38.328 6889 ERROR nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Failed to start libvirt guest: InvalidMetadataPath: /openstack/2013-10-17/vendor_data.json
2018-09-10 14:14:38.330 6889 INFO os_vif [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:f7:9e:2a,bridge_name='brq5f363b24-40',has_traffic_filtering=True,id=d9b24239-a15f-4758-bc22-147523b10a73,network=Network(5f363b24-40f9-486b-a39e-7a935f19924a),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapd9b24239-a1')
2018-09-10 14:14:38.346 6889 INFO nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Deleting instance files /var/lib/nova/instances/54118c10-309d-4ccb-8ca4-d5ee6d851be5_del
2018-09-10 14:14:38.347 6889 INFO nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Deletion of /var/lib/nova/instances/54118c10-309d-4ccb-8ca4-d5ee6d851be5_del complete
2018-09-10 14:14:38.602 6889 ERROR nova.compute.manager [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Instance failed to spawn: InvalidMetadataPath: /openstack/2013-10-17/vendor_data.json

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1793606

Title:
  Setting vendordata_providers to DynamicJSON causes config drive to
  stop working

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  When the vendordata_providers option is set to DynamicJSON, the config
  drive ceases to function and instances fail to spawn with a
  'InvalidMetadataPath: /openstack/2013-10-17/vendor_data.json' error.
  Specifically, this issue occurred when spawning amphora for Octavia.
  This issue can be worked around by adding StaticJSON (the default
  value)into the vendordata_providers option.

  Steps to reproduce
  ==================
  After an initial deployment and validation of Octavia on Queens, I set the nova vendordata_providers option to DynamicJSON.  After setting this value, Octavia was no longer able to spawn new amphora instances. This was isolated to the config drive not being available (see log below). 

  I was able to work around the issue by adding the default StaticJSON
  value back into the nova.conf along with DynamicJSON (i.e.,
  vendordata_providers: "DynamicJSON,StaticJSON").

  Expected result
  ===============
  The vendordata_providers flag should be able to be set to DynamicJSON only without requiring StaticJSON to be specified as well.

  
  Environment
  ===========
  OpenStack Queens (nova-r17.1.0)

  Logs & Configs
  ==============

  api section from nova.conf (not functional)

  [api]
  auth_strategy = keystone
  enable_instance_password = True
  use_forwarded_for = False
  vendordata_dynamic_targets = xxxx@http://my-vendordata-service
  vendordata_jsonfile_path = /etc/nova/vendor_data.json
  vendordata_providers = DynamicJSON

  api section from nova.conf (functional)

  [api]
  auth_strategy = keystone
  enable_instance_password = True
  use_forwarded_for = False
  vendordata_dynamic_targets = xxxx@http://my-vendordata-service
  vendordata_jsonfile_path = /etc/nova/vendor_data.json
  vendordata_providers = DynamicJSON,StaticJSON

  nova.log errors from the relevant hypervisor:

  2018-09-10 14:14:38.106 6889 INFO nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Using config drive
  2018-09-10 14:14:38.328 6889 ERROR nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Failed to start libvirt guest: InvalidMetadataPath: /openstack/2013-10-17/vendor_data.json
  2018-09-10 14:14:38.330 6889 INFO os_vif [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:f7:9e:2a,bridge_name='brq5f363b24-40',has_traffic_filtering=True,id=d9b24239-a15f-4758-bc22-147523b10a73,network=Network(5f363b24-40f9-486b-a39e-7a935f19924a),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapd9b24239-a1')
  2018-09-10 14:14:38.346 6889 INFO nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Deleting instance files /var/lib/nova/instances/54118c10-309d-4ccb-8ca4-d5ee6d851be5_del
  2018-09-10 14:14:38.347 6889 INFO nova.virt.libvirt.driver [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Deletion of /var/lib/nova/instances/54118c10-309d-4ccb-8ca4-d5ee6d851be5_del complete
  2018-09-10 14:14:38.602 6889 ERROR nova.compute.manager [req-a73839b5-391f-4135-94f8-7c0b07980416 bcfaa1d2ec364aa6bf335df27a45323f b216d01422ea408994bd81a83835bfe1 - default default] [instance: 54118c10-309d-4ccb-8ca4-d5ee6d851be5] Instance failed to spawn: InvalidMetadataPath: /openstack/2013-10-17/vendor_data.json

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1793606/+subscriptions
Follow ups

[Bug 1793606] Re: Setting vendordata_providers to DynamicJSON causes config drive to stop working
From: Balazs Gibizer, 2020-09-16