yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1742963] Re: Cannot boot VM with Contrail SDN controller

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1742963@xxxxxxxxxxxxxxxxxx>
Date: Wed, 21 Feb 2018 22:38:58 -0000
Reply-to: Bug 1742963 <1742963@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/533212
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1f5fe3190bf2e0987945a6ef9ec430673c9fa736
Submitter: Zuul
Branch:    master

commit 1f5fe3190bf2e0987945a6ef9ec430673c9fa736
Author: Édouard Thuleau <ethuleau@xxxxxxxxxxx>
Date:   Fri Jan 12 16:20:32 2018 +0100

    Update plugs Contrail methods to work with privsep
    
    As privsep uses the msgpack to send method arguments to the privsep
    daemon, we could not use anymore custom data type like
    nova.objects.instance.Instance.
    
    Change-Id: I09f04d5b2f1cb39339ad7c4569186db5d361797a
    Closes-Bug: #1742963


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1742963

Title:
  Cannot boot VM with Contrail SDN controller

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  In Progress

Bug description:
  On the master branch, nova-compute fails to create vif on the Contrail vrouter compute agent and the instance fails to spawn:
  ...
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     six.reraise(self.type_, self.value, self.tb)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/opt/stack/openstack/nova/nova/virt/libvirt/driver.py", line 5238, 
  in _create_domain_and_network
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     self.plug_vifs(instance, network_info)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/opt/stack/openstack/nova/nova/virt/libvirt/driver.py", line 755, in plug_vifs
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     self.vif_driver.plug(instance, vif)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/opt/stack/openstack/nova/nova/virt/libvirt/vif.py", line 769, in p
  lug
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     func(instance, vif)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/opt/stack/openstack/nova/nova/virt/libvirt/vif.py", line 727, in plug_vrouter
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     instance, vif, ip_addr, ip6_addr, ptype)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 207, in _wrap
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     return self.channel.remote_call(name, args, kwargs)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 192, in remote_call
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     result = self.send_recv((Message.CALL.value, name, args, kwargs))
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line 163, in send_recv
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     self.writer.send((myid, msg))
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line 
  54, in send
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     buf = msgpack.packb(msg, use_bin_type=True)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "/usr/local/lib/python2.7/dist-packages/msgpack/__init__.py", line 47, in packb
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]     return Packer(**kwargs).pack(o)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msg
  pack/_packer.cpp:3661)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msg
  pack/_packer.cpp:3503)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "msgpack/_packer.pyx", line 221, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3230)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "msgpack/_packer.pyx", line 221, in msgpack._packer.Packer._pack (ms
  gpack/_packer.cpp:3230)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "msgpack/_packer.pyx", line 221, in msgpack._packer.Packer._pack (ms
  gpack/_packer.cpp:3230)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]   File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (ms
  gpack/_packer.cpp:3382)
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] TypeError: can't serialize Instance(access_ip_v4=None,access_ip_v6=None,arc
  hitecture=None,auto_disk_config=False,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2018-01-12T15:05:54Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=None,disable_terminate=False,display_description='vm1',display_name='vm1',ec2_ids=EC2Ids,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),host='ethuleau-contrail-mast
  er',hostname='vm1',id=27,image_ref='a9c4ac60-dddd-4445-8199-02e0e88b1431',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=KeyPairList,launch_index=0,launched_at=None,launched_on='ethuleau-contrail-master',locked=False,locked_by=None,memory_mb=256,metadata={},migration_context=<?>,new_flavor=None,node='ethuleau-contrail-master',numa_topology=None,old_flavor=None,os_type=None,pci_devices=PciDeviceList,pci_requests=InstancePCIRequests,power_state=0,progress=0,project_id='80f750bf6b804de79bbd50ae8d31df32',ramdisk_id='',reservation_id='r-p3ur515e',root_device_name='/dev/vda',root_gb=0
  ,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='anotherrole,Member',image_base_image_ref='a9c4ac60-dddd-4445-8199-02e0e88b1431',image_container_format='bare',
  image_disk_format='qcow2',image_min_disk='0',image_min_ram='0',network_allocated='True',owner_project_name='demo',owner_user_name='demo'},tags=TagList,task_state='spawning',terminated_at=None,updated_at=2018-01-12T15:05:56Z,user_data=None,user_id='837fc18d88384a62b84635f358a75eee',uuid=8748627c-e31f-4c90-83e2-16abdf9c1e2c,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='building')
  Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1742963/+subscriptions
References

[Bug 1742963] [NEW] Cannot boot VM with Contrail SDN controller
From: Édouard Thuleau, 2018-01-12