yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1933813] [NEW] OSError: Premature eof waiting for privileged process error in Ussuri release of Neutron

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Vadim Ponomarev <1933813@xxxxxxxxxxxxxxxxxx>
Date: Mon, 28 Jun 2021 09:48:35 -0000
Reply-to: Bug 1933813 <1933813@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

After updating Neutron from Queens to Ussuri we started getting the
errors "OSError: Premature eof waiting for privileged process" for all
of neutron agents and for different operations (create network,
enable/disable dhcp etc.). All errors raises inside librari oslo.privsep
which was updated from version 1.27.0 to 2.1.2 [1]. This problem is
floating and very difficult to debug, it is impossible to understand
what exactly is happening. The problem is consistently reproduced by
tempest tests in our infrastructure.

Environment:
Neutron Ussuri, ML2 plugin + linux-bridge + custom agent HPB.

The examples of tracebacks:

OSError: Premature eof waiting for privileged process
  File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
    sync = self.process_network_devices(device_info)
  File "osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
  File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 214, in process_network_devices
    resync_a = self.treat_devices_added_updated(devices_added_updated)
  File "osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
  File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 231, in treat_devices_added_updated
    self._process_device_if_exists(device_details)
  File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 258, in _process_device_if_exists
    device, device_details['device_owner'])
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 594, in plug_interface
    network_segment.mtu)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 530, in add_tap_interface
    return False
  File "oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 522, in add_tap_interface
    tap_device_name, device_owner, mtu)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 555, in _add_tap_interface
    mtu):
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 510, in ensure_physical_in_bridge
    segmentation_id)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 256, in ensure_vlan_bridge
    if self.ensure_bridge(bridge_name, interface):
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 458, in ensure_bridge
    self.update_interface_ip_details(bridge_name, interface)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 417, in update_interface_ip_details
    ips, gateway = self.get_interface_details(source, ip_version)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 274, in get_interface_details
    ip_version=ip_version)
  File "neutron/agent/linux/ip_lib.py", line 560, in list
    **kwargs)
  File "neutron/agent/linux/ip_lib.py", line 1367, in get_devices_with_ip
    devices = privileged.get_link_devices(namespace, **link_args)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error


OSError: Premature eof waiting for privileged process
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
    self._plumber.delete_trunk_subports(trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
    return self.delete_subports_by_port_id(trunk.port_id)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
    if not ip_lib.device_exists(device, namespace=self.namespace):
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 73, in handle_subports
    self.wire_trunk(context, trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
    if not self._plumber.trunk_on_host(trunk):
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
    return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 73, in handle_subports
    self.wire_trunk(context, trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
    if not self._plumber.trunk_on_host(trunk):
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
    return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
    self.wire_trunk(context, trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
    if not self._plumber.trunk_on_host(trunk):
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
    return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
    self._plumber.delete_trunk_subports(trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
    return self.delete_subports_by_port_id(trunk.port_id)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
    if not ip_lib.device_exists(device, namespace=self.namespace):
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
    self._plumber.delete_trunk_subports(trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
    return self.delete_subports_by_port_id(trunk.port_id)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
    if not ip_lib.device_exists(device, namespace=self.namespace):
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
    self.wire_trunk(context, trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
    if not self._plumber.trunk_on_host(trunk):
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
    return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
    self.wire_trunk(context, trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
    if not self._plumber.trunk_on_host(trunk):
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
    return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
    self._plumber.delete_trunk_subports(trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
    return self.delete_subports_by_port_id(trunk.port_id)
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
    if not ip_lib.device_exists(device, namespace=self.namespace):
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 73, in handle_subports
    self.wire_trunk(context, trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
    if not self._plumber.trunk_on_host(trunk):
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
    return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
    cons_registry.push(context, resource_type, resource_objs, event_type)
  File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
    callback(context, resource_type, resource_list, event_type)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
    self.wire_trunk(context, trunk)
  File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
    if not self._plumber.trunk_on_host(trunk):
  File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
    return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
  File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
    return IPDevice(device_name, namespace=namespace).exists()
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
    self.agent.mgr.delete_bridge(bridge_name)
  File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
    if bridge_device.exists():
  File "neutron/agent/linux/ip_lib.py", line 328, in exists
    return privileged.interface_exists(self.name, self.namespace)
  File "oslo_privsep/priv_context.py", line 247, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "oslo_privsep/daemon.py", line 214, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "oslo_privsep/comm.py", line 172, in send_recv
    reply = future.result()
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
    result = func(ctxt, **new_args)
  File "oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)
  File "oslo_privsep/comm.py", line 111, in result
    raise self.error

[1] https://github.com/openstack/oslo.privsep/compare/1.27.0...2.1.2

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1933813

Title:
  OSError: Premature eof waiting for privileged process error in Ussuri
  release of Neutron

Status in neutron:
  New

Bug description:
  After updating Neutron from Queens to Ussuri we started getting the
  errors "OSError: Premature eof waiting for privileged process" for all
  of neutron agents and for different operations (create network,
  enable/disable dhcp etc.). All errors raises inside librari
  oslo.privsep which was updated from version 1.27.0 to 2.1.2 [1]. This
  problem is floating and very difficult to debug, it is impossible to
  understand what exactly is happening. The problem is consistently
  reproduced by tempest tests in our infrastructure.

  Environment:
  Neutron Ussuri, ML2 plugin + linux-bridge + custom agent HPB.

  The examples of tracebacks:

  OSError: Premature eof waiting for privileged process
    File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
      sync = self.process_network_devices(device_info)
    File "osprofiler/profiler.py", line 160, in wrapper
      result = f(*args, **kwargs)
    File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 214, in process_network_devices
      resync_a = self.treat_devices_added_updated(devices_added_updated)
    File "osprofiler/profiler.py", line 160, in wrapper
      result = f(*args, **kwargs)
    File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 231, in treat_devices_added_updated
      self._process_device_if_exists(device_details)
    File "neutron/plugins/ml2/drivers/agent/_common_agent.py", line 258, in _process_device_if_exists
      device, device_details['device_owner'])
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 594, in plug_interface
      network_segment.mtu)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 530, in add_tap_interface
      return False
    File "oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "site-packages/six.py", line 702, in reraise
      raise value.with_traceback(tb)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 522, in add_tap_interface
      tap_device_name, device_owner, mtu)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 555, in _add_tap_interface
      mtu):
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 510, in ensure_physical_in_bridge
      segmentation_id)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 256, in ensure_vlan_bridge
      if self.ensure_bridge(bridge_name, interface):
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 458, in ensure_bridge
      self.update_interface_ip_details(bridge_name, interface)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 417, in update_interface_ip_details
      ips, gateway = self.get_interface_details(source, ip_version)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 274, in get_interface_details
      ip_version=ip_version)
    File "neutron/agent/linux/ip_lib.py", line 560, in list
      **kwargs)
    File "neutron/agent/linux/ip_lib.py", line 1367, in get_devices_with_ip
      devices = privileged.get_link_devices(namespace, **link_args)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error

  
  OSError: Premature eof waiting for privileged process
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
      self._plumber.delete_trunk_subports(trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
      return self.delete_subports_by_port_id(trunk.port_id)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
      if not ip_lib.device_exists(device, namespace=self.namespace):
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 73, in handle_subports
      self.wire_trunk(context, trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
      if not self._plumber.trunk_on_host(trunk):
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
      return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 73, in handle_subports
      self.wire_trunk(context, trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
      if not self._plumber.trunk_on_host(trunk):
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
      return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
      self.wire_trunk(context, trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
      if not self._plumber.trunk_on_host(trunk):
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
      return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
      self._plumber.delete_trunk_subports(trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
      return self.delete_subports_by_port_id(trunk.port_id)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
      if not ip_lib.device_exists(device, namespace=self.namespace):
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
      self._plumber.delete_trunk_subports(trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
      return self.delete_subports_by_port_id(trunk.port_id)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
      if not ip_lib.device_exists(device, namespace=self.namespace):
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
      self.wire_trunk(context, trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
      if not self._plumber.trunk_on_host(trunk):
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
      return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
      self.wire_trunk(context, trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
      if not self._plumber.trunk_on_host(trunk):
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
      return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 57, in handle_trunks
      self._plumber.delete_trunk_subports(trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 64, in delete_trunk_subports
      return self.delete_subports_by_port_id(trunk.port_id)
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 68, in delete_subports_by_port_id
      if not ip_lib.device_exists(device, namespace=self.namespace):
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 73, in handle_subports
      self.wire_trunk(context, trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
      if not self._plumber.trunk_on_host(trunk):
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
      return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "neutron/api/rpc/handlers/resources_rpc.py", line 289, in push
      cons_registry.push(context, resource_type, resource_objs, event_type)
    File "neutron/api/rpc/callbacks/consumer/registry.py", line 38, in push
      callback(context, resource_type, resource_list, event_type)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 54, in handle_trunks
      self.wire_trunk(context, trunk)
    File "neutron/services/trunk/drivers/linuxbridge/agent/driver.py", line 101, in wire_trunk
      if not self._plumber.trunk_on_host(trunk):
    File "neutron/services/trunk/drivers/linuxbridge/agent/trunk_plumber.py", line 38, in trunk_on_host
      return ip_lib.device_exists(trunk_dev, namespace=self.namespace)
    File "neutron/agent/linux/ip_lib.py", line 748, in device_exists
      return IPDevice(device_name, namespace=namespace).exists()
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 912, in network_delete
      self.agent.mgr.delete_bridge(bridge_name)
    File "neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 598, in delete_bridge
      if bridge_device.exists():
    File "neutron/agent/linux/ip_lib.py", line 328, in exists
      return privileged.interface_exists(self.name, self.namespace)
    File "oslo_privsep/priv_context.py", line 247, in _wrap
      return self.channel.remote_call(name, args, kwargs)
    File "oslo_privsep/daemon.py", line 214, in remote_call
      result = self.send_recv((Message.CALL.value, name, args, kwargs))
    File "oslo_privsep/comm.py", line 172, in send_recv
      reply = future.result()
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
      result = func(ctxt, **new_args)
    File "oslo_messaging/rpc/server.py", line 241, in inner
      return func(*args, **kwargs)
    File "oslo_privsep/comm.py", line 111, in result
      raise self.error

  [1] https://github.com/openstack/oslo.privsep/compare/1.27.0...2.1.2

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1933813/+subscriptions
Follow ups

[Bug 1933813] Re: OSError: Premature eof waiting for privileged process error in Ussuri release of Neutron
From: Vadim Ponomarev, 2021-07-06