← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #88731

[Bug 1970383] [NEW] Segment-aware scheduling fails for non-admin users

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This is a follow-up to https://bugs.launchpad.net/nova/+bug/1967314

Having deployed the Nova scheduler configuration for routed provider
networks as follows (Xena deployment @
7df9379d6661233174d49fb7be8eda0828a5e5ca), this was found to resolve
issues around scheduling of instances to appropriate hypervisors, but it
appears to have surfaced a side effect.

[scheduler]
query_placement_for_routed_network_aggregates = True

When the above configuration is enabled, creation of new instances for
admin users works correctly, but for non-admin users against the same
networks results in the following error:

285768 ERROR oslo_messaging.rpc.server [req-79ca3cb3-eb52-4755-bba1-4c840c8ae5fc c35a1473225f422c90a6f75b25188bf2 d96f0cd70c6a4adbbbcf993502b264dc - default default] Exception during message handling: K>
285768 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
285768 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
285768 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
285768 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
285768 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
285768 ERROR oslo_messaging.rpc.server     request_filter.process_reqspec(context, spec_obj)
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 387, in process_reqspec
285768 ERROR oslo_messaging.rpc.server     filter(ctxt, request_spec)
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 41, in wrapper
285768 ERROR oslo_messaging.rpc.server     ran = fn(ctxt, request_spec)
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 348, in routed_networks_filter
285768 ERROR oslo_messaging.rpc.server     aggregates = utils.get_aggregates_for_routed_network(
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/utils.py", line 1406, in get_aggregates_for_routed_network
285768 ERROR oslo_messaging.rpc.server     segment_ids = network_api.get_segment_ids_for_network(
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/network/neutron.py", line 3721, in get_segment_ids_for_network
285768 ERROR oslo_messaging.rpc.server     return [subnet['segment_id'] for subnet in subnets
285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/network/neutron.py", line 3722, in <listcomp>
285768 ERROR oslo_messaging.rpc.server     if subnet['segment_id'] is not None]
285768 ERROR oslo_messaging.rpc.server KeyError: 'segment_id'
285768 ERROR oslo_messaging.rpc.server

It appears that the subnet dictionaries are returned empty from the
Neutron client library in this case, causing the KeyError.

As far as I can see, a matching command line request for 'openstack
subnet show X' as the same requesting user correctly includes the
'segment_id', but I don't know how similar this code path and the
permissions handling is.

I'd be happy to test out other requests or obtain additional logs if
useful.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1970383

Title:
  Segment-aware scheduling fails for non-admin users

Status in OpenStack Compute (nova):
  New

Bug description:
  This is a follow-up to https://bugs.launchpad.net/nova/+bug/1967314

  Having deployed the Nova scheduler configuration for routed provider
  networks as follows (Xena deployment @
  7df9379d6661233174d49fb7be8eda0828a5e5ca), this was found to resolve
  issues around scheduling of instances to appropriate hypervisors, but
  it appears to have surfaced a side effect.

  [scheduler]
  query_placement_for_routed_network_aggregates = True

  When the above configuration is enabled, creation of new instances for
  admin users works correctly, but for non-admin users against the same
  networks results in the following error:

  285768 ERROR oslo_messaging.rpc.server [req-79ca3cb3-eb52-4755-bba1-4c840c8ae5fc c35a1473225f422c90a6f75b25188bf2 d96f0cd70c6a4adbbbcf993502b264dc - default default] Exception during message handling: K>
  285768 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  285768 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  285768 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  285768 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
  285768 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
  285768 ERROR oslo_messaging.rpc.server     request_filter.process_reqspec(context, spec_obj)
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 387, in process_reqspec
  285768 ERROR oslo_messaging.rpc.server     filter(ctxt, request_spec)
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 41, in wrapper
  285768 ERROR oslo_messaging.rpc.server     ran = fn(ctxt, request_spec)
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 348, in routed_networks_filter
  285768 ERROR oslo_messaging.rpc.server     aggregates = utils.get_aggregates_for_routed_network(
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/utils.py", line 1406, in get_aggregates_for_routed_network
  285768 ERROR oslo_messaging.rpc.server     segment_ids = network_api.get_segment_ids_for_network(
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/network/neutron.py", line 3721, in get_segment_ids_for_network
  285768 ERROR oslo_messaging.rpc.server     return [subnet['segment_id'] for subnet in subnets
  285768 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/network/neutron.py", line 3722, in <listcomp>
  285768 ERROR oslo_messaging.rpc.server     if subnet['segment_id'] is not None]
  285768 ERROR oslo_messaging.rpc.server KeyError: 'segment_id'
  285768 ERROR oslo_messaging.rpc.server

  It appears that the subnet dictionaries are returned empty from the
  Neutron client library in this case, causing the KeyError.

  As far as I can see, a matching command line request for 'openstack
  subnet show X' as the same requesting user correctly includes the
  'segment_id', but I don't know how similar this code path and the
  permissions handling is.

  I'd be happy to test out other requests or obtain additional logs if
  useful.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1970383/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next