← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #79499

[Bug 1838817] [NEW] neutron: refreshing vif model for a server gets the same network multiple times

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

As of this change in Rocky:

https://review.opendev.org/#/c/585339/

When refreshing the vif model for an instance, e.g. when we get a
network-changed event with a specific port ID:

https://logs.opendev.org/26/674326/2/experimental/nova-osprofiler-
redis/899a204/controller/logs/screen-n-cpu.txt.gz#_Aug_02_18_35_50_884613

Aug 02 18:35:50.884613 ubuntu-bionic-vexxhost-sjc1-0009723918 nova-
compute[20428]: DEBUG nova.network.neutronv2.api [req-1a4c2dbf-df86-4044
-a59f-f751a53c5ea6 req-b0e1e2f7-d126-4e55-909d-4803816ca80f service
nova] [instance: 5bbe0419-fbeb-4667-8c56-785fdc1d0a62] Refreshing
network info cache for port 252040d6-4469-46ec-88c3-85e599a43104
{{(pid=20428) _get_instance_nw_info
/opt/stack/nova/nova/network/neutronv2/api.py:1756}}

We get the network for the port multiple times, first here:

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L2966

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L392

And then we pass that list of 1 network dict to _build_vif_model here:

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L2850

and pass it to _nw_info_build_network here:

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L2883

Which then calls _get_physnet_tunneled_info which gets the network again
here:

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L1904

and/or here:

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L1928

Furthermore, when we're doing forced _heal_instance_info_cache (stein+)
we'll refresh the vif model for all ports that are currently attached to
the server:

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L3015

And rebuild the vif model per port here:

https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L3027

If there is more than one port on the same network attached to the
server, we'll be calling show_network for each port even though we're
getting the same data when those ports are on the same network.

I noticed this while checking some osprofiler results and noticed the
network-changed event on the port-targeted refresh took a relatively
long time:

https://logs.opendev.org/26/674326/2/experimental/nova-osprofiler-
redis/899a204/osprofiler-traces/trace-fc50ca23-a6c2-474a-
ac07-e61e706eb27d.html.gz

** Affects: nova
     Importance: Medium
     Assignee: Matt Riedemann (mriedem)
         Status: Triaged

** Affects: nova/rocky
     Importance: Medium
         Status: Confirmed

** Affects: nova/stein
     Importance: Medium
         Status: Confirmed


** Tags: neutron performance

** Changed in: nova
       Status: New => Triaged

** Changed in: nova
   Importance: Undecided => Medium

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Also affects: nova/stein
   Importance: Undecided
       Status: New

** Changed in: nova/rocky
       Status: New => Confirmed

** Changed in: nova/stein
   Importance: Undecided => Medium

** Changed in: nova/stein
       Status: New => Confirmed

** Changed in: nova/rocky
   Importance: Undecided => Medium

** Changed in: nova
     Assignee: (unassigned) => Matt Riedemann (mriedem)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1838817

Title:
  neutron: refreshing vif model for a server gets the same network
  multiple times

Status in OpenStack Compute (nova):
  Triaged
Status in OpenStack Compute (nova) rocky series:
  Confirmed
Status in OpenStack Compute (nova) stein series:
  Confirmed

Bug description:
  As of this change in Rocky:

  https://review.opendev.org/#/c/585339/

  When refreshing the vif model for an instance, e.g. when we get a
  network-changed event with a specific port ID:

  https://logs.opendev.org/26/674326/2/experimental/nova-osprofiler-
  redis/899a204/controller/logs/screen-n-cpu.txt.gz#_Aug_02_18_35_50_884613

  Aug 02 18:35:50.884613 ubuntu-bionic-vexxhost-sjc1-0009723918 nova-
  compute[20428]: DEBUG nova.network.neutronv2.api [req-1a4c2dbf-
  df86-4044-a59f-f751a53c5ea6 req-b0e1e2f7-d126-4e55-909d-4803816ca80f
  service nova] [instance: 5bbe0419-fbeb-4667-8c56-785fdc1d0a62]
  Refreshing network info cache for port 252040d6-4469-46ec-
  88c3-85e599a43104 {{(pid=20428) _get_instance_nw_info
  /opt/stack/nova/nova/network/neutronv2/api.py:1756}}

  We get the network for the port multiple times, first here:

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L2966

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L392

  And then we pass that list of 1 network dict to _build_vif_model here:

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L2850

  and pass it to _nw_info_build_network here:

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L2883

  Which then calls _get_physnet_tunneled_info which gets the network
  again here:

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L1904

  and/or here:

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L1928

  Furthermore, when we're doing forced _heal_instance_info_cache
  (stein+) we'll refresh the vif model for all ports that are currently
  attached to the server:

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L3015

  And rebuild the vif model per port here:

  https://github.com/openstack/nova/blob/600ecf3d9a5116d040cd18023ff270b91b06247d/nova/network/neutronv2/api.py#L3027

  If there is more than one port on the same network attached to the
  server, we'll be calling show_network for each port even though we're
  getting the same data when those ports are on the same network.

  I noticed this while checking some osprofiler results and noticed the
  network-changed event on the port-targeted refresh took a relatively
  long time:

  https://logs.opendev.org/26/674326/2/experimental/nova-osprofiler-
  redis/899a204/osprofiler-traces/trace-fc50ca23-a6c2-474a-
  ac07-e61e706eb27d.html.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1838817/+subscriptions
  Thread Previous • Date Previous • • Thread Next