← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1868726] [NEW] Unhandled KeyError during subnet creation

 

Public bug reported:

I'm attempting to use tripleo's network isolation technique [0] on
stable/train, but my undercloud neutron service is failing during subnet
creation for the overcloud with a generic python KeyError:

2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager [req-be6ef908-8d3c-4e20-84b5-80ef1d8ab94e 8a8ad2f8e3f7478b8ce5e84f620cb4db 255eb2ea84ed4b29a55aceffcecf2a5d - default default] Error during notification for neutron.services.segments.plugin.NovaSegmentNotifier._notify_subnet_deleted--9223372036852953322 subnet, after_delete: KeyError: 'gateway_ip'
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/services/segments/plugin.py", line 321, in _notify_subnet_deleted
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     total, reserved = self._calculate_inventory_total_and_reserved(subnet)
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/services/segments/plugin.py", line 271, in _calculate_inventory_total_and_reserved
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     if subnet['gateway_ip']:
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager KeyError: 'gateway_ip'
2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager

While I'm not entirely convinced I didn't mis-configure something, the
error bubbled up through neutron's server.log isn't descriptive in what
I did wrong and doesn't seem to be validated by neutron's API layer.

Here are the parameters I'm using to deploy my overcloud (for those
interested in seeing how I'm doing that):

    (undercloud) [stack@undercloud ~]$ cat network-data.yaml
    - name: External
      vip: true
      name_lower: external
      vlan: 100
      ip_subnet: '10.0.0.0/24'
      allocation_pools: [{'start': '10.0.0.10', 'end': '10.0.0.99'}]
      gateway_ip: 10.0.0.5
      mtu: 1450
    - name: InternalApi
      name_lower: internal_api
      vip: true
      vlan: 10
      ip_subnet: '172.17.0.0/24'
      allocation_pools: [{'start': '172.17.0.10', 'end': '172.17.0.250'}]
      gateway_ip: '172.17.0.254'
      mtu: 1450
    - name: Storage
      vip: true
      vlan: 20
      name_lower: storage
      ip_subnet: '172.18.0.0/24'
      allocation_pools: [{'start': '172.18.0.10', 'end': '172.18.0.250'}]
      gateway_ip: '172.18.0.254'
      mtu: 1450
    - name: StorageMgmt
      name_lower: storage_mgmt
      vip: true
      vlan: 30
      ip_subnet: '172.19.0.0/24'
      allocation_pools: [{'start': '172.19.0.10', 'end': '172.19.0.250'}]
      mtu: 1450
    - name: Tenant
      vip: false  # Tenant network does not use VIPs
      name_lower: tenant
      vlan: 40
      ip_subnet: '172.16.0.0/24'
      allocation_pools: [{'start': '172.16.0.10', 'end': '172.16.0.250'}]
      gateway_ip: '172.16.0.254'
      mtu: 1450

    (undercloud) [stack@undercloud ~]$ cat parameters.yaml
    # Ansible managed

    resource_registry:
      OS::TripleO::Controller::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
      OS::TripleO::Compute::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml

    parameter_defaults:
      RootStackName: overcloud
      ControllerCount: 1
      ComputeCount: 1
      NeutronBridgeMappings: datacentre:br-ex,tenant:br-isolated
      NeutronExternalNetworkBridge: ''
      NeutronNetworkType: geneve
      NeutronNetworkVLANRanges: tenant:1000:2000
      NeutronTunnelTypes: geneve

[0] https://docs.openstack.org/project-deploy-guide/tripleo-
docs/latest/features/network_isolation.html

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

- I'm attempting to use tripleo's network isolation technique [0], but my
- undercloud neutron service is failing during subnet creation for the
- overcloud with a generic python KeyError:
+ I'm attempting to use tripleo's network isolation technique [0] on
+ stable/train, but my undercloud neutron service is failing during subnet
+ creation for the overcloud with a generic python KeyError:
  
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager [req-be6ef908-8d3c-4e20-84b5-80ef1d8ab94e 8a8ad2f8e3f7478b8ce5e84f620cb4db 255eb2ea84ed4b29a55aceffcecf2a5d - default default] Error during notification for neutron.services.segments.plugin.NovaSegmentNotifier._notify_subnet_deleted--9223372036852953322 subnet, after_delete: KeyError: 'gateway_ip'
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/services/segments/plugin.py", line 321, in _notify_subnet_deleted
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     total, reserved = self._calculate_inventory_total_and_reserved(subnet)
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/services/segments/plugin.py", line 271, in _calculate_inventory_total_and_reserved
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     if subnet['gateway_ip']:
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager KeyError: 'gateway_ip'
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager
  
- 
- While I'm not entirely convinced I didn't mis-configure something, the error bubbled up through neutron's server.log isn't descriptive in what I did wrong and doesn't seem to be validated by neutron's API layer.
+ While I'm not entirely convinced I didn't mis-configure something, the
+ error bubbled up through neutron's server.log isn't descriptive in what
+ I did wrong and doesn't seem to be validated by neutron's API layer.
  
  Here are the parameters I'm using to deploy my overcloud (for those
  interested in seeing how I'm doing that):
  
-     (undercloud) [stack@undercloud ~]$ cat network-data.yaml
-     - name: External
-       vip: true
-       name_lower: external
-       vlan: 100
-       ip_subnet: '10.0.0.0/24'
-       allocation_pools: [{'start': '10.0.0.10', 'end': '10.0.0.99'}]
-       gateway_ip: 10.0.0.5
-       mtu: 1450
-     - name: InternalApi
-       name_lower: internal_api
-       vip: true
-       vlan: 10
-       ip_subnet: '172.17.0.0/24'
-       allocation_pools: [{'start': '172.17.0.10', 'end': '172.17.0.250'}]
-       gateway_ip: '172.17.0.254'
-       mtu: 1450
-     - name: Storage
-       vip: true
-       vlan: 20
-       name_lower: storage
-       ip_subnet: '172.18.0.0/24'
-       allocation_pools: [{'start': '172.18.0.10', 'end': '172.18.0.250'}]
-       gateway_ip: '172.18.0.254'
-       mtu: 1450
-     - name: StorageMgmt
-       name_lower: storage_mgmt
-       vip: true
-       vlan: 30
-       ip_subnet: '172.19.0.0/24'
-       allocation_pools: [{'start': '172.19.0.10', 'end': '172.19.0.250'}]
-       mtu: 1450
-     - name: Tenant
-       vip: false  # Tenant network does not use VIPs
-       name_lower: tenant
-       vlan: 40
-       ip_subnet: '172.16.0.0/24'
-       allocation_pools: [{'start': '172.16.0.10', 'end': '172.16.0.250'}]
-       gateway_ip: '172.16.0.254'
-       mtu: 1450
-      
-     (undercloud) [stack@undercloud ~]$ cat parameters.yaml
-     # Ansible managed
-      
-     resource_registry:
-       OS::TripleO::Controller::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
-       OS::TripleO::Compute::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
-      
-     parameter_defaults:
-       RootStackName: overcloud
-       ControllerCount: 1
-       ComputeCount: 1
-       NeutronBridgeMappings: datacentre:br-ex,tenant:br-isolated
-       NeutronExternalNetworkBridge: ''
-       NeutronNetworkType: geneve
-       NeutronNetworkVLANRanges: tenant:1000:2000
-       NeutronTunnelTypes: geneve
+     (undercloud) [stack@undercloud ~]$ cat network-data.yaml
+     - name: External
+       vip: true
+       name_lower: external
+       vlan: 100
+       ip_subnet: '10.0.0.0/24'
+       allocation_pools: [{'start': '10.0.0.10', 'end': '10.0.0.99'}]
+       gateway_ip: 10.0.0.5
+       mtu: 1450
+     - name: InternalApi
+       name_lower: internal_api
+       vip: true
+       vlan: 10
+       ip_subnet: '172.17.0.0/24'
+       allocation_pools: [{'start': '172.17.0.10', 'end': '172.17.0.250'}]
+       gateway_ip: '172.17.0.254'
+       mtu: 1450
+     - name: Storage
+       vip: true
+       vlan: 20
+       name_lower: storage
+       ip_subnet: '172.18.0.0/24'
+       allocation_pools: [{'start': '172.18.0.10', 'end': '172.18.0.250'}]
+       gateway_ip: '172.18.0.254'
+       mtu: 1450
+     - name: StorageMgmt
+       name_lower: storage_mgmt
+       vip: true
+       vlan: 30
+       ip_subnet: '172.19.0.0/24'
+       allocation_pools: [{'start': '172.19.0.10', 'end': '172.19.0.250'}]
+       mtu: 1450
+     - name: Tenant
+       vip: false  # Tenant network does not use VIPs
+       name_lower: tenant
+       vlan: 40
+       ip_subnet: '172.16.0.0/24'
+       allocation_pools: [{'start': '172.16.0.10', 'end': '172.16.0.250'}]
+       gateway_ip: '172.16.0.254'
+       mtu: 1450
  
+     (undercloud) [stack@undercloud ~]$ cat parameters.yaml
+     # Ansible managed
+ 
+     resource_registry:
+       OS::TripleO::Controller::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
+       OS::TripleO::Compute::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
+ 
+     parameter_defaults:
+       RootStackName: overcloud
+       ControllerCount: 1
+       ComputeCount: 1
+       NeutronBridgeMappings: datacentre:br-ex,tenant:br-isolated
+       NeutronExternalNetworkBridge: ''
+       NeutronNetworkType: geneve
+       NeutronNetworkVLANRanges: tenant:1000:2000
+       NeutronTunnelTypes: geneve
  
  [0] https://docs.openstack.org/project-deploy-guide/tripleo-
  docs/latest/features/network_isolation.html

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1868726

Title:
  Unhandled KeyError during subnet creation

Status in neutron:
  New

Bug description:
  I'm attempting to use tripleo's network isolation technique [0] on
  stable/train, but my undercloud neutron service is failing during
  subnet creation for the overcloud with a generic python KeyError:

  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager [req-be6ef908-8d3c-4e20-84b5-80ef1d8ab94e 8a8ad2f8e3f7478b8ce5e84f620cb4db 255eb2ea84ed4b29a55aceffcecf2a5d - default default] Error during notification for neutron.services.segments.plugin.NovaSegmentNotifier._notify_subnet_deleted--9223372036852953322 subnet, after_delete: KeyError: 'gateway_ip'
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/services/segments/plugin.py", line 321, in _notify_subnet_deleted
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     total, reserved = self._calculate_inventory_total_and_reserved(subnet)
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/services/segments/plugin.py", line 271, in _calculate_inventory_total_and_reserved
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager     if subnet['gateway_ip']:
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager KeyError: 'gateway_ip'
  2020-03-24 00:12:05.020 35 ERROR neutron_lib.callbacks.manager

  While I'm not entirely convinced I didn't mis-configure something, the
  error bubbled up through neutron's server.log isn't descriptive in
  what I did wrong and doesn't seem to be validated by neutron's API
  layer.

  Here are the parameters I'm using to deploy my overcloud (for those
  interested in seeing how I'm doing that):

      (undercloud) [stack@undercloud ~]$ cat network-data.yaml
      - name: External
        vip: true
        name_lower: external
        vlan: 100
        ip_subnet: '10.0.0.0/24'
        allocation_pools: [{'start': '10.0.0.10', 'end': '10.0.0.99'}]
        gateway_ip: 10.0.0.5
        mtu: 1450
      - name: InternalApi
        name_lower: internal_api
        vip: true
        vlan: 10
        ip_subnet: '172.17.0.0/24'
        allocation_pools: [{'start': '172.17.0.10', 'end': '172.17.0.250'}]
        gateway_ip: '172.17.0.254'
        mtu: 1450
      - name: Storage
        vip: true
        vlan: 20
        name_lower: storage
        ip_subnet: '172.18.0.0/24'
        allocation_pools: [{'start': '172.18.0.10', 'end': '172.18.0.250'}]
        gateway_ip: '172.18.0.254'
        mtu: 1450
      - name: StorageMgmt
        name_lower: storage_mgmt
        vip: true
        vlan: 30
        ip_subnet: '172.19.0.0/24'
        allocation_pools: [{'start': '172.19.0.10', 'end': '172.19.0.250'}]
        mtu: 1450
      - name: Tenant
        vip: false  # Tenant network does not use VIPs
        name_lower: tenant
        vlan: 40
        ip_subnet: '172.16.0.0/24'
        allocation_pools: [{'start': '172.16.0.10', 'end': '172.16.0.250'}]
        gateway_ip: '172.16.0.254'
        mtu: 1450

      (undercloud) [stack@undercloud ~]$ cat parameters.yaml
      # Ansible managed

      resource_registry:
        OS::TripleO::Controller::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
        OS::TripleO::Compute::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml

      parameter_defaults:
        RootStackName: overcloud
        ControllerCount: 1
        ComputeCount: 1
        NeutronBridgeMappings: datacentre:br-ex,tenant:br-isolated
        NeutronExternalNetworkBridge: ''
        NeutronNetworkType: geneve
        NeutronNetworkVLANRanges: tenant:1000:2000
        NeutronTunnelTypes: geneve

  [0] https://docs.openstack.org/project-deploy-guide/tripleo-
  docs/latest/features/network_isolation.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1868726/+subscriptions