yahoo-eng-team team mailing list archive

[Carlos Goncalves <cgoncalves@xxxxxxxxxx>]

Public bug reported:

Neutron API returns HTTP 201 (Created) for security group rule create
requests, although it takes longer to apply the configuration to the
port. This means for a period of time the firewall on the port is
outdated, eventually posing a security risk or applications to
fail/misbehave. Even though not tested, it might even be that the
q-agent could completely miss the SG rule add event from the Neutron
server and never apply it.

The log below is of a security group rule create request from Octavia to
Neutron. Neutron returns HTTP 201 but the q-agent has not yet applied
the configuration. The Octavia tempest test expects the load balancer
VIP to conform to the security group rules but fails as the q-agent
still have not applied the new security group rule to the port yet.

Mar 03 17:33:24.786466 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.network_tasks.UpdateVIP' (10c8bae1-19b1-4757-9530-12ac29384565) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=8984) _task_receiver /usr/local/lib/python3.6/dist-packages/taskflow/listeners/logging.py:194}}
Mar 03 17:33:24.787574 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.controller.worker.v1.tasks.network_tasks [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Updating VIP of load_balancer 61145d72-04e1-49bd-bcb0-5c215ed217ea. {{(pid=8984) execute /opt/stack/octavia/octavia/controller/worker/v1/tasks/network_tasks.py:472}}
Mar 03 17:33:24.805139 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension security-group found enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:66}}
Mar 03 17:33:24.819184 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension dns-integration is not enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:70}}
Mar 03 17:33:24.832337 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension qos found enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:66}}
Mar 03 17:33:24.847909 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension allowed-address-pairs found enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:66}}
Mar 03 17:33:25.221590 ubuntu-bionic-airship-kna1-0014969351 neutron-server[7030]: INFO neutron.wsgi [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] 10.0.1.16,10.0.1.16 "POST /v2.0/security-group-rules HTTP/1.1" status: 201  len: 725 time: 0.1413145
Mar 03 17:33:25.224900 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.network_tasks.UpdateVIP' (10c8bae1-19b1-4757-9530-12ac29384565) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=8984) _task_receiver /usr/local/lib/python3.6/dist-packages/taskflow/listeners/logging.py:183}}
Mar 03 17:33:25.224298 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: DEBUG neutron.agent.resource_cache [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Received new resource SecurityGroupRule: SecurityGroupRule(created_at=2020-03-03T17:33:25Z,description='',direction='ingress',ethertype='IPv4',id=73e2e34d-a813-4846-8f85-2b8daae5d29c,port_range_max=8080,port_range_min=8080,project_id='e821f6bae64f4fa0bca1c230fbf4b364',protocol='tcp',remote_group_id=<?>,remote_ip_prefix=192.0.1.0/32,revision_number=0,security_group_id=14216a23-b9c5-4cb3-b42d-c76b22c643ec,updated_at=2020-03-03T17:33:25Z) {{(pid=7528) record_resource_update /opt/stack/neutron/neutron/agent/resource_cache.py:192}}
Mar 03 17:33:25.224767 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: DEBUG neutron_lib.callbacks.manager [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_rule_update--9223372036854365827'] for SecurityGroupRule, after_update {{(pid=7528) _notify_loop /usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py:193}}
Mar 03 17:33:25.225185 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: INFO neutron.agent.securitygroups_rpc [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Security group rule updated ['14216a23-b9c5-4cb3-b42d-c76b22c643ec']
Mar 03 17:33:25.225605 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: DEBUG neutron.agent.securitygroups_rpc [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Adding ['d0849264-740f-4a37-b77d-182d6f121067'] devices to the list of devices for which firewall needs to be refreshed {{(pid=7528) _security_group_updated /opt/stack/neutron/neutron/agent/securitygroups_rpc.py:206}}

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1866353

Title:
  Neutron API returning HTTP 201 for SG rule create  when not fully
  created yet

Status in neutron:
  New

Bug description:
  Neutron API returns HTTP 201 (Created) for security group rule create
  requests, although it takes longer to apply the configuration to the
  port. This means for a period of time the firewall on the port is
  outdated, eventually posing a security risk or applications to
  fail/misbehave. Even though not tested, it might even be that the
  q-agent could completely miss the SG rule add event from the Neutron
  server and never apply it.

  The log below is of a security group rule create request from Octavia
  to Neutron. Neutron returns HTTP 201 but the q-agent has not yet
  applied the configuration. The Octavia tempest test expects the load
  balancer VIP to conform to the security group rules but fails as the
  q-agent still have not applied the new security group rule to the port
  yet.

  Mar 03 17:33:24.786466 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.network_tasks.UpdateVIP' (10c8bae1-19b1-4757-9530-12ac29384565) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=8984) _task_receiver /usr/local/lib/python3.6/dist-packages/taskflow/listeners/logging.py:194}}
  Mar 03 17:33:24.787574 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.controller.worker.v1.tasks.network_tasks [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Updating VIP of load_balancer 61145d72-04e1-49bd-bcb0-5c215ed217ea. {{(pid=8984) execute /opt/stack/octavia/octavia/controller/worker/v1/tasks/network_tasks.py:472}}
  Mar 03 17:33:24.805139 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension security-group found enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:66}}
  Mar 03 17:33:24.819184 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension dns-integration is not enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:70}}
  Mar 03 17:33:24.832337 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension qos found enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:66}}
  Mar 03 17:33:24.847909 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.network.drivers.neutron.base [None req-6bbb57f5-2a06-4e8e-9ddd-6da259333fd7 None None] Neutron extension allowed-address-pairs found enabled {{(pid=8984) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:66}}
  Mar 03 17:33:25.221590 ubuntu-bionic-airship-kna1-0014969351 neutron-server[7030]: INFO neutron.wsgi [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] 10.0.1.16,10.0.1.16 "POST /v2.0/security-group-rules HTTP/1.1" status: 201  len: 725 time: 0.1413145
  Mar 03 17:33:25.224900 ubuntu-bionic-airship-kna1-0014969351 octavia-worker[8605]: DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.network_tasks.UpdateVIP' (10c8bae1-19b1-4757-9530-12ac29384565) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=8984) _task_receiver /usr/local/lib/python3.6/dist-packages/taskflow/listeners/logging.py:183}}
  Mar 03 17:33:25.224298 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: DEBUG neutron.agent.resource_cache [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Received new resource SecurityGroupRule: SecurityGroupRule(created_at=2020-03-03T17:33:25Z,description='',direction='ingress',ethertype='IPv4',id=73e2e34d-a813-4846-8f85-2b8daae5d29c,port_range_max=8080,port_range_min=8080,project_id='e821f6bae64f4fa0bca1c230fbf4b364',protocol='tcp',remote_group_id=<?>,remote_ip_prefix=192.0.1.0/32,revision_number=0,security_group_id=14216a23-b9c5-4cb3-b42d-c76b22c643ec,updated_at=2020-03-03T17:33:25Z) {{(pid=7528) record_resource_update /opt/stack/neutron/neutron/agent/resource_cache.py:192}}
  Mar 03 17:33:25.224767 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: DEBUG neutron_lib.callbacks.manager [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_rule_update--9223372036854365827'] for SecurityGroupRule, after_update {{(pid=7528) _notify_loop /usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py:193}}
  Mar 03 17:33:25.225185 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: INFO neutron.agent.securitygroups_rpc [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Security group rule updated ['14216a23-b9c5-4cb3-b42d-c76b22c643ec']
  Mar 03 17:33:25.225605 ubuntu-bionic-airship-kna1-0014969351 neutron-openvswitch-agent[7528]: DEBUG neutron.agent.securitygroups_rpc [None req-137e4288-fac0-490b-b828-8b43a94f675c admin admin] Adding ['d0849264-740f-4a37-b77d-182d6f121067'] devices to the list of devices for which firewall needs to be refreshed {{(pid=7528) _security_group_updated /opt/stack/neutron/neutron/agent/securitygroups_rpc.py:206}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1866353/+subscriptions