← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #83398

[Bug 1878299] Re: [floatingip port_forwarding] changing external port to used value hangs with retriable exception

  Thread Previous • Date Previous • • Thread Next 
Reviewed:  https://review.opendev.org/738145
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f3797403486ba43100ff9ec8917d38c8723e12b2
Submitter: Zuul
Branch:    master

commit f3797403486ba43100ff9ec8917d38c8723e12b2
Author: Flavio Fernandes <flaviof@xxxxxxxxxx>
Date:   Thu Jun 25 23:34:32 2020 -0400

    port_forwarding: validate args before invoking db update
    
    Add validator to update_floatingip_port_forwarding so codepath does not
    attempt performing invalid database operation. With that, operation fails
    right away, with a hint on the offending argument(s).
    
    Change-Id: I8284b22c5d691bfd9eadeb8590c3d4b27d261b04
    Closes-Bug: #1878299


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1878299

Title:
  [floatingip port_forwarding] changing external port to used value
  hangs with retriable exception

Status in neutron:
  Fix Released

Bug description:
  When trying to change external-protocol-port to a value that is already used by another pf row, there is no error checking and we end up 'stuck' until the api times out. The neutron db is catching
  the improper config, but the validation should likely not allow it to get that far.

  
  (overcloud) [stack@undercloud-0 ~]$ openstack floating ip port forwarding list ${FIP_UUID}
  +--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
  | ID                                   | Internal Port ID                     | Internal IP Address | Internal Port | External Port | Protocol |
  +--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
  | 5a8515b8-9e03-442f-a8d7-41acd11f59b5 | 63767606-35ea-4c08-b6c8-1216d0c407e8 | 192.168.30.159      |            22 |          2021 | tcp      |
  | 60693ea5-e985-404d-87ce-798dd4f6f4da | d5a31eba-89cb-40a8-ba98-d6ca1a8fffb1 | 192.168.30.84       |            22 |          2020 | tcp      |
  +--------------------------------------+--------------------------------------+---------------------+---------------+---------------+----------+
  (overcloud) [stack@undercloud-0 ~]$
  (overcloud) [stack@undercloud-0 ~]$

  
  # now, doing something invalid: try to change external-protocol-port of $PF_ID2 to the port that is being used by $PF_ID  (60693ea5-e985-404d-87ce-798dd4f6f4da)

  (overcloud) [stack@undercloud-0 ~]$ echo ${PF_ID2}
  5a8515b8-9e03-442f-a8d7-41acd11f59b5

  (overcloud) [stack@undercloud-0 ~]$ time openstack floating ip port
  forwarding set --external-protocol-port 2020 ${FIP_UUID} ${PF_ID2}

  HttpException: 504: Server Error for url:
  http://10.0.0.125:9696/v2.0/floatingips/aee2c979-31a1-453a-b508-319c71fee9dc/port_forwardings/5a8515b8-9e03
  -442f-a8d7-41acd11f59b5, The server didn't respond in time.: 504
  Gateway Time-out

  real    2m3.552s
  user    0m1.672s
  sys     0m0.727s

  =-==-

  [root@controller-0 neutron]# tail -F server.log
  ...
  2020-05-12 17:54:15.900 26 DEBUG neutron.api.v2.base [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body: {'port_forwarding': {'external_port': 2020}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
  2020-05-12 17:54:16.137 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57154) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:16.370 26 DEBUG neutron_lib.db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key 'uniq_port_forwardings0floatingip_id0external_port0protocol'")
  [SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE portforwardings.id = %(portforwardings_id)s]
  [parameters: {'external_port': 2020, 'portforwardings_id': '5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
  (Background on this error at: http://sqlalche.me/e/gkpj) wrapped /usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
  2020-05-12 17:54:16.371 26 DEBUG oslo_db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for function neutron.api.v2.base.Controller._update wrapper /usr/lib/python3.6/site-packages/oslo_db/api.py:156
  2020-05-12 17:54:16.458 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.141', 48380) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:17.882 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.84', 39608) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:17.946 26 DEBUG neutron.api.v2.base [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Request body: {'port_forwarding': {'external_port': 2020}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
  2020-05-12 17:54:18.145 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.38', 57244) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:18.292 26 DEBUG neutron_lib.db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Retry wrapper got retriable exception: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'aee2c979-31a1-453a-b508-319c71fee9dc-2020-tcp' for key 'uniq_port_forwardings0floatingip_id0external_port0protocol'")
  [SQL: UPDATE portforwardings SET external_port=%(external_port)s WHERE portforwardings.id = %(portforwardings_id)s]
  [parameters: {'external_port': 2020, 'portforwardings_id': '5a8515b8-9e03-442f-a8d7-41acd11f59b5'}]
  (Background on this error at: http://sqlalche.me/e/gkpj) wrapped /usr/lib/python3.6/site-packages/neutron_lib/db/api.py:183
  2020-05-12 17:54:18.292 26 DEBUG oslo_db.api [req-f0726861-9f58-4e35-b3b0-15a8bd4ada5b 251059acfcae468c89fa33c988910832 ea7d486cda284d8fa7f3eaf8351f080d - default default] Performing DB retry for function neutron.api.v2.base.Controller._update wrapper /usr/lib/python3.6/site-packages/oslo_db/api.py:156
  2020-05-12 17:54:20.474 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.141', 48566) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:22.027 27 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-8005a4d6-5356-4472-aa28-6debbf9f1175 b4758ff03a5e44b8b23e758ca548d08d 8b966122dec44bed80b5c4fd53919a0d - default default] Attributes excluded by policy engine: ['shared'] _exclude_attributes_by_policy /usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:256
  2020-05-12 17:54:22.166 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57438) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:22.481 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.141', 48664) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:23.903 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.84', 39922) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:24.175 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57534) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:24.489 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.141', 48760) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:25.910 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.84', 40016) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:26.183 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.38', 57626) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:26.494 29 DEBUG neutron.wsgi [-] (29) accepted ('172.17.1.141', 48854) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  2020-05-12 17:54:28.195 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.38', 57722) server /usr/lib/python3.6/site-packages/eventlet/wsgi.py:985
  ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1878299/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next