yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1752418] [NEW] fs017: failure to delete IP causing tempest failures

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Young <halcyondude@xxxxxxxxx>
Date: Wed, 28 Feb 2018 20:38:39 -0000
Reply-to: Bug 1752418 <1752418@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

master promotion is being blocked by a failure in periodic-multinode-
1ctlr-featureset017

master: 66b3734b4a57941c2d66dfc7d7961b2925c46b92_e4e594a6

---

http://38.145.34.55/master.log

2018-02-28 20:21:26,221 2381 INFO     promoter Skipping promotion of
tripleo-ci-testing to current-tripleo, missing successful jobs:
['periodic-multinode-1ctlr-featureset016', 'periodic-multinode-1ctlr-
featureset017', 'periodic-multinode-1ctlr-featureset019']

---

It appears that tempest is failing telemetry tests:

- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-
centos-7-multinode-1ctlr-featureset017-master/45442f0/tempest.html.gz

(hypothesis from mwhahaha) a failure to delete an IP, or failing to
delete it fast enough is causing this

- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-
centos-7-multinode-1ctlr-
featureset017-master/45442f0/subnode-2/var/log/containers/neutron/server.log.txt.gz#_2018-02-28_19_13_39_116

---

2018-02-28 19:13:38.980 27 INFO neutron.wsgi [req-8500f497-a51a-45b6-9713-5d4fd8bcb500 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] 192.168.24.3 "DELETE /v2.0/routers/c8260734-ffe9-4450-a3c7-a4cff0fdc42e HTTP/1.1" status: 204  len: 168 time: 1.3387601
2018-02-28 19:13:38.982 27 DEBUG neutron.wsgi [-] (27) accepted ('192.168.24.3', 53010) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2018-02-28 19:13:39.041 27 DEBUG neutron.db.db_base_plugin_v2 [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Deleting subnet 87f0f9b4-edab-454d-a7d9-0144816b54b5 delete_subnet /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py:999
2018-02-28 19:13:39.042 27 DEBUG neutron_lib.callbacks.manager [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Notify callbacks [] for subnet, before_delete _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-02-28 19:13:39.116 27 INFO neutron.db.db_base_plugin_v2 [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Found port (48798d40-cdc3-415a-9244-2dfaac84f134, 10.100.0.6) having IP allocation on subnet 87f0f9b4-edab-454d-a7d9-0144816b54b5, cannot delete
2018-02-28 19:13:39.118 27 INFO neutron.pecan_wsgi.hooks.translation [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] DELETE failed (client error): There was a conflict when trying to complete your request.
2018-02-28 19:13:39.118 27 DEBUG neutron.pecan_wsgi.hooks.notifier [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] No notification will be sent due to unsuccessful status code: 409 after /usr/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/notifier.py:79

** Affects: neutron
     Importance: Undecided
         Status: Invalid

** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1752418

Title:
  fs017: failure to delete IP causing tempest failures

Status in neutron:
  Invalid

Bug description:
  master promotion is being blocked by a failure in periodic-multinode-
  1ctlr-featureset017

  master: 66b3734b4a57941c2d66dfc7d7961b2925c46b92_e4e594a6

  ---

  http://38.145.34.55/master.log

  2018-02-28 20:21:26,221 2381 INFO     promoter Skipping promotion of
  tripleo-ci-testing to current-tripleo, missing successful jobs:
  ['periodic-multinode-1ctlr-featureset016', 'periodic-multinode-1ctlr-
  featureset017', 'periodic-multinode-1ctlr-featureset019']

  ---

  It appears that tempest is failing telemetry tests:

  - https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-
  centos-7-multinode-1ctlr-featureset017-master/45442f0/tempest.html.gz

  (hypothesis from mwhahaha) a failure to delete an IP, or failing to
  delete it fast enough is causing this

  - https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-
  centos-7-multinode-1ctlr-
  featureset017-master/45442f0/subnode-2/var/log/containers/neutron/server.log.txt.gz#_2018-02-28_19_13_39_116

  ---

  2018-02-28 19:13:38.980 27 INFO neutron.wsgi [req-8500f497-a51a-45b6-9713-5d4fd8bcb500 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] 192.168.24.3 "DELETE /v2.0/routers/c8260734-ffe9-4450-a3c7-a4cff0fdc42e HTTP/1.1" status: 204  len: 168 time: 1.3387601
  2018-02-28 19:13:38.982 27 DEBUG neutron.wsgi [-] (27) accepted ('192.168.24.3', 53010) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
  2018-02-28 19:13:39.041 27 DEBUG neutron.db.db_base_plugin_v2 [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Deleting subnet 87f0f9b4-edab-454d-a7d9-0144816b54b5 delete_subnet /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py:999
  2018-02-28 19:13:39.042 27 DEBUG neutron_lib.callbacks.manager [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Notify callbacks [] for subnet, before_delete _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
  2018-02-28 19:13:39.116 27 INFO neutron.db.db_base_plugin_v2 [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Found port (48798d40-cdc3-415a-9244-2dfaac84f134, 10.100.0.6) having IP allocation on subnet 87f0f9b4-edab-454d-a7d9-0144816b54b5, cannot delete
  2018-02-28 19:13:39.118 27 INFO neutron.pecan_wsgi.hooks.translation [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] DELETE failed (client error): There was a conflict when trying to complete your request.
  2018-02-28 19:13:39.118 27 DEBUG neutron.pecan_wsgi.hooks.notifier [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] No notification will be sent due to unsuccessful status code: 409 after /usr/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/notifier.py:79

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1752418/+subscriptions