yahoo-eng-team team mailing list archive

[Assaf Muller <amuller@xxxxxxxxxx>]

** Project changed: neutron => networking-ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1544999

Title:
  Encountered database  ACL.","error":"referential integrity violation"
  running OVN stack

Status in networking-ovn:
  Confirmed

Bug description:
  Scenario tested: A tenant has 2 networks, each with a subnet connected through a router. Each tenant gets two VMs each in a network. When these two VMs boot up, they send iperf traffic across the router.
  Scale tenants to 500 and measure scaling performance

  Results: Scaled up to 330 vms successfully and failed to boot VMs
  beyond 330VMs.

  The neturon servers including plugin were pegged at 100% and see this database integrity violation:
  2016-01-28 04:55:04.723 2757 WARNING requests.packages.urllib3.connectionpool [req-7964fc18-a95c-4464-a568-038a453d006e 7a0b7c6414734a93b5dffbc666534690 e53d6cd40c5140c58ec014eb56070917 - - -] Connection pool is full, discarding connection: identity.open.softlayer.com
  2016-01-28 04:55:04.772 2757 WARNING requests.packages.urllib3.connectionpool [-] Connection pool is full, discarding connection: identity.open.softlayer.com
  2016-01-28 04:55:05.776 2757 WARNING requests.packages.urllib3.connectionpool [req-6b80241f-0928-4d93-a80e-988a7b3e9690 7a0b7c6414734a93b5dffbc666534690 e53d6cd40c5140c58ec014eb56070917 - - -] Connection pool is full, discarding connection: identity.open.softlayer.com
  2016-01-28 04:55:55.380 2757 ERROR neutron.agent.ovsdb.impl_idl [-] OVSDB Error:
  {"details":"Table Logical_Switch column acls row 36d940b2-26cc-426a-bda6-dd2491f18397 references nonexistent row 0644cffd-71ca-467f-8e1d-6652968870ef in table ACL.","error":"referential integrity violation"}

  2016-01-28 04:55:55.443 2757 ERROR neutron.agent.ovsdb.impl_idl [req-0bd44851-ecd9-4957-978f-350b52ada25b cc27c50b17fc4954905db5f3f3eed730 e53d6cd40c5140c58ec014eb56070917 - - -] Traceback (most recent call last):
  File "/opt/neutron/lib/python2.7/site-packages/neutron/agent/ovsdb/native/connection.py", line 99, in run
  txn.results.put(txn.do_commit())
  File "/opt/neutron/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 106, in do_commit
  raise RuntimeError(msg)
  RuntimeError: OVSDB Error:
  {"details":"Table Logical_Switch column acls row 36d940b2-26cc-426a-bda6-dd2491f18397 references nonexistent row 0644cffd-71ca-467f-8e1d-6652968870ef in table ACL.","error":"referential integrity violation"} 

  This problem occurs consistently scaling to 500 VMs.

  Also have seen this problem occurring in a 5 node devstack
  installation.  In this case we increased the Neutron api threads to 18
  to recreate this problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-ovn/+bug/1544999/+subscriptions