yahoo-eng-team team mailing list archive

[Mala Anand <manand@xxxxxxxxxx>]

Public bug reported:

Scenario tested: A tenant has 2 networks, each with a subnet connected through a router. Each tenant gets two VMs each in a network. When these two VMs boot up, they send iperf traffic across the router.
Scale tenants to 500 and measure scaling performance

Results: Scaled up to 330 vms successfully and failed to boot VMs beyond
330VMs.

The neturon servers including plugin were pegged at 100% and see this database integrity violation:
2016-01-28 04:55:04.723 2757 WARNING requests.packages.urllib3.connectionpool [req-7964fc18-a95c-4464-a568-038a453d006e 7a0b7c6414734a93b5dffbc666534690 e53d6cd40c5140c58ec014eb56070917 - - -] Connection pool is full, discarding connection: identity.open.softlayer.com
2016-01-28 04:55:04.772 2757 WARNING requests.packages.urllib3.connectionpool [-] Connection pool is full, discarding connection: identity.open.softlayer.com
2016-01-28 04:55:05.776 2757 WARNING requests.packages.urllib3.connectionpool [req-6b80241f-0928-4d93-a80e-988a7b3e9690 7a0b7c6414734a93b5dffbc666534690 e53d6cd40c5140c58ec014eb56070917 - - -] Connection pool is full, discarding connection: identity.open.softlayer.com
2016-01-28 04:55:55.380 2757 ERROR neutron.agent.ovsdb.impl_idl [-] OVSDB Error:
{"details":"Table Logical_Switch column acls row 36d940b2-26cc-426a-bda6-dd2491f18397 references nonexistent row 0644cffd-71ca-467f-8e1d-6652968870ef in table ACL.","error":"referential integrity violation"}

2016-01-28 04:55:55.443 2757 ERROR neutron.agent.ovsdb.impl_idl [req-0bd44851-ecd9-4957-978f-350b52ada25b cc27c50b17fc4954905db5f3f3eed730 e53d6cd40c5140c58ec014eb56070917 - - -] Traceback (most recent call last):
File "/opt/neutron/lib/python2.7/site-packages/neutron/agent/ovsdb/native/connection.py", line 99, in run
txn.results.put(txn.do_commit())
File "/opt/neutron/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 106, in do_commit
raise RuntimeError(msg)
RuntimeError: OVSDB Error:
{"details":"Table Logical_Switch column acls row 36d940b2-26cc-426a-bda6-dd2491f18397 references nonexistent row 0644cffd-71ca-467f-8e1d-6652968870ef in table ACL.","error":"referential integrity violation"} 

This problem occurs consistently scaling to 500 VMs.

Also have seen this problem occurring in a 5 node devstack installation.
In this case we increased the Neutron api threads to 18 to recreate this
problem.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1544999

Title:
  Encountered database  ACL.","error":"referential integrity violation"
  running OVN stack

Status in neutron:
  New

Bug description:
  Scenario tested: A tenant has 2 networks, each with a subnet connected through a router. Each tenant gets two VMs each in a network. When these two VMs boot up, they send iperf traffic across the router.
  Scale tenants to 500 and measure scaling performance

  Results: Scaled up to 330 vms successfully and failed to boot VMs
  beyond 330VMs.

  The neturon servers including plugin were pegged at 100% and see this database integrity violation:
  2016-01-28 04:55:04.723 2757 WARNING requests.packages.urllib3.connectionpool [req-7964fc18-a95c-4464-a568-038a453d006e 7a0b7c6414734a93b5dffbc666534690 e53d6cd40c5140c58ec014eb56070917 - - -] Connection pool is full, discarding connection: identity.open.softlayer.com
  2016-01-28 04:55:04.772 2757 WARNING requests.packages.urllib3.connectionpool [-] Connection pool is full, discarding connection: identity.open.softlayer.com
  2016-01-28 04:55:05.776 2757 WARNING requests.packages.urllib3.connectionpool [req-6b80241f-0928-4d93-a80e-988a7b3e9690 7a0b7c6414734a93b5dffbc666534690 e53d6cd40c5140c58ec014eb56070917 - - -] Connection pool is full, discarding connection: identity.open.softlayer.com
  2016-01-28 04:55:55.380 2757 ERROR neutron.agent.ovsdb.impl_idl [-] OVSDB Error:
  {"details":"Table Logical_Switch column acls row 36d940b2-26cc-426a-bda6-dd2491f18397 references nonexistent row 0644cffd-71ca-467f-8e1d-6652968870ef in table ACL.","error":"referential integrity violation"}

  2016-01-28 04:55:55.443 2757 ERROR neutron.agent.ovsdb.impl_idl [req-0bd44851-ecd9-4957-978f-350b52ada25b cc27c50b17fc4954905db5f3f3eed730 e53d6cd40c5140c58ec014eb56070917 - - -] Traceback (most recent call last):
  File "/opt/neutron/lib/python2.7/site-packages/neutron/agent/ovsdb/native/connection.py", line 99, in run
  txn.results.put(txn.do_commit())
  File "/opt/neutron/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 106, in do_commit
  raise RuntimeError(msg)
  RuntimeError: OVSDB Error:
  {"details":"Table Logical_Switch column acls row 36d940b2-26cc-426a-bda6-dd2491f18397 references nonexistent row 0644cffd-71ca-467f-8e1d-6652968870ef in table ACL.","error":"referential integrity violation"} 

  This problem occurs consistently scaling to 500 VMs.

  Also have seen this problem occurring in a 5 node devstack
  installation.  In this case we increased the Neutron api threads to 18
  to recreate this problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1544999/+subscriptions