yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1724444] Re: networking-sfc port pair creation fails in create_port_pair_postcommit on kolla deployment

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bernard Cafarelli <bcafarel@xxxxxxxxxx>
Date: Wed, 18 Oct 2017 15:59:31 -0000
Reply-to: Bug 1724444 <1724444@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Thanks, reassigning to networking-sfc for further investigation, as both
conf and database look good

** Project changed: neutron => networking-sfc

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1724444

Title:
  networking-sfc port pair creation fails in create_port_pair_postcommit
  on kolla deployment

Status in networking-sfc:
  New

Bug description:
  I have followed the blog [1] to configure the neutron-sfc using kolla
  deployment tool. After deployment I am able to successfully create an
  instance without any issue, but while creating neutron SFC port pair
  following [2] I am getting following error:

  ~~~
  [root@controller-1 ~]# openstack server list
  +--------------------------------------+-------+--------+-------------------------------+--------+---------+
  | ID                                   | Name  | Status | Networks                      | Image  | Flavor  |
  +--------------------------------------+-------+--------+-------------------------------+--------+---------+
  | 1a59573b-b557-488f-917c-536a2fd21f35 | FW    | ACTIVE | demo-net=10.0.0.14, 10.0.0.12 | cirros | m1.tiny |
  | c540bcc3-ce27-46e9-85e3-cc4d124194ae | demo1 | ACTIVE | demo-net=10.0.0.6             | cirros | m1.tiny |
  +--------------------------------------+-------+--------+-------------------------------+--------+---------+
  [root@controller-1 ~]#

  [root@controller-1 ~]# openstack port list
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+
  | ID                                   | Name | MAC Address       | Fixed IP Addresses                                                        | Status |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+
  | 0a3e2ac8-4ec7-4238-ad08-36528eed6743 | P0   | fa:16:3e:24:03:87 | ip_address='10.0.0.10', subnet_id='08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6'  | DOWN   |
  | 2ce17a23-e80c-453e-81a1-28f49043eef5 |      | fa:16:3e:6d:81:d8 | ip_address='10.0.0.1', subnet_id='08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6'   | DOWN   |
  | 49ba9fd8-5840-4ab3-b9f7-96fc59225b37 | P1   | fa:16:3e:19:35:45 | ip_address='10.0.0.14', subnet_id='08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6'  | ACTIVE |
  | 8f83b61d-3057-4d1c-a075-340271f845ee | P2   | fa:16:3e:44:20:29 | ip_address='10.0.0.12', subnet_id='08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6'  | ACTIVE |
  | a20e66c1-9c95-4c45-8008-8eadc5962cce |      | fa:16:3e:e0:40:dc | ip_address='10.0.0.6', subnet_id='08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6'   | ACTIVE |
  | d0dc19b2-1b63-4c92-a2d2-a2f26af06456 |      | fa:16:3e:37:8d:7f | ip_address='10.0.2.155', subnet_id='72bc8dea-2364-4138-a284-80ffa564362f' | DOWN   |
  | fcb49641-6b32-4064-a5e1-4aa65243352c |      | fa:16:3e:c3:2f:50 | ip_address='10.0.0.2', subnet_id='08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6'   | ACTIVE |
  +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+

  [root@controller-1 ~]# nova list
  +--------------------------------------+-------+--------+------------+-------------+-------------------------------+
  | ID                                   | Name  | Status | Task State | Power State | Networks                      |
  +--------------------------------------+-------+--------+------------+-------------+-------------------------------+
  | 1a59573b-b557-488f-917c-536a2fd21f35 | FW    | ACTIVE | -          | Running     | demo-net=10.0.0.14, 10.0.0.12 |
  | c540bcc3-ce27-46e9-85e3-cc4d124194ae | demo1 | ACTIVE | -          | Running     | demo-net=10.0.0.6             |
  +--------------------------------------+-------+--------+------------+-------------+-------------------------------+
  ~~~

  While creating sfc port pair group.

  ~~~
  [root@controller-1 ~]# openstack sfc port pair create --ingress P1 --egress P2 PPAIR
  create_port_pair_postcommit failed.
  Neutron server returns request_ids: ['req-5e062f5c-2860-422c-964d-af191f6b4c4d']
  ~~~

  
  Following call trace is reported in neutron-server container. 

  ~~~
  2017-10-18 04:13:36.505 23 ERROR networking_sfc.services.sfc.plugin [req-5e062f5c-2860-422c-964d-af191f6b4c4d 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] Cr
  eate port pair failed, deleting port_pair 'e66b3968-7d8a-4fd7-848b-563fc5a2e248': SfcDriverError: create_port_pair_postcommit failed.
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource [req-5e062f5c-2860-422c-964d-af191f6b4c4d 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] create failed
  : No details.: SfcDriverError: create_port_pair_postcommit failed.
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 98, in resource
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 435, in create
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 92, in wrapped
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self.force_reraise()
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self.force_reraise()
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 127, in wrapped
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self.force_reraise()
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 548, in _create
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     obj = do_create(body)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 530, in do_create
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self.force_reraise()
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 523, in do_create
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/plugin.py", line 126, in create_port_pair
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self.delete_port_pair(context, portpair_db['id'])
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self.force_reraise()
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/plugin.py", line 119, in create_port_pair
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self.driver_manager.create_port_pair_postcommit(portpair_context)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 142, in create_port_pair_postcommit
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     self._call_drivers("create_port_pair_postcommit", context)
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 113, in _call_drivers
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource     method=method_name
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource SfcDriverError: create_port_pair_postcommit failed.
  2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource
  2017-10-18 04:13:36.722 23 INFO neutron.wsgi [req-5e062f5c-2860-422c-964d-af191f6b4c4d 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] 192.168.133.254,192.168.133.100 "POST /v2.0/sfc/port_pairs HTTP/1.1" status: 500  len: 302 time: 1.2051969
  ~~~

  
  [1] http://networkop.co.uk/blog/2017/09/08/os-lab-docker/
  [2] http://networkop.co.uk/blog/2017/09/15/os-sfc-skydive/

  Regards,
  Vikrant Aggarwal

To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-sfc/+bug/1724444/+subscriptions
References

[Bug 1724444] [NEW] neutron sfc port group pair creation failed
From: vikrant, 2017-10-18