yahoo-eng-team team mailing list archive

[Dmitrii Shcherbakov <1761556@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

OpenStack Queens from UCA (xenial, GA kernel), 2 external subnets (one
routed provider network), 2 tenant subnets, all subnets in the same
address scope to trigger "fast exit" logic.

Tenant subnet cidr: 192.168.100.0/24
Other tenant subnet cidr: 192.168.200.0/24

Relevant agent configs:
http://paste.openstack.org/show/718514/

Commands and outputs:
http://paste.openstack.org/show/JFYmGJwF1pdtliQOfXgd/

Overall, a similar situation as with
https://bugs.launchpad.net/neutron/+bug/1759956 but with one tenant
subnet at first for which routes and rules do not get deleted at all.

Problem description:

* router add subnet tenantsubnet
* routes in fip namespace and rules in qrouter namespace get created and a distributed port gets created for DVR;
* router remove subnet tenantsubnet
* routes are still there, no new logged events in DVR l3 agent logs

If two networks are added then removing one of them triggers removal of
routes and rules and new messages are logged in l3 agent log (the rules
removed are affected by pad.lv/1759956).

A sequence of add subnet/remove subnet commands may result in errors
logged in l3 agent logs: http://paste.openstack.org/show/718511/


Sometimes after re-adding a tenantsubnet in presence of othertenantsubnet a proper route is added for a few seconds but then removed:

# just do some operations
(openstack) router add subnet pubrouter tenantsubnet
(openstack) router add subnet pubrouter othertenantsubnet
(openstack) router add subnet pubrouter tenantsubnet
(openstack) router add subnet pubrouter tenantsubnet
(openstack) router remove subnet pubrouter tenantsubnet

# lots of errors, see http://paste.openstack.org/show/718511/

# try again without restarting agents
(openstack) router add subnet pubrouter tenantsubnet # ran client command

# ... got 192.168.100.0/24 here for a few seconds while l3 agent was doing something
10.232.16.0/21 dev fg-7f42af4f-ad  proto kernel  scope link  src 10.232.17.5 
169.254.106.114/31 dev fpr-3182a7c6-b  proto kernel  scope link  src 169.254.106.115 
192.168.100.0/24 via 169.254.106.114 dev fpr-3182a7c6-b 
192.168.200.0/24 via 169.254.106.114 dev fpr-3182a7c6-b 

# finished server and l3 agent finished processing "router add subnet pubrouter tenantsubnet"
# route got deleted
root@ipotane:~# ip netns exec fip-64ab1ec3-4927-4f09-87f9-804e7f4f8748 ip r
10.232.16.0/21 dev fg-7f42af4f-ad  proto kernel  scope link  src 10.232.17.5 
169.254.106.114/31 dev fpr-3182a7c6-b  proto kernel  scope link  src 169.254.106.115 
192.168.200.0/24 via 169.254.106.114 dev fpr-3182a7c6-b  

There is something wrong with how tenant network add/remove
notifications are sent it seems because on first removal of a tenant
network nothing is logged in l3 agent logs but there is activity in
neutron server logs.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: cpe-onsite

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1761556

Title:
  [dvr][fast-exit] router add/remove subnet operations are not
  idempotent

Status in neutron:
  New

Bug description:
  OpenStack Queens from UCA (xenial, GA kernel), 2 external subnets (one
  routed provider network), 2 tenant subnets, all subnets in the same
  address scope to trigger "fast exit" logic.

  Tenant subnet cidr: 192.168.100.0/24
  Other tenant subnet cidr: 192.168.200.0/24

  Relevant agent configs:
  http://paste.openstack.org/show/718514/

  Commands and outputs:
  http://paste.openstack.org/show/JFYmGJwF1pdtliQOfXgd/

  Overall, a similar situation as with
  https://bugs.launchpad.net/neutron/+bug/1759956 but with one tenant
  subnet at first for which routes and rules do not get deleted at all.

  Problem description:

  * router add subnet tenantsubnet
  * routes in fip namespace and rules in qrouter namespace get created and a distributed port gets created for DVR;
  * router remove subnet tenantsubnet
  * routes are still there, no new logged events in DVR l3 agent logs

  If two networks are added then removing one of them triggers removal
  of routes and rules and new messages are logged in l3 agent log (the
  rules removed are affected by pad.lv/1759956).

  A sequence of add subnet/remove subnet commands may result in errors
  logged in l3 agent logs: http://paste.openstack.org/show/718511/

  
  Sometimes after re-adding a tenantsubnet in presence of othertenantsubnet a proper route is added for a few seconds but then removed:

  # just do some operations
  (openstack) router add subnet pubrouter tenantsubnet
  (openstack) router add subnet pubrouter othertenantsubnet
  (openstack) router add subnet pubrouter tenantsubnet
  (openstack) router add subnet pubrouter tenantsubnet
  (openstack) router remove subnet pubrouter tenantsubnet

  # lots of errors, see http://paste.openstack.org/show/718511/

  # try again without restarting agents
  (openstack) router add subnet pubrouter tenantsubnet # ran client command

  # ... got 192.168.100.0/24 here for a few seconds while l3 agent was doing something
  10.232.16.0/21 dev fg-7f42af4f-ad  proto kernel  scope link  src 10.232.17.5 
  169.254.106.114/31 dev fpr-3182a7c6-b  proto kernel  scope link  src 169.254.106.115 
  192.168.100.0/24 via 169.254.106.114 dev fpr-3182a7c6-b 
  192.168.200.0/24 via 169.254.106.114 dev fpr-3182a7c6-b 

  # finished server and l3 agent finished processing "router add subnet pubrouter tenantsubnet"
  # route got deleted
  root@ipotane:~# ip netns exec fip-64ab1ec3-4927-4f09-87f9-804e7f4f8748 ip r
  10.232.16.0/21 dev fg-7f42af4f-ad  proto kernel  scope link  src 10.232.17.5 
  169.254.106.114/31 dev fpr-3182a7c6-b  proto kernel  scope link  src 169.254.106.115 
  192.168.200.0/24 via 169.254.106.114 dev fpr-3182a7c6-b  

  There is something wrong with how tenant network add/remove
  notifications are sent it seems because on first removal of a tenant
  network nothing is logged in l3 agent logs but there is activity in
  neutron server logs.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1761556/+subscriptions