← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1804842] [NEW] When kill(sometines doesn't restart) the ovs switch or restart it in the compute nodes vm conectivity is lost

 

Public bug reported:

OSP 14

3 controllers + 3 computes + dvr

several vms in one compute with fip.

Problem 1 :

root@compute-2 heat-admin]# systemctl restart openvswitch

fip conectivity with undercloud vm is lost and no recover

conectivity with other computes is lost, but it is recovered restarting
neutron openvswitch agent container

root@compute-2 heat-admin]# systemctl restart openvswitch.

Problem 2:

After kill -9 "pid ovs switch"

Sometimes the ovs switch in not restarted automatically

Same problems that in the scenario1


[root@compute-2 heat-admin]# ps -ef | grep ovs
root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+   49054       1  1 13:20 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root       49217   17666  0 13:21 pts/0    00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# ps -ef | grep ovs
root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+   49054       1  0 13:20 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root       49421   17666  0 13:21 pts/0    00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# ps -ef | grep ovs
root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+   49054       1  0 13:20 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root       49423   17666  0 13:21 pts/0    00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# kill -9 49054
[root@compute-2 heat-admin]# ps -ef | grep ovs
root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root       49610   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# ps -ef | grep ovs
root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root       49628   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs

[root@compute-2 heat-admin]# date
Fri Nov 23 13:22:22 UTC 2018
[root@compute-2 heat-admin]# ps -ef | grep ovs
root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root       49788   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# 
[root@compute-2 heat-admin]# ps -ef | grep ovs
root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root       49790   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs


overcloud) [stack@undercloud-0 ~]$ openstack versions show
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
| Region Name | Service Type   | Version | Status     | Endpoint                          | Min Microversion | Max Microversion |
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
| regionOne   | block-storage  | 2.0     | DEPRECATED | http://10.0.0.101:8776/v2/        | None             | None             |
| regionOne   | block-storage  | 3.0     | CURRENT    | http://10.0.0.101:8776/v3/        | 3.0              | 3.55             |
| regionOne   | placement      | None    | CURRENT    | http://10.0.0.101:8778/placement/ | None             | None             |
| regionOne   | network        | 2.0     | CURRENT    | http://10.0.0.101:9696/v2.0/      | None             | None             |
| regionOne   | alarm          | 2.0     | CURRENT    | http://10.0.0.101:8042/v2         | None             | None             |
| regionOne   | cloudformation | 1.0     | CURRENT    | http://10.0.0.101:8000/v1/        | None             | None             |
| regionOne   | event          | 2.0     | CURRENT    | http://10.0.0.101:8977/v2         | None             | None             |
| regionOne   | orchestration  | 1.0     | CURRENT    | http://10.0.0.101:8004/v1/        | None             | None             |
| regionOne   | object-store   | 1.0     | CURRENT    | http://10.0.0.101:8080/v1/        | None             | None             |
| regionOne   | compute        | 2.0     | SUPPORTED  | http://10.0.0.101:8774/v2/        | None             | None             |
| regionOne   | compute        | 2.1     | CURRENT    | http://10.0.0.101:8774/v2.1/      | 2.1              | 2.65             |
| regionOne   | image          | 2.0     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | image          | 2.1     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | image          | 2.2     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | image          | 2.3     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | image          | 2.4     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | image          | 2.5     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | image          | 2.6     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | image          | 2.7     | CURRENT    | http://10.0.0.101:9292/v2/        | None             | None             |
| regionOne   | metric         | 1.0     | CURRENT    | http://10.0.0.101:8041/v1/        | None             | None             |
| regionOne   | identity       | 3.10    | CURRENT    | http://10.0.0.101:5000/v3/        | None             | None             |
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
(overcloud) [stack@undercloud-0 ~]$ cat /etc/re
redhat-lsb/       redhat-release    request-key.conf  request-key.d/    resolv.conf       
(overcloud) [stack@undercloud-0 ~]$ cat /etc/re
redhat-lsb/       redhat-release    request-key.conf  request-key.d/    resolv.conf       
(overcloud) [stack@undercloud-0 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1804842

Title:
  When kill(sometines doesn't restart) the ovs switch or restart it in
  the compute nodes   vm conectivity is lost

Status in neutron:
  New

Bug description:
  OSP 14

  3 controllers + 3 computes + dvr

  several vms in one compute with fip.

  Problem 1 :

  root@compute-2 heat-admin]# systemctl restart openvswitch

  fip conectivity with undercloud vm is lost and no recover

  conectivity with other computes is lost, but it is recovered
  restarting neutron openvswitch agent container

  root@compute-2 heat-admin]# systemctl restart openvswitch.

  Problem 2:

  After kill -9 "pid ovs switch"

  Sometimes the ovs switch in not restarted automatically

  Same problems that in the scenario1

  
  [root@compute-2 heat-admin]# ps -ef | grep ovs
  root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
  42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
  openvsw+   49054       1  1 13:20 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
  root       49217   17666  0 13:21 pts/0    00:00:00 grep --color=auto ovs
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# ps -ef | grep ovs
  root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
  42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
  openvsw+   49054       1  0 13:20 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
  root       49421   17666  0 13:21 pts/0    00:00:00 grep --color=auto ovs
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# ps -ef | grep ovs
  root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
  42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
  openvsw+   49054       1  0 13:20 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
  root       49423   17666  0 13:21 pts/0    00:00:00 grep --color=auto ovs
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# kill -9 49054
  [root@compute-2 heat-admin]# ps -ef | grep ovs
  root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
  42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
  root       49610   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# ps -ef | grep ovs
  root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
  42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
  root       49628   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs

  [root@compute-2 heat-admin]# date
  Fri Nov 23 13:22:22 UTC 2018
  [root@compute-2 heat-admin]# ps -ef | grep ovs
  root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
  42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
  root       49788   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# 
  [root@compute-2 heat-admin]# ps -ef | grep ovs
  root       10558    7292  0 12:09 ?        00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
  42435      46886   46871  0 13:17 ?        00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
  root       49790   17666  0 13:22 pts/0    00:00:00 grep --color=auto ovs

  
  overcloud) [stack@undercloud-0 ~]$ openstack versions show
  +-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
  | Region Name | Service Type   | Version | Status     | Endpoint                          | Min Microversion | Max Microversion |
  +-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
  | regionOne   | block-storage  | 2.0     | DEPRECATED | http://10.0.0.101:8776/v2/        | None             | None             |
  | regionOne   | block-storage  | 3.0     | CURRENT    | http://10.0.0.101:8776/v3/        | 3.0              | 3.55             |
  | regionOne   | placement      | None    | CURRENT    | http://10.0.0.101:8778/placement/ | None             | None             |
  | regionOne   | network        | 2.0     | CURRENT    | http://10.0.0.101:9696/v2.0/      | None             | None             |
  | regionOne   | alarm          | 2.0     | CURRENT    | http://10.0.0.101:8042/v2         | None             | None             |
  | regionOne   | cloudformation | 1.0     | CURRENT    | http://10.0.0.101:8000/v1/        | None             | None             |
  | regionOne   | event          | 2.0     | CURRENT    | http://10.0.0.101:8977/v2         | None             | None             |
  | regionOne   | orchestration  | 1.0     | CURRENT    | http://10.0.0.101:8004/v1/        | None             | None             |
  | regionOne   | object-store   | 1.0     | CURRENT    | http://10.0.0.101:8080/v1/        | None             | None             |
  | regionOne   | compute        | 2.0     | SUPPORTED  | http://10.0.0.101:8774/v2/        | None             | None             |
  | regionOne   | compute        | 2.1     | CURRENT    | http://10.0.0.101:8774/v2.1/      | 2.1              | 2.65             |
  | regionOne   | image          | 2.0     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | image          | 2.1     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | image          | 2.2     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | image          | 2.3     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | image          | 2.4     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | image          | 2.5     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | image          | 2.6     | SUPPORTED  | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | image          | 2.7     | CURRENT    | http://10.0.0.101:9292/v2/        | None             | None             |
  | regionOne   | metric         | 1.0     | CURRENT    | http://10.0.0.101:8041/v1/        | None             | None             |
  | regionOne   | identity       | 3.10    | CURRENT    | http://10.0.0.101:5000/v3/        | None             | None             |
  +-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
  (overcloud) [stack@undercloud-0 ~]$ cat /etc/re
  redhat-lsb/       redhat-release    request-key.conf  request-key.d/    resolv.conf       
  (overcloud) [stack@undercloud-0 ~]$ cat /etc/re
  redhat-lsb/       redhat-release    request-key.conf  request-key.d/    resolv.conf       
  (overcloud) [stack@undercloud-0 ~]$ cat /etc/redhat-release 
  Red Hat Enterprise Linux Server release 7.6 (Maipo)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1804842/+subscriptions


Follow ups