yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #75925
[Bug 1804842] [NEW] When kill(sometines doesn't restart) the ovs switch or restart it in the compute nodes vm conectivity is lost
Public bug reported:
OSP 14
3 controllers + 3 computes + dvr
several vms in one compute with fip.
Problem 1 :
root@compute-2 heat-admin]# systemctl restart openvswitch
fip conectivity with undercloud vm is lost and no recover
conectivity with other computes is lost, but it is recovered restarting
neutron openvswitch agent container
root@compute-2 heat-admin]# systemctl restart openvswitch.
Problem 2:
After kill -9 "pid ovs switch"
Sometimes the ovs switch in not restarted automatically
Same problems that in the scenario1
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+ 49054 1 1 13:20 ? 00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root 49217 17666 0 13:21 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+ 49054 1 0 13:20 ? 00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root 49421 17666 0 13:21 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+ 49054 1 0 13:20 ? 00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root 49423 17666 0 13:21 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# kill -9 49054
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49610 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49628 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]# date
Fri Nov 23 13:22:22 UTC 2018
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49788 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49790 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
overcloud) [stack@undercloud-0 ~]$ openstack versions show
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
| Region Name | Service Type | Version | Status | Endpoint | Min Microversion | Max Microversion |
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
| regionOne | block-storage | 2.0 | DEPRECATED | http://10.0.0.101:8776/v2/ | None | None |
| regionOne | block-storage | 3.0 | CURRENT | http://10.0.0.101:8776/v3/ | 3.0 | 3.55 |
| regionOne | placement | None | CURRENT | http://10.0.0.101:8778/placement/ | None | None |
| regionOne | network | 2.0 | CURRENT | http://10.0.0.101:9696/v2.0/ | None | None |
| regionOne | alarm | 2.0 | CURRENT | http://10.0.0.101:8042/v2 | None | None |
| regionOne | cloudformation | 1.0 | CURRENT | http://10.0.0.101:8000/v1/ | None | None |
| regionOne | event | 2.0 | CURRENT | http://10.0.0.101:8977/v2 | None | None |
| regionOne | orchestration | 1.0 | CURRENT | http://10.0.0.101:8004/v1/ | None | None |
| regionOne | object-store | 1.0 | CURRENT | http://10.0.0.101:8080/v1/ | None | None |
| regionOne | compute | 2.0 | SUPPORTED | http://10.0.0.101:8774/v2/ | None | None |
| regionOne | compute | 2.1 | CURRENT | http://10.0.0.101:8774/v2.1/ | 2.1 | 2.65 |
| regionOne | image | 2.0 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.1 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.2 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.3 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.4 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.5 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.6 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.7 | CURRENT | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | metric | 1.0 | CURRENT | http://10.0.0.101:8041/v1/ | None | None |
| regionOne | identity | 3.10 | CURRENT | http://10.0.0.101:5000/v3/ | None | None |
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
(overcloud) [stack@undercloud-0 ~]$ cat /etc/re
redhat-lsb/ redhat-release request-key.conf request-key.d/ resolv.conf
(overcloud) [stack@undercloud-0 ~]$ cat /etc/re
redhat-lsb/ redhat-release request-key.conf request-key.d/ resolv.conf
(overcloud) [stack@undercloud-0 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1804842
Title:
When kill(sometines doesn't restart) the ovs switch or restart it in
the compute nodes vm conectivity is lost
Status in neutron:
New
Bug description:
OSP 14
3 controllers + 3 computes + dvr
several vms in one compute with fip.
Problem 1 :
root@compute-2 heat-admin]# systemctl restart openvswitch
fip conectivity with undercloud vm is lost and no recover
conectivity with other computes is lost, but it is recovered
restarting neutron openvswitch agent container
root@compute-2 heat-admin]# systemctl restart openvswitch.
Problem 2:
After kill -9 "pid ovs switch"
Sometimes the ovs switch in not restarted automatically
Same problems that in the scenario1
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+ 49054 1 1 13:20 ? 00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root 49217 17666 0 13:21 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+ 49054 1 0 13:20 ? 00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root 49421 17666 0 13:21 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
openvsw+ 49054 1 0 13:20 ? 00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
root 49423 17666 0 13:21 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# kill -9 49054
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49610 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49628 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]# date
Fri Nov 23 13:22:22 UTC 2018
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49788 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]#
[root@compute-2 heat-admin]# ps -ef | grep ovs
root 10558 7292 0 12:09 ? 00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock
42435 46886 46871 0 13:17 ? 00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh
root 49790 17666 0 13:22 pts/0 00:00:00 grep --color=auto ovs
overcloud) [stack@undercloud-0 ~]$ openstack versions show
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
| Region Name | Service Type | Version | Status | Endpoint | Min Microversion | Max Microversion |
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
| regionOne | block-storage | 2.0 | DEPRECATED | http://10.0.0.101:8776/v2/ | None | None |
| regionOne | block-storage | 3.0 | CURRENT | http://10.0.0.101:8776/v3/ | 3.0 | 3.55 |
| regionOne | placement | None | CURRENT | http://10.0.0.101:8778/placement/ | None | None |
| regionOne | network | 2.0 | CURRENT | http://10.0.0.101:9696/v2.0/ | None | None |
| regionOne | alarm | 2.0 | CURRENT | http://10.0.0.101:8042/v2 | None | None |
| regionOne | cloudformation | 1.0 | CURRENT | http://10.0.0.101:8000/v1/ | None | None |
| regionOne | event | 2.0 | CURRENT | http://10.0.0.101:8977/v2 | None | None |
| regionOne | orchestration | 1.0 | CURRENT | http://10.0.0.101:8004/v1/ | None | None |
| regionOne | object-store | 1.0 | CURRENT | http://10.0.0.101:8080/v1/ | None | None |
| regionOne | compute | 2.0 | SUPPORTED | http://10.0.0.101:8774/v2/ | None | None |
| regionOne | compute | 2.1 | CURRENT | http://10.0.0.101:8774/v2.1/ | 2.1 | 2.65 |
| regionOne | image | 2.0 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.1 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.2 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.3 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.4 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.5 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.6 | SUPPORTED | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | image | 2.7 | CURRENT | http://10.0.0.101:9292/v2/ | None | None |
| regionOne | metric | 1.0 | CURRENT | http://10.0.0.101:8041/v1/ | None | None |
| regionOne | identity | 3.10 | CURRENT | http://10.0.0.101:5000/v3/ | None | None |
+-------------+----------------+---------+------------+-----------------------------------+------------------+------------------+
(overcloud) [stack@undercloud-0 ~]$ cat /etc/re
redhat-lsb/ redhat-release request-key.conf request-key.d/ resolv.conf
(overcloud) [stack@undercloud-0 ~]$ cat /etc/re
redhat-lsb/ redhat-release request-key.conf request-key.d/ resolv.conf
(overcloud) [stack@undercloud-0 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1804842/+subscriptions
Follow ups
