yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #38782
[Bug 1498506] [NEW] LB_NFV[Alchemy] linux kernel soft lockup after create 200 LB on sinalge tenant with 30M traffic on each LB
Public bug reported:
LB_NFV[Alchemy] linux kernel soft lockup after create 200 LB on sinalge tenant with 30M traffic on each LB
CentOS Linux release 7.1.1503 (Core)
reproduce step
1 create 1 client and 1 backend server by docker container
2 create a lb and add backend as member
3 send 30M traffic with 100 session by siege
setsid siege -t1000H {vip_ip_address}/test_150KB.iso -c 100
4 repeat step 1 to 3 for create 200 LB with traffic
after step 4 , you will look at a prompt on LB agent host node
and have some traffic will can not pass on LB from client and can't ping to vip address from client container
log
[root@scalenetwork1 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
Message from syslogd@scalenetwork1 at Sep 13 23:24:05 ...
kernel:BUG: soft lockup - CPU#5 stuck for 23s! [rcuos/6:24] <<<<<<<<<<<<<<<<<<<<
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]# sar -P ALL 60 1
Linux 3.10.0-229.11.1.el7.x86_64 (scalenetwork1) 09/13/2015 _x86_64_ (8 CPU)
11:24:22 PM CPU %user %nice %system %iowait %steal %idle
11:25:22 PM all 11.71 0.00 43.13 0.00 0.00 45.17
11:25:22 PM 0 13.81 0.00 34.69 0.00 0.00 51.49
11:25:22 PM 1 6.11 0.00 19.80 0.00 0.00 74.10
11:25:22 PM 2 14.09 0.00 34.71 0.00 0.00 51.21
11:25:22 PM 3 13.66 0.00 35.00 0.00 0.00 51.34
11:25:22 PM 4 15.34 0.00 38.34 0.00 0.00 46.32
11:25:22 PM 5 0.05 0.00 99.78 0.00 0.00 0.17
11:25:22 PM 6 14.95 0.00 41.34 0.00 0.00 43.71
11:25:22 PM 7 15.80 0.00 39.70 0.00 0.00 44.50
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 11.71 0.00 43.13 0.00 0.00 45.17
Average: 0 13.81 0.00 34.69 0.00 0.00 51.49
Average: 1 6.11 0.00 19.80 0.00 0.00 74.10
Average: 2 14.09 0.00 34.71 0.00 0.00 51.21
Average: 3 13.66 0.00 35.00 0.00 0.00 51.34
Average: 4 15.34 0.00 38.34 0.00 0.00 46.32
Average: 5 0.05 0.00 99.78 0.00 0.00 0.17
Average: 6 14.95 0.00 41.34 0.00 0.00 43.71
Average: 7 15.80 0.00 39.70 0.00 0.00 44.50
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1498506
Title:
LB_NFV[Alchemy] linux kernel soft lockup after create 200 LB on
sinalge tenant with 30M traffic on each LB
Status in neutron:
New
Bug description:
LB_NFV[Alchemy] linux kernel soft lockup after create 200 LB on sinalge tenant with 30M traffic on each LB
CentOS Linux release 7.1.1503 (Core)
reproduce step
1 create 1 client and 1 backend server by docker container
2 create a lb and add backend as member
3 send 30M traffic with 100 session by siege
setsid siege -t1000H {vip_ip_address}/test_150KB.iso -c 100
4 repeat step 1 to 3 for create 200 LB with traffic
after step 4 , you will look at a prompt on LB agent host node
and have some traffic will can not pass on LB from client and can't ping to vip address from client container
log
[root@scalenetwork1 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
Message from syslogd@scalenetwork1 at Sep 13 23:24:05 ...
kernel:BUG: soft lockup - CPU#5 stuck for 23s! [rcuos/6:24] <<<<<<<<<<<<<<<<<<<<
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]# sar -P ALL 60 1
Linux 3.10.0-229.11.1.el7.x86_64 (scalenetwork1) 09/13/2015 _x86_64_ (8 CPU)
11:24:22 PM CPU %user %nice %system %iowait %steal %idle
11:25:22 PM all 11.71 0.00 43.13 0.00 0.00 45.17
11:25:22 PM 0 13.81 0.00 34.69 0.00 0.00 51.49
11:25:22 PM 1 6.11 0.00 19.80 0.00 0.00 74.10
11:25:22 PM 2 14.09 0.00 34.71 0.00 0.00 51.21
11:25:22 PM 3 13.66 0.00 35.00 0.00 0.00 51.34
11:25:22 PM 4 15.34 0.00 38.34 0.00 0.00 46.32
11:25:22 PM 5 0.05 0.00 99.78 0.00 0.00 0.17
11:25:22 PM 6 14.95 0.00 41.34 0.00 0.00 43.71
11:25:22 PM 7 15.80 0.00 39.70 0.00 0.00 44.50
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 11.71 0.00 43.13 0.00 0.00 45.17
Average: 0 13.81 0.00 34.69 0.00 0.00 51.49
Average: 1 6.11 0.00 19.80 0.00 0.00 74.10
Average: 2 14.09 0.00 34.71 0.00 0.00 51.21
Average: 3 13.66 0.00 35.00 0.00 0.00 51.34
Average: 4 15.34 0.00 38.34 0.00 0.00 46.32
Average: 5 0.05 0.00 99.78 0.00 0.00 0.17
Average: 6 14.95 0.00 41.34 0.00 0.00 43.71
Average: 7 15.80 0.00 39.70 0.00 0.00 44.50
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1498506/+subscriptions
Follow ups
