debcrafters-packages team mailing list archive
-
debcrafters-packages team
-
Mailing list archive
-
Message #01531
[Bug 2025647] Re: nl_recv returned with error: No buffer space available
hey hector, maybe you could help track this down. how did you create the
VM, or rather, what did you do for this to happen?
** Also affects: libnl3 (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of
Debcrafters packages, which is subscribed to libnl3 in Ubuntu.
https://bugs.launchpad.net/bugs/2025647
Title:
nl_recv returned with error: No buffer space available
Status in libnl3 package in Ubuntu:
New
Status in libvirt package in Ubuntu:
New
Bug description:
We are seeing the following error when creating VMs:
Jun 28 19:33:29 SR-0001 libvirtd[5289]: nl_recv returned with error:
No buffer space available
This problem occurs in deployments with different OS versions.
Ubuntu 20.04.4 LTS
ii libnl-3-200:amd64 3.4.0-1
ii libvirt-daemon 6.0.0-0ubuntu8.16
Ubuntu 22.04 LTS
ii libnl-3-200:amd64 3.5.0-0.1
ii libvirt-daemon 8.0.0-1ubuntu7.5
The host has around 40 native interfaces (bounds, vlans, tunneling bridges, mgmt bridges, physical NICs, etc), and plus 40 or more tap interfaces (depending on the number of VMs allocated), but we observed errors with more than 40 tap interfaces added to the host.
I saw with strace that libvirt receives messages with more than 128k (one message with approximately 170k), apparently the MSG_PEEK mechanism seems to be working, but it may fail in some cases.
In a "good received flow" of incoming messages I see something like the log below, multiple messages for the same sequence number until an error occurs.
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[[{nlmsg_len=1336, nlmsg_type=RTM_NEWLINK, nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517},....
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL,
iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC},
MSG_PEEK|MSG_TRUNC) = 2840
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[[{nlmsg_len=1420, nlmsg_type=RTM_NEWLINK,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694,
nlmsg_pid=3840517},.....
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL,
iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC},
MSG_PEEK|MSG_TRUNC) = 30740
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[[{nlmsg_len=1420, nlmsg_type=RTM_NEWLINK,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517},....
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL,
iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC},
MSG_PEEK|MSG_TRUNC) = 31292
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[[{nlmsg_len=1772, nlmsg_type=RTM_NEWLINK,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517},....
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL, iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 31984
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[[{nlmsg_len=1716, nlmsg_type=RTM_NEWLINK,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517}, ...
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL, iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 31296
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[[{nlmsg_len=1428, nlmsg_type=RTM_NEWLINK,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517},...
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL, iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC},
MSG_PEEK|MSG_TRUNC) = 31280
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[[{nlmsg_len=1428, nlmsg_type=RTM_NEWLINK,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517}, ...
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL,
iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC},
MSG_PEEK|MSG_TRUNC) = 24124
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[[{nlmsg_len=1428, nlmsg_type=RTM_NEWLINK,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517},
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL,
iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC},
MSG_PEEK|MSG_TRUNC) = 20
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[{nlmsg_len=20, nlmsg_type=NLMSG_DONE,
nlmsg_flags=NLM_F_MULTI, nlmsg_seq=1688146694, nlmsg_pid=3840517}, 0],
iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
3840517 sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[{nlmsg_len=48, nlmsg_type=RTM_NEWQDISC,
nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE,
nlmsg_seq=1688146695, nlmsg_pid=0}, {tcm_family=AF_UNSPEC,
tcm_ifindex=if_nametoindex("tapf28271ff-34"), tcm_handle=0,
tcm_parent=4294967295, tcm_info=0}, [{nla_len=12, nla_type=TCA_KIND},
"noqueue"]], iov_len=48}], msg_iovlen=1, msg_controllen=0,
msg_flags=0}, 0) = 48
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=NULL,
iov_len=0}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_TRUNC},
MSG_PEEK|MSG_TRUNC) = 36
3840517 recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12,
msg_iov=[{iov_base=[{nlmsg_len=36, nlmsg_type=NLMSG_ERROR,
nlmsg_flags=NLM_F_CAPPED, nlmsg_seq=1688146695, nlmsg_pid=3840517},
{error=0, msg={nlmsg_len=48, nlmsg_type=RTM_NEWQDISC,
nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE,
nlmsg_seq=1688146695, nlmsg_pid=0}}], iov_len=32768}], msg_iovlen=1,
msg_controllen=0, msg_flags=0}, 0) = 36
There are some abandoned discussions as pointed out in [1], [2], and [3] about MSG_PEEK and libnl to dynamically allocate more buffers, libvirt already sets MSG_PEEK at startup and sets the default buffer to 128k, but even so, messages like this are still logged anyway, what would it be the problem? Any idea?
This problem does not occur when we have few interfaces on the hypervisor, would it be a scaling problem with libnl?
Regards,
Roberto
[1] - https://listman.redhat.com/archives/libvir-list/2017-July/msg00566.html
[2] - https://www.mail-archive.com/libvir-list@xxxxxxxxxx/msg149554.html
[3] - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1794997
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/2025647/+subscriptions