yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1813198] [NEW] TestNetworkBasicOps:test_subnet_details intermittently fails with "cat: can't open '/var/run/udhcpc..pid': No such file or directory"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 24 Jan 2019 19:41:55 -0000
Reply-to: Bug 1813198 <1813198@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen here:

http://logs.openstack.org/78/570078/17/check/tempest-slow/161ea32/job-
output.txt.gz#_2019-01-24_18_26_22_886987

2019-01-24 18:26:22.886987 | controller | {0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details [125.520364s] ... FAILED
2019-01-24 18:26:22.887067 | controller |
2019-01-24 18:26:22.887166 | controller | Captured traceback:
2019-01-24 18:26:22.887251 | controller | ~~~~~~~~~~~~~~~~~~~
2019-01-24 18:26:22.887370 | controller |     Traceback (most recent call last):
2019-01-24 18:26:22.887545 | controller |       File "tempest/common/utils/__init__.py", line 89, in wrapper
2019-01-24 18:26:22.887663 | controller |         return f(*func_args, **func_kwargs)
2019-01-24 18:26:22.887868 | controller |       File "tempest/scenario/test_network_basic_ops.py", line 629, in test_subnet_details
2019-01-24 18:26:22.887952 | controller |         renew_delay),
2019-01-24 18:26:22.888141 | controller |       File "tempest/lib/common/utils/test_utils.py", line 107, in call_until_true
2019-01-24 18:26:22.888237 | controller |         if func(*args, **kwargs):
2019-01-24 18:26:22.888443 | controller |       File "tempest/scenario/test_network_basic_ops.py", line 621, in check_new_dns_server
2019-01-24 18:26:22.888583 | controller |         dhcp_client=CONF.scenario.dhcp_client)
2019-01-24 18:26:22.888776 | controller |       File "tempest/common/utils/linux/remote_client.py", line 140, in renew_lease
2019-01-24 18:26:22.888957 | controller |         return getattr(self, '_renew_lease_' + dhcp_client)(fixed_ip=fixed_ip)
2019-01-24 18:26:22.889161 | controller |       File "tempest/common/utils/linux/remote_client.py", line 116, in _renew_lease_udhcpc
2019-01-24 18:26:22.889279 | controller |         format(path=file_path, nic=nic_name))
2019-01-24 18:26:22.889474 | controller |       File "tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
2019-01-24 18:26:22.889595 | controller |         return function(self, *args, **kwargs)
2019-01-24 18:26:22.889793 | controller |       File "tempest/lib/common/utils/linux/remote_client.py", line 108, in exec_command
2019-01-24 18:26:22.890231 | controller |         return self.ssh_client.exec_command(cmd)
2019-01-24 18:26:22.890402 | controller |       File "tempest/lib/common/ssh.py", line 202, in exec_command
2019-01-24 18:26:22.890520 | controller |         stderr=err_data, stdout=out_data)
2019-01-24 18:26:22.890848 | controller |     tempest.lib.exceptions.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; cat /var/run/udhcpc..pid', exit status: 1, stderr:
2019-01-24 18:26:22.891027 | controller |     cat: can't open '/var/run/udhcpc..pid': No such file or directory
2019-01-24 18:26:22.891068 | controller |
2019-01-24 18:26:22.891142 | controller |     stdout:


Looks like the problem would be in the file name "udchpc..pid" -- too many extension separator dots. Maybe something in this code:

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/utils/linux/remote_client.py#n111

    def _renew_lease_udhcpc(self, fixed_ip=None):
        """Renews DHCP lease via udhcpc client. """
        file_path = '/var/run/udhcpc.'
        nic_name = self.get_nic_name_by_ip(fixed_ip)
        pid = self.exec_command('cat {path}{nic}.pid'.
                                format(path=file_path, nic=nic_name))
        pid = pid.strip()
        cmd = 'sudo /bin/kill -{sig} {pid}'.format(pid=pid, sig='USR1')
        self.exec_command(cmd)

The nic_name must be coming back empty and that's how we get
/var/run/udhcpc..pid.

** Affects: neutron
     Importance: Undecided
         Status: New

** Affects: tempest
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1813198

Title:
  TestNetworkBasicOps:test_subnet_details intermittently fails with
  "cat: can't open '/var/run/udhcpc..pid': No such file or directory"

Status in neutron:
  New
Status in tempest:
  New

Bug description:
  Seen here:

  http://logs.openstack.org/78/570078/17/check/tempest-slow/161ea32/job-
  output.txt.gz#_2019-01-24_18_26_22_886987

  2019-01-24 18:26:22.886987 | controller | {0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details [125.520364s] ... FAILED
  2019-01-24 18:26:22.887067 | controller |
  2019-01-24 18:26:22.887166 | controller | Captured traceback:
  2019-01-24 18:26:22.887251 | controller | ~~~~~~~~~~~~~~~~~~~
  2019-01-24 18:26:22.887370 | controller |     Traceback (most recent call last):
  2019-01-24 18:26:22.887545 | controller |       File "tempest/common/utils/__init__.py", line 89, in wrapper
  2019-01-24 18:26:22.887663 | controller |         return f(*func_args, **func_kwargs)
  2019-01-24 18:26:22.887868 | controller |       File "tempest/scenario/test_network_basic_ops.py", line 629, in test_subnet_details
  2019-01-24 18:26:22.887952 | controller |         renew_delay),
  2019-01-24 18:26:22.888141 | controller |       File "tempest/lib/common/utils/test_utils.py", line 107, in call_until_true
  2019-01-24 18:26:22.888237 | controller |         if func(*args, **kwargs):
  2019-01-24 18:26:22.888443 | controller |       File "tempest/scenario/test_network_basic_ops.py", line 621, in check_new_dns_server
  2019-01-24 18:26:22.888583 | controller |         dhcp_client=CONF.scenario.dhcp_client)
  2019-01-24 18:26:22.888776 | controller |       File "tempest/common/utils/linux/remote_client.py", line 140, in renew_lease
  2019-01-24 18:26:22.888957 | controller |         return getattr(self, '_renew_lease_' + dhcp_client)(fixed_ip=fixed_ip)
  2019-01-24 18:26:22.889161 | controller |       File "tempest/common/utils/linux/remote_client.py", line 116, in _renew_lease_udhcpc
  2019-01-24 18:26:22.889279 | controller |         format(path=file_path, nic=nic_name))
  2019-01-24 18:26:22.889474 | controller |       File "tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
  2019-01-24 18:26:22.889595 | controller |         return function(self, *args, **kwargs)
  2019-01-24 18:26:22.889793 | controller |       File "tempest/lib/common/utils/linux/remote_client.py", line 108, in exec_command
  2019-01-24 18:26:22.890231 | controller |         return self.ssh_client.exec_command(cmd)
  2019-01-24 18:26:22.890402 | controller |       File "tempest/lib/common/ssh.py", line 202, in exec_command
  2019-01-24 18:26:22.890520 | controller |         stderr=err_data, stdout=out_data)
  2019-01-24 18:26:22.890848 | controller |     tempest.lib.exceptions.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; cat /var/run/udhcpc..pid', exit status: 1, stderr:
  2019-01-24 18:26:22.891027 | controller |     cat: can't open '/var/run/udhcpc..pid': No such file or directory
  2019-01-24 18:26:22.891068 | controller |
  2019-01-24 18:26:22.891142 | controller |     stdout:

  
  Looks like the problem would be in the file name "udchpc..pid" -- too many extension separator dots. Maybe something in this code:

  http://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/utils/linux/remote_client.py#n111

      def _renew_lease_udhcpc(self, fixed_ip=None):
          """Renews DHCP lease via udhcpc client. """
          file_path = '/var/run/udhcpc.'
          nic_name = self.get_nic_name_by_ip(fixed_ip)
          pid = self.exec_command('cat {path}{nic}.pid'.
                                  format(path=file_path, nic=nic_name))
          pid = pid.strip()
          cmd = 'sudo /bin/kill -{sig} {pid}'.format(pid=pid, sig='USR1')
          self.exec_command(cmd)

  The nic_name must be coming back empty and that's how we get
  /var/run/udhcpc..pid.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1813198/+subscriptions
Follow ups

[Bug 1813198] Re: TestNetworkBasicOps:test_subnet_details intermittently fails with "cat: can't open '/var/run/udhcpc..pid': No such file or directory"
From: Matt Riedemann, 2019-01-29
[Bug 1813198] Re: TestNetworkBasicOps:test_subnet_details intermittently fails with "cat: can't open '/var/run/udhcpc..pid': No such file or directory"
From: Matt Riedemann, 2019-01-24