yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1809823@xxxxxxxxxxxxxxxxxx>]

You have been subscribed to a public bug:

Description
===========
on the undercloud ( pretty sure we also seen it on overcloud, i'll update when sure ) 
Without any action, we notice that neutron_api service is in "unhealthy" state and stop functioning. 
Log shows - 
2018-12-26 00:00:35.774 7 INFO oslo_service.service [-] Caught SIGHUP, stopping children
2018-12-26 00:00:36.077 40997 ERROR oslo_service.service [-] Error starting thread.: RuntimeError: A fixed interval looping call can only run one function at a time

openstack commands that needs neutron fails ( e.g openstack server list
)

Restarting the docker ( neutron_api ) resolves the problem.


Steps to reproduce
==================
Deploy. 
Wait 4 days. 

Expected result
===============
Service should remain healthy.. 

Actual result
=============
not healthy ..

Environment
===========
Rocky , container based.


Logs & Configs
==============

Logs : http://paste.openstack.org/show/738658/


More info: 
==========
Google showed this - 
https://bugs.launchpad.net/oslo.service/+bug/1547029
follow by - 
http://paste.openstack.org/show/487420/

It seems that if we'll add "eventlet.sleep(0)" in <<<HERE>>> below, it
might resolve the issue. -

    def run_service(service, done):
        """Service start wrapper.

        :param service: service to run
        :param done: event to wait on until a shutdown is triggered
        :returns: None

        """
        try:
            <<<<< HERE >>>>>>>> 
            service.start()
        except Exception:
            LOG.exception('Error starting thread.')
            raise SystemExit(1)
        else:
            done.wait()


The problem is that I didnt come up with an easy way to reproduce the issue in order to confirm it.

Any suggestions ?

** Affects: oslo.service
     Importance: High
         Status: Confirmed

** Affects: neutron
     Importance: Undecided
     Assignee: Yossi Ovadia (jabadia)
         Status: New

-- 
Neutron_api (unhealthy) after few days
https://bugs.launchpad.net/bugs/1809823
You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron.