yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87051
[Bug 1276694] Re: Openstack services should support SIGHUP signal
** Changed in: neutron
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1276694
Title:
Openstack services should support SIGHUP signal
Status in Cinder:
Invalid
Status in Glance:
Fix Released
Status in OpenStack Heat:
Fix Released
Status in OpenStack Identity (keystone):
Invalid
Status in Murano:
Confirmed
Status in neutron:
Fix Released
Status in OpenStack Compute (nova):
Fix Released
Status in oslo-incubator:
Invalid
Status in oslo.config:
Fix Released
Status in oslo.log:
Fix Released
Status in oslo.service:
Fix Released
Status in Sahara:
Fix Released
Bug description:
1)In order to more effectively manage the unlinked and open (lsof +L1)
log files descriptors w/o restarting the services, SIGHUP signal
should be accepted by every Openstack service.
That would allow, e.g. logrotate jobs to gracefully HUP services after
their log files were rotated. The only option we have for now is to
force the services restart, quite a poor option from the services
continuous accessibility PoV.
Note: according to http://en.wikipedia.org/wiki/Unix_signal
SIGHUP
... Many daemons will reload their configuration files and reopen their logfiles instead of exiting when receiving this signal.
Currently Murano and Glance are out of sync with Oslo SIGHUP support.
There is also the following issue exists for some of the services of OS projects with synced SIGHUP support:
2)
heat-api-cfn, heat-api, heat-api-cloudwatch, keystone: looks like the synced code is never being executed, thus SIGHUP is not supported for them. Here is a simple test scenario:
2.1) modify <python-path>/site-packages/<foo-service-name>/openstack/common/service.py
def _sighup_supported():
+ LOG.warning("SIGHUP is supported: {0}".format(hasattr(signal, 'SIGHUP')))
return hasattr(signal, 'SIGHUP')
2.2) restart service foo-service-name and check logs for "SIGHUP is supported", if service really supports it, the appropriate messages would be present in the logs.
2.3) issue kill -HUP <foo-service-pid> and check logs for "SIGHUP is supported" and "Caught SIGHUP", if service really supports it, the appropriate messages would be present in the logs. Besides that, the service should remain started and its main thread PID should not be changed.
e.g.
2.a) heat-engine supports HUPing:
#service openstack-heat-engine restart
<132>Apr 11 14:03:48 node-3 heat-heat.openstack.common.service WARNING: SIGHUP is supported: True
2.b)But heat-api don't know how to HUP:
#service openstack-heat-api restart
<134>Apr 11 14:06:22 node-3 heat-heat.api INFO: Starting Heat ReST API on 0.0.0.0:8004
<134>Apr 11 14:06:22 node-3 heat-eventlet.wsgi.server INFO: Starting single process server
2.c) HUPing heat-engine is OK
#pid=$(cat /var/run/heat/openstack-heat-engine.pid); kill -HUP $pid && echo $pid
16512
<134>Apr 11 14:12:15 node-3 heat-heat.openstack.common.service INFO: Caught SIGHUP, exiting
<132>Apr 11 14:12:15 node-3 heat-heat.openstack.common.service WARNING: SIGHUP is supported: True
<134>Apr 11 14:12:15 node-3 heat-heat.openstack.common.rpc.common INFO: Connected to AMQP server on ...
service openstack-heat-engine status
openstack-heat-engine (pid 16512) is running...
2.d) HUPed heat-api is dead now ;(
#kill -HUP $(cat /var/run/heat/openstack-heat-api.pid)
(no new logs)
# service openstack-heat-api status
openstack-heat-api dead but pid file exists
3)
nova-cert, nova-novncproxy, nova-objectstore, nova-consoleauth, nova-scheduler - unlike to case 2, after kill -HUP <foo-service-pid> command was issued, there would be a "Caught SIGHUP" message in the logs, BUT the associated service would have got dead anyway. Instead, the service should remain started and its main thread PID should not be changed (similar to the 2.c case).
So, looks like there are a lot of things still should be done to
ensure POSIX standards abidance in Openstack :-)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1276694/+subscriptions