← Back to team overview

enterprise-support team mailing list archive

  1. enterprise-support team
  2. Mailing list archive
  3. Message #05735

[Bug 1628535] [NEW] AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - Can't access maas server URL

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

We stopped being able to access maas through its web address and this
caused a crash:

  File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line 137, in get_tagged_nodes
    all_nodes_list = self.nodes_list(user)
  File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line 378, in nodes_list
    return json.loads(self.cmd(command))
  File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line 281, in cmd
    content = self.client.get(command).read()
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 237, in get
    url, method="GET", headers=headers)
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 116, in dispatch_query
    res = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>

On the apache2 error log, I saw:

ue Sep 27 06:25:12.159028 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Tue Sep 27 06:25:12.159257 2016] [core:notice] [pid 1895:tid 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
[Wed Sep 28 06:25:09.318844 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00493: SIGUSR1 received.  Doing graceful restart
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
[Wed Sep 28 06:25:12.038905 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Wed Sep 28 06:25:12.038944 2016] [core:notice] [pid 1895:tid 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
[Wed Sep 28 06:25:14.040425 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:15.041505 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:16.042554 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:17.043605 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:18.045062 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:19.046130 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:20.047475 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:21.048295 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
... <repeats non-stop>
ull, not at MaxRequestWorkers
[Wed Sep 28 12:53:25.662213 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:26.663278 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:27.664325 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:28.665413 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:29.666492 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers


The errors stopped when I restarted the apache2 service with sudo service apache2 restart.

I couldn't tell where to find the core. There was only this file from a week ago:
$ ls -l _usr_sbin_apache2.0.crash
-rw-r----- 1 root root 2526598 Sep 21 22:33 _usr_sbin_apache2.0.crash

I tried to rerun it again and timestamp changed but core info did not
appear to change.

I have collected access and error logs of the past couple of days as
well as other logs including maas logs. Tarball is attached to this bug
and it contains the 2 crash files:

$ ls -l _usr_sbin_apache2.0.crash.*
-rw-r----- 1 ubuntu ubuntu 2698755 Sep 28 13:42 _usr_sbin_apache2.0.crash.new
-rw-r----- 1 ubuntu ubuntu 2526598 Sep 21 22:33 _usr_sbin_apache2.0.crash.old


These are the package versions:

ii  maas                 2.0.0+bzr5189-0 all             "Metal as a Service" is a physical cloud and I
ii  apache2                               2.4.18-2ubuntu3.1                   amd64        Apache HTTP Server
ii  apache2-bin                           2.4.18-2ubuntu3.1                   amd64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                          2.4.18-2ubuntu3.1                   all          Apache HTTP Server (common files)
ii  apache2-utils                         2.4.18-2ubuntu3.1                   amd64        Apache HTTP Server (utility programs for web servers)
ii  init                                  1.29ubuntu2                         amd64        System-V-like init utilities - metapackage


$ lsb_release -rd
Description:	Ubuntu 16.04.1 LTS
Release:	16.04

** Affects: maas
     Importance: Undecided
         Status: New

** Affects: apache2 (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: oil

** Attachment added: "apache2.logs.tar.gz"
   https://bugs.launchpad.net/bugs/1628535/+attachment/4750157/+files/apache2.logs.tar.gz

** Summary changed:

- AH00493: SIGUSR1 received.  Doing graceful restart - Apache2 crashes
+ AH00493: SIGUSR1 received.  Doing graceful restart - Apache2 crashes - Can't access maas server URL

** Also affects: maas
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Server/Client Support Team, which is subscribed to apache2 in Ubuntu.
Matching subscriptions: Ubuntu Server/Client Support Team
https://bugs.launchpad.net/bugs/1628535

Title:
  AH00493: SIGUSR1 received.  Doing graceful restart - Apache2 crashes -
  Can't access maas server URL

To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1628535/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next