sts-sponsors team mailing list archive
-
sts-sponsors team
-
Mailing list archive
-
Message #00650
[Bug 1752683] Re: race condition on rmm for module ldap (ldap cache)
Notes on the apache-deps pending sru autopkgtest failures:
For trusty, the 'puppet' autopkgtest is failing:
http://autopkgtest.ubuntu.com/packages/p/puppet/trusty/amd64
the failure for this sru is here:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-trusty/trusty/amd64/p/puppet/20180405_220028_67950@/log.gz
it appears to have been failing since 2016, however:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-trusty/trusty/amd64/p/puppet/20160713_165538@/log.gz
so it's not related to this sru and can be ignored.
For xenial, the gvfs (s390x) autopkgtest fails, but the log indicates it's a testbed setup script failure - so almost certainly some transient issue with the s390x autopkgtest setup, not anything caused by this sru.
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/s390x/g/gvfs/20180405_214901_9f312@/log.gz
Additionally, the gvfs autopkgtest has been failing for over a month, so it seems safe to ignore its failures w.r.t this sru:
http://autopkgtest.ubuntu.com/packages/g/gvfs/xenial/s390x
Also for xenial, libapache2-mod-perl2 autopkgtest is failing. This started failing with the last version of apache, however before that this autopkgtest wasn't run since 2016 - so it's very likely something else introduced this test failure, since the last xenial apache2 version was removed from -proposed and its code is not included in this upload, but the libapache2-mod-perl2 autopkgtest failure is the same. I believe this can be ignored as unrelated to this sru.
http://autopkgtest.ubuntu.com/packages/lib/libapache2-mod-perl2/xenial/amd64
For artful, resource-agent (s390x) autopkgtest fails:
http://autopkgtest.ubuntu.com/packages/c/cacti/artful/amd64
however the specific failing tests are first 'IPaddr2' which fails with what looks like a broken test (and certainly seems unrelated to apache2):
autopkgtest [20:43:48]: test IPaddr2: [-----------------------
sed: -e expression #1, char 11: unterminated `s' command
autopkgtest [20:43:48]: test IPaddr2: -----------------------]
autopkgtest [20:43:49]: test IPaddr2: - - - - - - - - - - results - - - - - - - - - -
IPaddr2 FAIL non-zero exit status 1
autopkgtest [20:43:49]: test IPaddr2: - - - - - - - - - - stderr - - - - - - - - - -
sed: -e expression #1, char 11: unterminated `s' command
and the tests 'command4' and 'command6' which are both mysql tests, not
for apache2. So appear unrelated to this.
Also for artful, the cacti autopkgtest fails:
http://autopkgtest.ubuntu.com/packages/c/cacti/artful/amd64
This failure is during a test that has something to do with apache2, so possibly is related:
autopkgtest [20:31:42]: test check-all-pages: [-----------------------
--2018-04-05 20:31:42-- http://localhost/cacti/index.php
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 134 [text/html]
Saving to: ‘/tmp/tmp.JiG38g3XKE’
0K 100%
24.8M=0s
2018-04-05 20:31:43 (24.8 MB/s) - ‘/tmp/tmp.JiG38g3XKE’ saved [134/134]
--2018-04-05 20:31:43-- http://localhost/cacti/index.php
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 134 [text/html]
Saving to: ‘/tmp/tmp.ZWxUDKERUh’
0K 100%
27.8M=0s
2018-04-05 20:31:44 (27.8 MB/s) - ‘/tmp/tmp.ZWxUDKERUh’ saved [134/134]
--2018-04-05 20:31:44-- http://localhost/cacti/index.php
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 134 [text/html]
Saving to: ‘localhost/cacti/index.php’
0K 100%
26.2M=0s
2018-04-05 20:31:45 (26.2 MB/s) - ‘localhost/cacti/index.php’ saved
[134/134]
FINISHED --2018-04-05 20:31:45--
Total wall clock time: 0.8s
Downloaded: 1 files, 134 in 0s (26.2 MB/s)
localhost/cacti/graphs.php not found
Copying /var/log/cacti/cacti.log to artifacts
Copying /var/log/apache2/access.log to artifacts
Copying /var/log/apache2/error.log to artifacts
autopkgtest [20:31:45]: test check-all-pages: -----------------------]
autopkgtest [20:31:45]: test check-all-pages: - - - - - - - - - - results - - - - - - - - - -
check-all-pages FAIL non-zero exit status 179
I'll look a bit more that this artful cacti autopkgtest to determine if it is caused by this sru or not. However, as @inaddy's last comment indicates the apache2 package itself is verified and i'll mark it as such for each release.
** Tags added: verification-done-artful verification-done-trusty
verification-done-xenial
--
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1752683
Title:
race condition on rmm for module ldap (ldap cache)
Status in Apache2 Web Server:
Fix Released
Status in apache2 package in Ubuntu:
Fix Released
Status in apache2 source package in Trusty:
Fix Committed
Status in apache2 source package in Xenial:
Fix Committed
Status in apache2 source package in Artful:
Fix Committed
Status in apache2 source package in Bionic:
Fix Released
Bug description:
[Impact]
* Apache users using ldap module might face this if using multiple
threads and shared memory activated for apr memory allocator (default
in Ubuntu).
[Test Case]
* Configure apache to use ldap module, for authentication e.g., and wait for the race condition to happen.
* Analysis made out of a dump from a production environment.
* Bug has been reported multiple times upstream in the past 10 years.
[Regression Potential]
* ldap module has broken locking mechanism when using apr mem mgmt.
* ldap would continue to have broken locking mechanism.
* race conditions could still exist.
* could could brake ldap module.
* patch is upstreamed in next version to be released.
[Other Info]
ORIGINAL CASE DESCRIPTION:
Problem summary:
apr_rmm_init acts as a relocatable memory management initialization
it is used in: mod_auth_digest and util_ldap_cache
From the dump was brought to my knowledge, in the following sequence:
- util_ldap_compare_node_copy()
- util_ald_strdup()
- apr_rmm_calloc()
- find_block_of_size()
Had a "cache->rmm_addr" with no lock at "find_block_of_size()"
cache->rmm_addr->lock { type = apr_anylock_none }
And an invalid "next" offset (out of rmm->base->firstfree).
This rmm_addr was initialized with NULL as a locking mechanism:
From apr-utils:
apr_rmm_init()
if (!lock) { <-- 2nd argument to apr_rmm_init()
nulllock.type = apr_anylock_none; <--- found in the dump
nulllock.lock.pm = NULL;
lock = &nulllock;
}
From apache:
# mod_auth_digest
sts = apr_rmm_init(&client_rmm,
NULL, /* no lock, we'll do the locking ourselves */
apr_shm_baseaddr_get(client_shm),
shmem_size, ctx);
# util_ldap_cache
result = apr_rmm_init(&st->cache_rmm, NULL,
apr_shm_baseaddr_get(st->cache_shm), size,
st->pool);
It appears that the ldap module chose to use "rmm" for memory allocation, using
the shared memory approach, but without explicitly definiting a lock to it.
Without it, its up to the caller to guarantee that there are locks for rmm
synchronization (just like mod_auth_digest does, using global mutexes).
Because of that, there was a race condition in "find_block_of_size" and a call
touching "rmm->base->firstfree", possibly "move_block()", in a multi-threaded
apache environment, since there were no lock guarantees inside rmm logic (lock
was "apr_anylock_none" and the locking calls don't do anything).
In find_block_of_size:
apr_rmm_off_t next = rmm->base->firstfree;
We have:
rmm->base->firstfree
Decimal:356400
Hex:0x57030
But "next" turned into:
Name : next
Decimal:8320808657351632189
Hex:0x737973636970653d
Causing:
struct rmm_block_t *blk = (rmm_block_t*)((char*)rmm->base +
next);
if (blk->size == size)
To segfault.
Upstream bugs:
https://bz.apache.org/bugzilla/show_bug.cgi?id=58483
https://bz.apache.org/bugzilla/show_bug.cgi?id=60296
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=814980#15
To manage notifications about this bug go to:
https://bugs.launchpad.net/apache2/+bug/1752683/+subscriptions
