I'm opening this bug up more publicly and including IBM.

We plan to deliver a fix for this bug in rhel-7.3.

This bug was created to track the backport of the patches to fix upstream sourceware bug 19048.

The fixes to be backported are as follows:
---
commit 1bd5483e104c8bde6e61dc5e3f8a848bc861872d
Author: Florian Weimer <fweimer>
Date:   Tue Dec 29 20:32:35 2015 +0100

    malloc: Test various special cases related to allocation failures
    
    This test case exercises unusual code paths in allocation functions,
    related to allocation failures.  Specifically, the test can reveal
    the following bugs:
    
    (a) calloc returns non-zero memory on fallback to sysmalloc.
    (b) calloc can self-deadlock because it fails to release
        the arena lock on certain allocation failures.
    (c) pvalloc can dereference a NULL arena pointer.
    
    (a) and (b) appear specific to a faulty downstream backport.
    (c) was fixed as part of commit 10ad46bc6526edc5c7afcc57112da96917ff3629.
    
    The test for (a) was inspired by a reproducer supplied by Jeff Layton.
---
commit 7962541a32eff5597bc4207e781cfac8d1bb0d87
Author: Florian Weimer <fweimer>
Date:   Wed Dec 23 17:23:33 2015 +0100

    malloc: Update comment for list_lock
---
commit 90c400bd4904b0240a148f0b357a5cbc36179239
Author: Florian Weimer <fweimer>
Date:   Mon Dec 21 16:42:46 2015 +0100

    malloc: Fix list_lock/arena lock deadlock [BZ #19182]
    
        * malloc/arena.c (list_lock): Document lock ordering requirements.
        (free_list_lock): New lock.
        (ptmalloc_lock_all): Comment on free_list_lock.
        (ptmalloc_unlock_all2): Reinitialize free_list_lock.
        (detach_arena): Update comment.  free_list_lock is now needed.
        (_int_new_arena): Use free_list_lock around detach_arena call.
        Acquire arena lock after list_lock.  Add comment, including FIXME
        about incorrect synchronization.
        (get_free_list): Switch to free_list_lock.
        (reused_arena): Acquire free_list_lock around detach_arena call
        and attached threads counter update.  Add two FIXMEs about
        incorrect synchronization.
        (arena_thread_freeres): Switch to free_list_lock.
        * malloc/malloc.c (struct malloc_state): Update comments to
        mention free_list_lock.
---
commit 3da825ce483903e3a881a016113b3e59fd4041de
Author: Florian Weimer <fweimer>
Date:   Wed Dec 16 12:39:48 2015 +0100

    malloc: Fix attached thread reference count handling [BZ #19243]
    
    reused_arena can increase the attached thread count of arenas on the
    free list.  This means that the assertion that the reference count is
    zero is incorrect.  In this case, the reference count initialization
    is incorrect as well and could cause arenas to be put on the free
    list too early (while they still have attached threads).
    
        * malloc/arena.c (get_free_list): Remove assert and adjust
        reference count handling.  Add comment about reused_arena
        interaction.
        (reused_arena): Add comments abount get_free_list interaction.
        * malloc/tst-malloc-thread-exit.c: New file.
        * malloc/Makefile (tests): Add tst-malloc-thread-exit.
        (tst-malloc-thread-exit): Link against libpthread.
---
commit 400e12265d99964f8445bb6d717321eb73152cc5
Author: Florian Weimer <fweimer>
Date:   Tue Nov 24 16:37:15 2015 +0100

    Replace MUTEX_INITIALIZER with _LIBC_LOCK_INITIALIZER in generic code
    
        * sysdeps/mach/hurd/libc-lock.h (_LIBC_LOCK_INITIALIZER): Define.
        (__libc_lock_define_initialized): Use it.
        * sysdeps/nptl/libc-lockP.h (_LIBC_LOCK_INITIALIZER): Define.
        * malloc/arena.c (list_lock): Use _LIBC_LOCK_INITIALIZER.
        * malloc/malloc.c (main_arena): Likewise.
        * sysdeps/generic/malloc-machine.h (MUTEX_INITIALIZER): Remove.
        * sysdeps/nptl/malloc-machine.h (MUTEX_INITIALIZER): Remove.
---
commit a62719ba90e2fa1728890ae7dc8df9e32a622e7b
Author: Florian Weimer <fweimer>
Date:   Wed Oct 28 19:32:46 2015 +0100

    malloc: Prevent arena free_list from turning cyclic [BZ #19048]
    
        [BZ# 19048]
        * malloc/malloc.c (struct malloc_state): Update comment.  Add
        attached_threads member.
        (main_arena): Initialize attached_threads.
        * malloc/arena.c (list_lock): Update comment.
        (ptmalloc_lock_all, ptmalloc_unlock_all): Likewise.
        (ptmalloc_unlock_all2): Reinitialize arena reference counts.
        (deattach_arena): New function.
        (_int_new_arena): Initialize arena reference count and deattach
        replaced arena.
        (get_free_list, reused_arena): Update reference count and deattach
        replaced arena.
        (arena_thread_freeres): Update arena reference count and only put
        unreferenced arenas on the free list.

---
commit 6782806d8f6664d87d17bb30f8ce4e0c7c931e17
Author: Florian Weimer <fweimer>
Date:   Sat Oct 17 12:06:48 2015 +0200

    malloc: Rewrite with explicit TLS access using __thread
---

*** Bug 1297423 has been marked as a duplicate of this bug. ***

Note: I will make this bug public soon so that others can comment if they feel so inclined.

*** Bug 1330623 has been marked as a duplicate of this bug. ***

In our case (vertica database server) once the arena freelist goes circular, 
it affects the application moving forward independent of concurrency (as the application becomes sick).

This causes significant performance degradation in high concurrency situations.

We tested the efficacy of the patch posted on sourceware internally by (re)building glibc and the patch was stable and improved performance under concurrent load.
A couple customers have tested it not just for stability (it is) but also for performance (it helps).

What is the target release date for 7.3, thus glibc-2.17-131.el7 will be available?

What are the prospects for publishing glibc-2.17-131.el7 as an update prior to 7.3?

What are the prospects of backporting this to the glibc-2.12 stream for RHEL6?

(In reply to Sumeet Keswani from comment #26)
> In our case (vertica database server) once the arena freelist goes circular, 
> it affects the application moving forward independent of concurrency (as the
> application becomes sick).
> 
> This causes significant performance degradation in high concurrency
> situations.
> 
> We tested the efficacy of the patch posted on sourceware internally by
> (re)building glibc and the patch was stable and improved performance under
> concurrent load.
> A couple customers have tested it not just for stability (it is) but also
> for performance (it helps).

You should see a similar performance improvement on Red Hat Enterprise Linux 6.8 Beta, where we fixed this issue as bug 1264189 (currently private).

is this fix included in glibc-2.12-1.192 ?

does not show up in this advisory? (RHBA-2016:0834-1)
https://rhn.redhat.com/errata/RHBA-2016-0834.html

How can users on RHEL 6.X get this fix?

can i get access to BZ 1264189

Hello,
I have requested HPE access to BZ 1264189.
Please note this BZ was closed with errata:
https://rhn.redhat.com/errata/RHBA-2016-0834.html

Thank You
Joe Kachuck

Its not listed in the errata (RHBA-2016:0834-1) hence i was not certain how to point users to that for a fix.

(In reply to Sumeet Keswani from comment #33)
> Its not listed in the errata (RHBA-2016:0834-1) hence i was not certain how
> to point users to that for a fix.

This bug was fixed with RHBA-2016:0834-1 for Red Hat Enterprise Linux 6.8, under bug 1264189.  That bug largely consists of private comments and is not very illuminating to external parties as a result.

The upstream bug (https://sourceware.org/bugzilla/show_bug.cgi?id=19048) had a test.c and a check-free_list.sh script attached to it. Running the script against a running instance of the test exposes the bug on ppc64 and s390x (not sure why not on other architectures) even on the patched glibc. So this needs to be looked at again. Florian's doing that right now.

Arjun,  is this verified on both architectures, ppc64 and s390x ?

(In reply to Georg Markgraf from comment #41)
> Arjun,  is this verified on both architectures, ppc64 and s390x ?

Georg, yes the verification happened on all the rhel-7 supported architectures, incl ppc64 and s390x.

------- Comment From MSTRUBEL.com 2016-09-12 13:27 EDT-------
I ran the arena tests (from glibc bugzilla) and did straces with and without TRIM option enabled using old & new glibc version. It seems the patches are effective.

Thank you very much for taking care of this issue,

I think, that this BZ can be closed now.

best regards
Matthias Strubel

(In reply to IBM Bug Proxy from comment #43)
> ------- Comment From MSTRUBEL.com 2016-09-12 13:27 EDT-------
> I ran the arena tests (from glibc bugzilla) and did straces with and without
> TRIM option enabled using old & new glibc version. It seems the patches are
> effective.
> 
> Thank you very much for taking care of this issue,

Thank you for the additional testing.

> I think, that this BZ can be closed now.

This bug will be closed automatically once we ship the update as part of Red Hat Enterprise Linux 7.3.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2573.html