fclose will call free, invoking its hook, then fprintf which would indirectly
try to allocate a buffer, and this can cause malloc to be used (thus its hook
to be invoked) if libio uses malloc instead of mmap; given any malloc/free hook
locks the internal lock, this leads to a deadlock.
To prevent this hook roundtrip at muntrace, first unset MALLSTREAM and the
hooks, and only after that close the trace file.
Using madvise with MADV_DONTNEED to release memory back to the kernel
is not sufficient to change the commit charge accounted against the
process on Linux. It is OK however, when overcommit is enabled or is
heuristic. However, when overcommit is restricted to a percentage of
memory setting the contents of /proc/sys/vm/overcommit_memory as 2, it
makes a difference since memory requests will fail. Hence, we do what
we do with secure exec binaries, which is to call mmap on the region
to be dropped with MAP_FIXED. This internally unmaps the pages in
question and reduces the amount of memory accounted against the
process.
[BZ #1349]
malloc_usable_size returns the usable size in an allocated chunk,
which may be >= the requested size. In the case of MALLOC_CHECK_ being
exported to > 0 however, only the requested size is usable, since a
magic value is written at the end of the request size to trap writes
beyond request bounds. Hence, when MALLOC_CHECK_ is exported to > 0,
malloc_usable_size() should return the request size.
* malloc.c/arena.c (reused_arena): New parameter, avoid_arena.
When avoid_arena is set, don't retry in the that arena. Pick the
next one, whatever it might be.
(arena_get2): New parameter avoid_arena, pass through to reused_arena.
(arena_lock): Pass in new parameter to arena_get2.
* malloc/malloc.c (__libc_memalign): Pass in new parameter to
arena_get2.
(__libc_malloc): Unify retrying after main arena failure with
__libc_memalign version.
(__libc_valloc, __libc_pvalloc, __libc_calloc): Likewise.
tst-obprintf failed with GCC 4.7.
It turned out that this is the fault of GCC optimizing away the
following from malloc/mcheck.c:
/* We call malloc() once here to ensure it is initialized. */
void *p = malloc (0);
free (p);
gcc sees the malloc(0);free pair and removes it completely.
And now malloc is not properly initialized and we screw up if both
mcheck is used (via tst-obprintf) and MALLOC_CHECK_ is set (as it is in my
environment).
