Commit Graph

114 Commits

Author SHA1 Message Date
Joseph Myers
d614a75396 Update copyright dates with scripts/update-copyrights. 2020-01-01 00:14:33 +00:00
Paul Eggert
5a82c74822 Prefer https to http for gnu.org and fsf.org URLs
Also, change sources.redhat.com to sourceware.org.
This patch was automatically generated by running the following shell
script, which uses GNU sed, and which avoids modifying files imported
from upstream:

sed -ri '
  s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g
  s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g
' \
  $(find $(git ls-files) -prune -type f \
      ! -name '*.po' \
      ! -name 'ChangeLog*' \
      ! -path COPYING ! -path COPYING.LIB \
      ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \
      ! -path manual/texinfo.tex ! -path scripts/config.guess \
      ! -path scripts/config.sub ! -path scripts/install-sh \
      ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \
      ! -path INSTALL ! -path  locale/programs/charmap-kw.h \
      ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \
      ! '(' -name configure \
            -execdir test -f configure.ac -o -f configure.in ';' ')' \
      ! '(' -name preconfigure \
            -execdir test -f preconfigure.ac ';' ')' \
      -print)

and then by running 'make dist-prepare' to regenerate files built
from the altered files, and then executing the following to cleanup:

  chmod a+x sysdeps/unix/sysv/linux/riscv/configure
  # Omit irrelevant whitespace and comment-only changes,
  # perhaps from a slightly-different Autoconf version.
  git checkout -f \
    sysdeps/csky/configure \
    sysdeps/hppa/configure \
    sysdeps/riscv/configure \
    sysdeps/unix/sysv/linux/csky/configure
  # Omit changes that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines
  git checkout -f \
    sysdeps/powerpc/powerpc64/ppc-mcount.S \
    sysdeps/unix/sysv/linux/s390/s390-64/syscall.S
  # Omit change that caused a pre-commit check to fail like this:
  # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline
  git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 02:43:31 -07:00
DJ Delorie
c48d92b430 Add glibc.malloc.mxfast tunable
* elf/dl-tunables.list: Add glibc.malloc.mxfast.
* manual/tunables.texi: Document it.
* malloc/malloc.c (do_set_mxfast): New.
(__libc_mallopt): Call it.
* malloc/arena.c: Add mxfast tunable.
* malloc/tst-mxfast.c: New.
* malloc/Makefile: Add it.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2019-08-09 14:04:03 -04:00
H.J. Lu
2d398aa272 Remove do_set_mallopt_check prototype
Remove do_set_mallopt_check prototype since it is unused.

	* malloc/arena.c (do_set_mallopt_check): Removed.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: DJ Delorie <dj@redhat.com>
2019-04-23 13:12:25 -07:00
Joseph Myers
04277e02d7 Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2019-01-01 00:11:28 +00:00
Florian Weimer
1ecba1fafc malloc: Convert the unlink macro to the unlink_chunk function
This commit is in preparation of turning the macro into a proper
function.  The output arguments of the macro were in fact unused.

Also clean up uses of __builtin_expect.
2018-11-12 14:35:06 +01:00
Florian Weimer
124e025864 Run thread shutdown functions in an explicit order
This removes the __libc_thread_subfreeres hook in favor of explict
calls.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
2018-06-26 15:27:12 +02:00
Joseph Myers
688903eb3e Update copyright dates with scripts/update-copyrights.
* All files with FSF copyright notices: Update copyright dates
	using scripts/update-copyrights.
	* locale/programs/charmap-kw.h: Regenerated.
	* locale/programs/locfile-kw.h: Likewise.
2018-01-01 00:32:25 +00:00
Florian Weimer
0a947e061d malloc: Call tcache destructor in arena_thread_freeres
It does not make sense to register separate cleanup functions for arena
and tcache since they're always going to be called together.  Call the
tcache cleanup function from within arena_thread_freeres since it at
least makes the order of those cleanups clear in the code.

Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
2017-11-23 14:47:31 +01:00
Wilco Dijkstra
3381be5cde Improve malloc initialization sequence
The current malloc initialization is quite convoluted. Instead of
sometimes calling malloc_consolidate from ptmalloc_init, call
malloc_init_state early so that the main_arena is always initialized.
The special initialization can now be removed from malloc_consolidate.
This also fixes BZ #22159.

Check all calls to malloc_consolidate and remove calls that are
redundant initialization after ptmalloc_init, like in int_mallinfo
and __libc_mallopt (but keep the latter as consolidation is required for
set_max_fast).  Update comments to improve clarity.

Remove impossible initialization check from _int_malloc, fix assert
in do_check_malloc_state to ensure arena->top != 0.  Fix the obvious bugs
in do_check_free_chunk and do_check_remalloced_chunk to enable single
threaded malloc debugging (do_check_malloc_state is not thread safe!).

	[BZ #22159]
	* malloc/arena.c (ptmalloc_init): Call malloc_init_state.
	* malloc/malloc.c (do_check_free_chunk): Fix build bug.
	(do_check_remalloced_chunk): Fix build bug.
	(do_check_malloc_state): Add assert that checks arena->top.
	(malloc_consolidate): Remove initialization.
	(int_mallinfo): Remove call to malloc_consolidate.
	(__libc_mallopt): Clarify why malloc_consolidate is needed.
2017-10-17 18:55:16 +01:00
Florian Weimer
0c71122c0c malloc: Remove the internal_function attribute 2017-08-31 15:59:06 +02:00
Florian Weimer
a9da0bb266 malloc: Remove corrupt arena flag
This is no longer needed because we now abort immediately
once heap corruption is detected.
2017-08-30 20:08:34 +02:00
Florian Weimer
ac3ed168d0 malloc: Remove check_action variable [BZ #21754]
Clean up calls to malloc_printerr and trim its argument list.

This also removes a few bits of work done before calling
malloc_printerr (such as unlocking operations).

The tunable/environment variable still enables the lightweight
additional malloc checking, but mallopt (M_CHECK_ACTION)
no longer has any effect.
2017-08-30 20:08:34 +02:00
DJ Delorie
d5c3fafc43 Add per-thread cache to malloc
* config.make.in: Enable experimental malloc option.
* configure.ac: Likewise.
* configure: Regenerate.
* manual/install.texi: Document it.
* INSTALL: Regenerate.
* malloc/Makefile: Likewise.
* malloc/malloc.c: Add per-thread cache (tcache).
(tcache_put): New.
(tcache_get): New.
(tcache_thread_freeres): New.
(tcache_init): New.
(__libc_malloc): Use cached chunks if available.
(__libc_free): Initialize tcache if needed.
(__libc_realloc): Likewise.
(__libc_calloc): Likewise.
(_int_malloc): Prefill tcache when appropriate.
(_int_free): Likewise.
(do_set_tcache_max): New.
(do_set_tcache_count): New.
(do_set_tcache_unsorted_limit): New.
* manual/probes.texi: Document new probes.
* malloc/arena.c: Add new tcache tunables.
* elf/dl-tunables.list: Likewise.
* manual/tunables.texi: Document them.
* NEWS: Mention the per-thread cache.
2017-07-06 13:37:30 -04:00
Siddhesh Poyarekar
44330b6d32 tunables: Clean up hooks to get and set tunables
The TUNABLE_SET_VALUE and family of macros (and my later attempt to
add a TUNABLE_GET) never quite went together very well because the
overall interface was not clearly defined.  This patch is an attempt
to do just that.

This patch consolidates the API to two simple sets of macros,
TUNABLE_GET* and TUNABLE_SET*.  If TUNABLE_NAMESPACE is defined,
TUNABLE_GET takes just the tunable name, type and a (optionally NULL)
callback function to get the value of the tunable.  The callback
function, if non-NULL, is called if the tunable was externally set
(i.e. via GLIBC_TUNABLES or any future mechanism).  For example:

    val = TUNABLE_GET (check, int32_t, check_callback)

returns the value of the glibc.malloc.check tunable (assuming
TUNABLE_NAMESPACE is set to malloc) as an int32_t into VAL after
calling check_callback.

Likewise, TUNABLE_SET can be used to set the value of the tunable,
although this is currently possible only in the dynamic linker before
it relocates itself.  For example:

  TUNABLE_SET (check, int32_t, 2)

will set glibc.malloc.check to 2.  Of course, this is not possible
since we set (or read) glibc.malloc.check long after it is relocated.

To access or set a tunable outside of TUNABLE_NAMESPACE, use the
TUNABLE_GET_FULL and TUNABLE_SET_FULL macros, which have the following
prototype:

  TUNABLE_GET_FULL (glibc, tune, hwcap_mask, uint64_t, NULL)
  TUNABLE_SET_FULL (glibc, tune, hwcap_mask, uint64_t, 0xffff)

In future the tunable list may get split into mutable and immutable
tunables where mutable tunables can be modified by the library and
userspace after relocation as well and TUNABLE_SET will be more useful
than it currently is.  However whenever we actually do that split, we
will have to ensure that the mutable tunables are protected with
locks.

	* elf/Versions (__tunable_set_val): Rename to __tunable_get_val.
	* elf/dl-tunables.c: Likewise.
	(do_tunable_update_val): New function.
	(__tunable_set_val): New function.
	(__tunable_get_val): Call CB only if the tunable was externally
	initialized.
	(tunables_strtoul): Replace strval with initialized.
	* elf/dl-tunables.h (strval): Replace with a bool initialized.
	(TUNABLE_ENUM_NAME, TUNABLE_ENUM_NAME1): Adjust names to
	prevent collision.
	(__tunable_set_val): New function.
	(TUNABLE_GET, TUNABLE_GET_FULL): New macros.
	(TUNABLE_SET, TUNABLE_SET_FULL): Likewise.
	(TUNABLE_SET_VAL): Remove.
	(TUNABLE_SET_VAL_WITH_CALLBACK): Likewise.
	* README.tunables: Document the new macros.
	* malloc/arena.c (ptmalloc_init): Adjust.
2017-06-07 11:11:36 +05:30
Siddhesh Poyarekar
8cbc826c37 Fix getting tunable values on big-endian (BZ #21109)
The code to set value passed a tunable_val_t, which when cast to
int32_t on big-endian gives the wrong value.  Instead, use
tunable_val_t.numval instead, which can then be safely cast into
int32_t.
2017-02-08 14:17:17 +05:30
Joseph Myers
bfff8b1bec Update copyright dates with scripts/update-copyrights. 2017-01-01 00:14:16 +00:00
Siddhesh Poyarekar
67e58f3941 Add framework for tunables
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users.  tunables/README has
instructions for glibc developers to add new tunables.

Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script.  This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user.  It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.

	* manual/install.texi: Add --enable-tunables option.
	* INSTALL: Regenerate.
	* README.tunables: New file.
	* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
	(before-compile): Generate dl-tunable-list.h early.
	* config.h.in: Add HAVE_TUNABLES.
	* config.make.in: Add have-tunables.
	* configure.ac: Add --enable-tunables option.
	* configure: Regenerate.
	* csu/init-first.c (__libc_init_first): Move
	__libc_init_secure earlier...
	* csu/init-first.c (LIBC_START_MAIN):... to here.
	Include dl-tunables.h, libc-internal.h.
	(LIBC_START_MAIN) [!SHARED]: Initialize tunables for static
	binaries.
	* elf/Makefile (dl-routines): Add dl-tunables.
	* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
	namespace.
	* elf/dl-support (_dl_nondynamic_init): Unset MALLOC_CHECK_
	only when !HAVE_TUNABLES.
	* elf/rtld.c (process_envvars): Likewise.
	* elf/dl-sysdep.c [HAVE_TUNABLES]: Include dl-tunables.h
	(_dl_sysdep_start): Call __tunables_init.
	* elf/dl-tunable-types.h: New file.
	* elf/dl-tunables.c: New file.
	* elf/dl-tunables.h: New file.
	* elf/dl-tunables.list: New file.
	* malloc/tst-malloc-usable-static.c: New test case.
	* malloc/Makefile (tests-static): Add it.
	* malloc/arena.c [HAVE_TUNABLES]: Include dl-tunables.h.
	Define TUNABLE_NAMESPACE.
	(DL_TUNABLE_CALLBACK (set_mallopt_check)): New function.
	(DL_TUNABLE_CALLBACK_FNDECL): New macro.  Use it to define
	callback functions.
	(ptmalloc_init): Set tunable values.
	* scripts/gen-tunables.awk: New file.
	* sysdeps/mach/hurd/dl-sysdep.c: Include dl-tunables.h.
	(_dl_sysdep_start): Call __tunables_init.
2016-12-31 23:49:24 +05:30
Florian Weimer
e9c4fe93b3 malloc: Use accessors for chunk metadata access
This change allows us to change the encoding of these struct members
in a centralized fashion.
2016-10-28 16:45:45 +02:00
Florian Weimer
cbb47fa1c6 malloc: Manual part of conversion to __libc_lock
This removes the old mutex_t-related definitions from malloc-machine.h,
too.
2016-09-21 16:28:08 +02:00
Florian Weimer
4bf5f2224b malloc: Automated part of conversion to __libc_lock 2016-09-06 12:49:54 +02:00
Florian Weimer
f88aab5d50 malloc: Preserve arena free list/thread count invariant [BZ #20370]
It is necessary to preserve the invariant that if an arena is
on the free list, it has thread attach count zero.  Otherwise,
when arena_thread_freeres sees the zero attach count, it will
add it, and without the invariant, an arena could get pushed
to the list twice, resulting in a cycle.

One possible execution trace looks like this:

Thread 1 examines free list and observes it as empty.
Thread 2 exits and adds its arena to the free list,
  with attached_threads == 0).
Thread 1 selects this arena in reused_arena (not from the free list).
Thread 1 increments attached_threads and attaches itself.
  (The arena remains on the free list.)
Thread 1 exits, decrements attached_threads,
  and adds the arena to the free list.

The final step creates a cycle in the usual way (by overwriting the
next_free member with the former list head, while there is another
list item pointing to the arena structure).

tst-malloc-thread-exit exhibits this issue, but it was only visible
with a debugger because the incorrect fix in bug 19243 removed
the assert from get_free_list.
2016-08-02 12:24:50 +02:00
Florian Weimer
a3b473373e malloc: Avoid premature fallback to mmap [BZ #20284]
Before this change, the while loop in reused_arena which avoids
returning a corrupt arena would never execute its body if the selected
arena were not corrupt.  As a result, result == begin after the loop,
and the function returns NULL, triggering fallback to mmap.
2016-06-21 21:29:21 +02:00
Florian Weimer
92e1ab0eb5 Revert __malloc_initialize_hook symbol poisoning
It turns out the Emacs-internal malloc implementation uses
__malloc_* symbols.  If glibc poisons them in <stdc-pre.h>,
Emacs will no longer compile.
2016-06-20 11:11:29 +02:00
Florian Weimer
2ba3cfa160 malloc: Remove __malloc_initialize_hook from the API [BZ #19564]
__malloc_initialize_hook is interposed by application code, so
the usual approach to define a compatibility symbol does not work.
This commit adds a new mechanism based on #pragma GCC poison in
<stdc-predef.h>.
2016-06-10 10:46:05 +02:00
Florian Weimer
186fe877f3 malloc: Add missing internal_function attributes on function definitions
Fixes build on i386 after commit 29d794863c.
2016-04-14 12:54:22 +02:00
Florian Weimer
8a727af925 malloc: Remove malloc hooks from fork handler
The fork handler now runs so late that there is no risk anymore that
other fork handlers in the same thread use malloc, so it is no
longer necessary to install malloc hooks which made a subset
of malloc functionality available to the thread that called fork.
2016-04-14 09:18:30 +02:00
Florian Weimer
29d794863c malloc: Run fork handler as late as possible [BZ #19431]
Previously, a thread M invoking fork would acquire locks in this order:

  (M1) malloc arena locks (in the registered fork handler)
  (M2) libio list lock

A thread F invoking flush (NULL) would acquire locks in this order:

  (F1) libio list lock
  (F2) individual _IO_FILE locks

A thread G running getdelim would use this order:

  (G1) _IO_FILE lock
  (G2) malloc arena lock

After executing (M1), (F1), (G1), none of the threads can make progress.

This commit changes the fork lock order to:

  (M'1) libio list lock
  (M'2) malloc arena locks

It explicitly encodes the lock order in the implementations of fork,
and does not rely on the registration order, thus avoiding the deadlock.
2016-04-14 09:17:02 +02:00
Florian Weimer
59eda029a8 malloc: Remove NO_THREADS
No functional change.  It was not possible to build without
threading support before.
2016-02-19 17:07:45 +01:00
Florian Weimer
00d4e2ea35 malloc: Remove arena_mem variable
The computed value is never used.  The accesses were data races.
2016-02-19 17:06:33 +01:00
Joseph Myers
f7a9f785e5 Update copyright dates with scripts/update-copyrights. 2016-01-04 16:05:18 +00:00
Florian Weimer
7962541a32 malloc: Update comment for list_lock 2015-12-23 17:23:33 +01:00
Florian Weimer
90c400bd49 malloc: Fix list_lock/arena lock deadlock [BZ #19182]
* malloc/arena.c (list_lock): Document lock ordering requirements.
	(free_list_lock): New lock.
	(ptmalloc_lock_all): Comment on free_list_lock.
	(ptmalloc_unlock_all2): Reinitialize free_list_lock.
	(detach_arena): Update comment.  free_list_lock is now needed.
	(_int_new_arena): Use free_list_lock around detach_arena call.
	Acquire arena lock after list_lock.  Add comment, including FIXME
	about incorrect synchronization.
	(get_free_list): Switch to free_list_lock.
	(reused_arena): Acquire free_list_lock around detach_arena call
	and attached threads counter update.  Add two FIXMEs about
	incorrect synchronization.
	(arena_thread_freeres): Switch to free_list_lock.
	* malloc/malloc.c (struct malloc_state): Update comments to
	mention free_list_lock.
2015-12-21 16:42:46 +01:00
Florian Weimer
3da825ce48 malloc: Fix attached thread reference count handling [BZ #19243]
reused_arena can increase the attached thread count of arenas on the
free list.  This means that the assertion that the reference count is
zero is incorrect.  In this case, the reference count initialization
is incorrect as well and could cause arenas to be put on the free
list too early (while they still have attached threads).

	* malloc/arena.c (get_free_list): Remove assert and adjust
	reference count handling.  Add comment about reused_arena
	interaction.
	(reused_arena): Add comments abount get_free_list interaction.
	* malloc/tst-malloc-thread-exit.c: New file.
	* malloc/Makefile (tests): Add tst-malloc-thread-exit.
	(tst-malloc-thread-exit): Link against libpthread.
2015-12-16 12:39:48 +01:00
Florian Weimer
400e12265d Replace MUTEX_INITIALIZER with _LIBC_LOCK_INITIALIZER in generic code
* sysdeps/mach/hurd/libc-lock.h (_LIBC_LOCK_INITIALIZER): Define.
	(__libc_lock_define_initialized): Use it.
	* sysdeps/nptl/libc-lockP.h (_LIBC_LOCK_INITIALIZER): Define.
	* malloc/arena.c (list_lock): Use _LIBC_LOCK_INITIALIZER.
	* malloc/malloc.c (main_arena): Likewise.
	* sysdeps/generic/malloc-machine.h (MUTEX_INITIALIZER): Remove.
	* sysdeps/nptl/malloc-machine.h (MUTEX_INITIALIZER): Remove.
2015-11-24 16:37:15 +01:00
Florian Weimer
a62719ba90 malloc: Prevent arena free_list from turning cyclic [BZ #19048]
[BZ# 19048]
	* malloc/malloc.c (struct malloc_state): Update comment.  Add
	attached_threads member.
	(main_arena): Initialize attached_threads.
	* malloc/arena.c (list_lock): Update comment.
	(ptmalloc_lock_all, ptmalloc_unlock_all): Likewise.
	(ptmalloc_unlock_all2): Reinitialize arena reference counts.
	(deattach_arena): New function.
	(_int_new_arena): Initialize arena reference count and deattach
	replaced arena.
	(get_free_list, reused_arena): Update reference count and deattach
	replaced arena.
	(arena_thread_freeres): Update arena reference count and only put
	unreferenced arenas on the free list.
2015-10-28 21:29:23 +01:00
Florian Weimer
6782806d8f malloc: Rewrite with explicit TLS access using __thread 2015-10-17 12:06:48 +02:00
Carlos O'Donell
e4bc326dbb malloc: Consistently apply trim_threshold to all heaps (Bug 17195)
In the per-thread arenas we apply trim_threshold-based checks
to the extra space between the pad and the top_area. This isn't
quite accurate and instead we should be harmonizing with the way
in which trim_treshold is applied everywhere else like sysrtim
and _int_free. The trimming check should be based on the size of
the top chunk and only the size of the top chunk. The following
patch harmonizes the trimming and make it consistent for the main
arena and thread arenas.

In the old code a large padding request might have meant that
trimming was not triggered. Now trimming is considered first based
on the chunk, then the pad is subtracted, and the remainder trimmed.
This is how all the other trimmings operate. I didn't measure the
performance difference of this change because it corrects what I
consider to be a behavioural anomaly. We'll need some profile driven
optimization to make this code better, and even there Ondrej and
others have better ideas on how to speedup malloc.

Tested on x86_64 with no regressions. Already reviewed by Siddhesh
Poyarekar and Mel Gorman here and discussed here:
https://sourceware.org/ml/libc-alpha/2015-05/msg00002.html
2015-10-07 22:21:36 -04:00
Siddhesh Poyarekar
c3b9ef8dfc Don't use the main arena in retry path if it is corrupt
If allocation on a non-main arena fails, the main arena is used
without checking to see if it is corrupt.  Add a check that avoids the
main arena if it is corrupt.

	* malloc/arena.c (arena_get_retry): Don't use main_arena if it is
	corrupt.
2015-08-24 14:33:07 +05:30
Siddhesh Poyarekar
92a9b22d70 Drop unused first argument from arena_get2
The arena pointer in the first argument to arena_get2 was used in the
old days before per-thread arenas.  They're unused now and hence can
be dropped.

ChangeLog:

	* malloc/arena.c (arena_get2): Drop unused argument.
	(arena_lock): Adjust.
	(arena_get_retry): Likewise.
2015-08-24 14:32:07 +05:30
Mel Gorman
f8ef472c0f malloc: Do not corrupt the top of a threaded heap if top chunk is MINSIZE [BZ #18502]
mksquashfs was reported in openSUSE to be causing segmentation faults when
creating installation images. Testing showed that mksquashfs sometimes
failed and could be reproduced within 10 attempts. The core dump looked
like the heap top was corrupted and was pointing to an unmapped area. In
other cases, this has been due to an application corrupting glibc structures
but mksquashfs appears to be fine in this regard.

The problem is that heap_trim is "growing" the top into unmapped space.
If the top chunk == MINSIZE then top_area is -1 and this check does not
behave as expected due to a signed/unsigned comparison

  if (top_area <= pad)
    return 0;

The next calculation extra = ALIGN_DOWN(top_area - pad, pagesz) calculates
extra as a negative number which also is unnoticed due to a signed/unsigned
comparison. We then call shrink_heap(heap, negative_number) which crashes
later. This patch adds a simple check against MINSIZE to make sure extra
does not become negative. It adds a cast to hint to the reader that this
is a signed vs unsigned issue.

Without the patch, mksquash fails within 10 attempts. With it applied, it
completed 1000 times without error. The standard test suite "make check"
showed no changes in the summary of test results.
2015-06-26 22:47:45 +02:00
Siddhesh Poyarekar
fff94fa224 Avoid deadlock in malloc on backtrace (BZ #16159)
When the malloc subsystem detects some kind of memory corruption,
depending on the configuration it prints the error, a backtrace, a
memory map and then aborts the process.  In this process, the
backtrace() call may result in a call to malloc, resulting in
various kinds of problematic behavior.

In one case, the malloc it calls may detect a corruption and call
backtrace again, and a stack overflow may result due to the infinite
recursion.  In another case, the malloc it calls may deadlock on an
arena lock with the malloc (or free, realloc, etc.) that detected the
corruption.  In yet another case, if the program is linked with
pthreads, backtrace may do a pthread_once initialization, which
deadlocks on itself.

In all these cases, the program exit is not as intended.  This is
avoidable by marking the arena that malloc detected a corruption on,
as unusable.  The following patch does that.  Features of this patch
are as follows:

- A flag is added to the mstate struct of the arena to indicate if the
  arena is corrupt.

- The flag is checked whenever malloc functions try to get a lock on
  an arena.  If the arena is unusable, a NULL is returned, causing the
  malloc to use mmap or try the next arena.

- malloc_printerr sets the corrupt flag on the arena when it detects a
  corruption

- free does not concern itself with the flag at all.  It is not
  important since the backtrace workflow does not need free.  A free
  in a parallel thread may cause another corruption, but that's not
  new

- The flag check and set are not atomic and may race.  This is fine
  since we don't care about contention during the flag check.  We want
  to make sure that the malloc call in the backtrace does not trip on
  itself and all that action happens in the same thread and not across
  threads.

I verified that the test case does not show any regressions due to
this patch.  I also ran the malloc benchmarks and found an
insignificant difference in timings (< 2%).

	* malloc/Makefile (tests): New test case tst-malloc-backtrace.
	* malloc/arena.c (arena_lock): Check if arena is corrupt.
	(reused_arena): Find a non-corrupt arena.
	(heap_trim): Pass arena to unlink.
	* malloc/hooks.c (malloc_check_get_size): Pass arena to
	malloc_printerr.
	(top_check): Likewise.
	(free_check): Likewise.
	(realloc_check): Likewise.
	* malloc/malloc.c (malloc_printerr): Add arena argument.
	(unlink): Likewise.
	(munmap_chunk): Adjust.
	(ARENA_CORRUPTION_BIT): New macro.
	(arena_is_corrupt): Likewise.
	(set_arena_corrupt): Likewise.
	(sysmalloc): Use mmap if there are no usable arenas.
	(_int_malloc): Likewise.
	(__libc_malloc): Don't fail if arena_get returns NULL.
	(_mid_memalign): Likewise.
	(__libc_calloc): Likewise.
	(__libc_realloc): Adjust for additional argument to
	malloc_printerr.
	(_int_free): Likewise.
	(malloc_consolidate): Likewise.
	(_int_realloc): Likewise.
	(_int_memalign): Don't touch corrupt arenas.
	* malloc/tst-malloc-backtrace.c: New test case.
2015-05-19 06:40:38 +05:30
Mel Gorman
c26efef979 malloc: Consistently apply trim_threshold to all heaps [BZ #17195]
Trimming heaps is a balance between saving memory and the system overhead
required to update page tables and discard allocated pages. The malloc
option M_TRIM_THRESHOLD is a tunable that users are meant to use to decide
where this balance point is but it is only applied to the main arena.

For scalability reasons, glibc malloc has per-thread heaps but these are
shrunk with madvise() if there is one page free at the top of the heap.
In some circumstances this can lead to high system overhead if a thread
has a control flow like

    while (data_to_process) {
        buf = malloc(large_size);
        do_stuff();
        free(buf);
    }

For a large size, the free() will call madvise (pagetable teardown, page
free and TLB flush) every time followed immediately by a malloc (fault,
kernel page alloc, zeroing and charge accounting). The kernel overhead
can dominate such a workload.

This patch allows the user to tune when madvise gets called by applying
the trim threshold to the per-thread heaps and using similar logic to the
main arena when deciding whether to shrink. Alternatively if the dynamic
brk/mmap threshold gets adjusted then the new values will be obeyed by
the per-thread heaps.

Bug 17195 was a test case motivated by a problem encountered in scientific
applications written in python that performance badly due to high page fault
overhead. The basic operation of such a program was posted by Julian Taylor
https://sourceware.org/ml/libc-alpha/2015-02/msg00373.html

With this patch applied, the overhead is eliminated. All numbers in this
report are in seconds and were recorded by running Julian's program 30
times.

pyarray
                                 glibc               madvise
                                  2.21                    v2
System  min             1.81 (  0.00%)        0.00 (100.00%)
System  mean            1.93 (  0.00%)        0.02 ( 99.20%)
System  stddev          0.06 (  0.00%)        0.01 ( 88.99%)
System  max             2.06 (  0.00%)        0.03 ( 98.54%)
Elapsed min             3.26 (  0.00%)        2.37 ( 27.30%)
Elapsed mean            3.39 (  0.00%)        2.41 ( 28.84%)
Elapsed stddev          0.14 (  0.00%)        0.02 ( 82.73%)
Elapsed max             4.05 (  0.00%)        2.47 ( 39.01%)

               glibc     madvise
                2.21          v2
User          141.86      142.28
System         57.94        0.60
Elapsed       102.02       72.66

Note that almost a minutes worth of system time is eliminted and the
program completes 28% faster on average.

To illustrate the problem without python this is a basic test-case for
the worst case scenario where every free is a madvise followed by a an alloc

/* gcc bench-free.c -lpthread -o bench-free */
static int num = 1024;

void __attribute__((noinline,noclone)) dostuff (void *p)
{
}

void *worker (void *data)
{
  int i;

  for (i = num; i--;)
    {
      void *m = malloc (48*4096);
      dostuff (m);
      free (m);
    }

  return NULL;
}

int main()
{
  int i;
  pthread_t t;
  void *ret;
  if (pthread_create (&t, NULL, worker, NULL))
    exit (2);
  if (pthread_join (t, &ret))
    exit (3);
  return 0;
}

Before the patch, this resulted in 1024 calls to madvise. With the patch applied,
madvise is called twice because the default trim threshold is high enough to avoid
this.

This a more complex case where there is a mix of frees. It's simply a different worker
function for the test case above

void *worker (void *data)
{
  int i;
  int j = 0;
  void *free_index[num];

  for (i = num; i--;)
    {
      void *m = malloc ((i % 58) *4096);
      dostuff (m);
      if (i % 2 == 0) {
        free (m);
      } else {
        free_index[j++] = m;
      }
    }
  for (; j >= 0; j--)
    {
      free(free_index[j]);
    }

  return NULL;
}

glibc 2.21 calls malloc 90305 times but with the patch applied, it's
called 13438. Increasing the trim threshold will decrease the number of
times it's called with the option of eliminating the overhead.

ebizzy is meant to generate a workload resembling common web application
server workloads. It is threaded with a large working set that at its core
has an allocation, do_stuff, free loop that also hits this case. The primary
metric of the benchmark is records processed per second. This is running on
my desktop which is a single socket machine with an I7-4770 and 8 cores.
Each thread count was run for 30 seconds. It was only run once as the
performance difference is so high that the variation is insignificant.

                glibc 2.21              patch
threads 1            10230              44114
threads 2            19153              84925
threads 4            34295             134569
threads 8            51007             183387

Note that the saving happens to be a concidence as the size allocated
by ebizzy was less than the default threshold. If a different number of
chunks were specified then it may also be necessary to tune the threshold
to compensate

This is roughly quadrupling the performance of this benchmark. The difference in
system CPU usage illustrates why.

ebizzy running 1 thread with glibc 2.21
10230 records/s 306904
real 30.00 s
user  7.47 s
sys  22.49 s

22.49 seconds was spent in the kernel for a workload runinng 30 seconds. With the
patch applied

ebizzy running 1 thread with patch applied
44126 records/s 1323792
real 30.00 s
user 29.97 s
sys   0.00 s

system CPU usage was zero with the patch applied. strace shows that glibc
running this workload calls madvise approximately 9000 times a second. With
the patch applied madvise was called twice during the workload (or 0.06
times per second).

2015-02-10  Mel Gorman  <mgorman@suse.de>

  [BZ #17195]
  * malloc/arena.c (free): Apply trim threshold to per-thread heaps
    as well as the main arena.
2015-04-02 12:14:14 +05:30
Carlos O'Donell
8a35c3fe12 Use alignment macros, pagesize and powerof2.
We are replacing all of the bespoke alignment code with
ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and PTR_ALIGN_DOWN.
This cleans up malloc/malloc.c, malloc/arena.c, and
elf/dl-reloc.c. It also makes all the code consistently
use pagesize, and powerof2 as required.

Code size is reduced with the removal of precomputed
pagemask, and use of pagesize instead. No measurable
difference in performance.

No regressions on x86_64.
2015-02-17 19:29:15 -05:00
Joseph Myers
b168057aaa Update copyright dates with scripts/update-copyrights. 2015-01-02 16:29:47 +00:00
Ondřej Bílka
a1ffb40e32 Use glibc_likely instead __builtin_expect. 2014-02-10 15:07:12 +01:00
Ondřej Bílka
bdfe308a16 Remove THREAD_STATS.
A THREAD_STATS macro duplicates gathering information that could be
obtained by systemtap probes instead.
2014-02-10 12:25:04 +01:00
Ondřej Bílka
6c8dbf00f5 Reformat malloc to gnu style. 2014-01-02 09:40:10 +01:00
Allan McRae
d4697bc93d Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
Ondřej Bílka
5782a80f9f Drop PER_THREAD conditionals from malloc. 2013-12-10 18:00:14 +01:00