glibc/benchtests
Noah Goldstein cf3acd774f Benchtests: Add benchtests for __memcmpeq
No bug. This commit adds __memcmpeq benchmarks. The benchmarks just
use the existing ones in memcmp. This will be useful for testing
implementations of __memcmpeq that do not just alias memcmp.
2021-10-27 13:03:46 -05:00
..
scripts benchtests: Fix validate_benchout.py exceptions 2021-09-16 09:19:55 +05:30
strcoll-inputs Remove sysdeps/*/tls-macros.h 2021-08-18 09:15:20 -07:00
acos-inputs Remove obsolete comments/name from acos-inputs, since slow path was removed. 2021-08-02 15:05:22 +02:00
acosh-inputs benchmark inputs for asinh and acosh 2013-12-31 12:05:16 +05:30
asin-inputs Remove obsolete comments/name from several benchtest input files. 2021-08-02 15:27:16 +02:00
asinh-inputs benchmark inputs for asinh and acosh 2013-12-31 12:05:16 +05:30
atan2-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
atan-inputs Remove obsolete comments/name from several benchtest input files. 2021-08-02 15:27:16 +02:00
atanh-inputs benchmark inputs for tanh and atanh 2013-12-31 12:06:30 +05:30
bench-malloc-simple.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-malloc-thread.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-math-inlines.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-memccpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-memchr.c Bench: Expand bench-memchr.c 2021-05-03 10:18:11 -07:00
bench-memcmp.c Benchtests: Add benchtests for __memcmpeq 2021-10-27 13:03:46 -05:00
bench-memcmpeq.c Benchtests: Add benchtests for __memcmpeq 2021-10-27 13:03:46 -05:00
bench-memcpy-large.c Bench: Add support for choose direction of memcpy in benchtests 2021-05-23 19:36:36 -04:00
bench-memcpy-random.c benchtests: Fixed bench-memcpy-random: buf1: mprotect failed 2021-05-26 12:01:06 +01:00
bench-memcpy-walk.c Bench: Add support for choose direction of memcpy in benchtests 2021-05-23 19:36:36 -04:00
bench-memcpy.c Bench: Add support for choose direction of memcpy in benchtests 2021-05-23 19:36:36 -04:00
bench-memmem.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-memmove-large.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-memmove-walk.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-memmove.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-mempcpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-memrchr.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-memset-large.c benchtests: Remove redundant assert.h 2021-09-13 09:08:26 +05:30
bench-memset-walk.c benchtests: Remove redundant assert.h 2021-09-13 09:08:26 +05:30
bench-memset.c benchtests: Add medium cases and increase iters in bench-memset.c 2021-10-08 15:13:06 -05:00
bench-pthread-locks.c benchtests: Fix pthread-locks test to produce valid json 2021-04-18 12:56:29 +05:30
bench-rawmemchr.c benchtests: Use JSON for bench-rawmemchr output 2021-05-17 11:10:19 -03:00
bench-skeleton.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-stpcpy_chk.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-stpcpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-stpncpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcasecmp.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcasestr.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcat.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strchr.c strchr: Add additional benchmarks and tests 2021-02-08 11:34:00 -08:00
bench-strchrnul.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcmp.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcoll.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcpy_chk.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strcspn.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-string.h tests: use xmalloc to allocate implementation array 2021-07-28 17:45:19 +05:30
bench-strlen.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strncasecmp.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strncat.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strncmp.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strncpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strnlen.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strpbrk.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strrchr.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strsep.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strspn.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strstr.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strtod.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-strtok.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-timing-type.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-timing.h Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-util.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-util.h Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcpcpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcpncpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcscat.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcschr.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcschrnul.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcscmp.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcscpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcscspn.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcslen.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcsncat.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcsncmp.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcsncpy.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcsnlen.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcspbrk.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcsrchr.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wcsspn.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wmemchr.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wmemcmp.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
bench-wmemset.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
cbrt-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
cbrtl-inputs add workload traces for cbrtl 2021-05-10 18:45:34 +02:00
cos-inputs Remove obsolete comments/name from several benchtest input files. 2021-08-02 15:27:16 +02:00
cosf-inputs Benchtests for sinf, cosf and sincosf 2017-10-13 14:19:45 +05:30
cosh-inputs benchmark inputs for sinh and cosh 2013-12-31 12:03:44 +05:30
erf-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
erfc-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
exp2-inputs benchmark inputs for exp2, log2, log and tan 2013-12-12 09:31:53 +05:30
exp2f-inputs Add exp2f and log2f benchmark trace 2017-09-20 10:04:12 +01:00
exp10-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
exp10f-inputs benchtests: Fix name of exp10f benchmark variant 2021-04-18 12:56:33 +05:30
exp-inputs benchtests: Add "workload" traces for exp 2020-09-10 15:25:22 -03:00
expf128-inputs benchtests: Run _Float128 tests only on architectures that support it 2020-09-23 16:11:57 +02:00
expf-inputs Add expf trace 2017-09-19 15:14:18 +01:00
expm1-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
ffs-inputs benchtests: Add benchtests for ffs and ffsll 2014-03-31 12:50:41 +01:00
ffsll-inputs benchtests: Add benchtests for ffs and ffsll 2014-03-31 12:50:41 +01:00
fmax-inputs benchtests: Add fmax/fmin benchmarks 2016-12-19 16:04:16 -02:00
fmaxf-inputs benchtests: Add fmaxf/fminf benchmarks 2016-12-19 16:04:16 -02:00
fmin-inputs benchtests: Add fmax/fmin benchmarks 2016-12-19 16:04:16 -02:00
fminf-inputs benchtests: Add fmaxf/fminf benchmarks 2016-12-19 16:04:16 -02:00
hypot-inputs benchtests: hypot benchmark 2019-07-08 17:14:04 -03:00
ilogb-inputs benchtests: Add ilogb* tests 2021-03-16 12:19:09 -03:00
ilogbf128-inputs benchtests: Add ilogb* tests 2021-03-16 12:19:09 -03:00
ilogbf-inputs benchtests: Add ilogb* tests 2021-03-16 12:19:09 -03:00
isfinite-inputs benchtests: Add isnan/isinf/isfinite benchmark 2019-06-12 11:46:30 -03:00
isinf-inputs benchtests: Add isnan/isinf/isfinite benchmark 2019-06-12 11:46:30 -03:00
isnan-inputs benchtests: Add isnan/isinf/isfinite benchmark 2019-06-12 11:46:30 -03:00
j0-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
j1-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
json-lib.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
json-lib.h Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
lgamma-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
log1p-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
log2-inputs benchmark inputs for exp2, log2, log and tan 2013-12-12 09:31:53 +05:30
log2f-inputs Add exp2f and log2f benchmark trace 2017-09-20 10:04:12 +01:00
log10-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
log-inputs benchmark inputs for exp2, log2, log and tan 2013-12-12 09:31:53 +05:30
logb-inputs benchtests: Add logb{f} benchmark 2019-07-08 17:22:22 -03:00
logbf-inputs benchtests: Add logb{f} benchmark 2019-07-08 17:22:22 -03:00
logf-inputs Add logf trace 2017-09-19 15:14:46 +01:00
Makefile Benchtests: Add benchtests for __memcmpeq 2021-10-27 13:03:46 -05:00
modf-inputs [benchtests] Use inputs file for modf 2014-03-29 09:35:50 +05:30
pow-inputs Remove obsolete comments/name from several benchtest input files. 2021-08-02 15:27:16 +02:00
powf128-inputs benchtests: Run _Float128 tests only on architectures that support it 2020-09-23 16:11:57 +02:00
powf-inputs Add powf trace 2017-06-20 16:50:37 +01:00
pthread_once-inputs benchtests: Add pthread_once common-case test. 2014-04-10 21:22:28 +02:00
pthread_once-source.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
README benchtests: Building benchmarks as static executables 2021-10-04 10:09:13 -07:00
rint-inputs Add more directives to benchmark input files 2013-10-07 11:51:25 +05:30
roundeven-inputs Add benchtests for roundeven and roundevenf. 2020-03-27 23:24:02 +00:00
roundevenf-inputs Add benchtests for roundeven and roundevenf. 2020-03-27 23:24:02 +00:00
sin-inputs Remove obsolete comments/name from several benchtest input files. 2021-08-02 15:27:16 +02:00
sincos-inputs benchtests: Add inputs from sin and cos to sincos 2015-12-09 00:10:51 +05:30
sincosf-inputs Benchtests for sinf, cosf and sincosf 2017-10-13 14:19:45 +05:30
sinf128-inputs benchtests: Run _Float128 tests only on architectures that support it 2020-09-23 16:11:57 +02:00
sinf-inputs benchtests: Add "workload" traces for sinf 2020-09-10 15:25:22 -03:00
sinh-inputs benchmark inputs for sinh and cosh 2013-12-31 12:03:44 +05:30
sprintf-inputs Add sprintf benchmark. 2015-05-21 10:02:52 -04:00
sprintf-source.c Add sprintf benchmark. 2015-05-21 10:02:52 -04:00
sqrt-inputs Benchmark test for sqrt function. 2013-12-02 09:37:18 -08:00
tan-inputs Remove obsolete comments/name from several benchtest input files. 2021-08-02 15:27:16 +02:00
tanh-inputs benchmark inputs for tanh and atanh 2013-12-31 12:06:30 +05:30
tgamma-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
thread_create-inputs nptl: Invert the mmap/mprotect logic on allocated stacks (BZ#18988) 2017-06-14 17:22:35 -03:00
thread_create-source.c Update copyright dates with scripts/update-copyrights 2021-01-02 12:17:34 -08:00
trunc-inputs Add benchtests for trunc and truncf. 2017-09-19 12:59:01 +00:00
truncf-inputs Add benchtests for trunc and truncf. 2017-09-19 12:59:01 +00:00
y0-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00
y1-inputs add workload traces for missing functions (double format) 2021-03-29 16:23:19 +02:00

Using the glibc microbenchmark suite
====================================

The glibc microbenchmark suite automatically generates code for specified
functions, builds and calls them repeatedly for given inputs to give some
basic performance properties of the function.

Running the benchmark:
=====================

The benchmark needs python 2.7 or later in addition to the
dependencies required to build the GNU C Library.  One may run the
benchmark by invoking make as follows:

  $ make bench

This runs each function for 10 seconds and appends its output to
benchtests/bench.out.  To ensure that the tests are rebuilt, one could run:

  $ make bench-clean

The duration of each test can be configured setting the BENCH_DURATION variable
in the call to make.  One should run `make bench-clean' before changing
BENCH_DURATION.

  $ make BENCH_DURATION=1 bench

The benchmark suite does function call measurements using architecture-specific
high precision timing instructions whenever available.  When such support is
not available, it uses clock_gettime (CLOCK_MONOTONIC).  One can force the
benchmark to use clock_gettime by invoking make as follows:

  $ make USE_CLOCK_GETTIME=1 bench

Again, one must run `make bench-clean' before changing the measurement method.

On x86 processors, RDTSCP instruction provides more precise timing data
than RDTSC instruction.  All x86 processors since 2010 support RDTSCP
instruction.  One can force the benchmark to use RDTSCP by invoking make
as follows:

  $ make USE_RDTSCP=1 bench

One must run `make bench-clean' before changing the measurement method.

Running benchmarks on another target:
====================================

If the target where you want to run benchmarks is not capable of building the
code or you're cross-building, you could build and execute the benchmark in
separate steps.  On the build system run:

  $ make bench-build

and then copy the source and build directories to the target and run the
benchmarks from the build directory as usual:

  $ make bench

make sure the copy preserves timestamps by using either rsync or scp -p
otherwise the above command may try to build the benchmark again.  Benchmarks
that require generated code to be executed during the build are skipped when
cross-building.

Building benchmarks as static executables:
=========================================

To build benchmarks as static executables, on the build system, run:

  $ make STATIC-BENCHTESTS=yes bench-build

You can copy benchmark executables to another machine and run them
without copying the source nor build directories.

Running subsets of benchmarks:
==============================

To run only a subset of benchmarks, one may invoke make as follows

  $ make bench BENCHSET="bench-pthread bench-math malloc-thread"

where BENCHSET may be a space-separated list of the following values:

    bench-math
    bench-pthread
    bench-string
    string-benchset
    wcsmbs-benchset
    stdlib-benchset
    stdio-common-benchset
    math-benchset
    malloc-thread

Adding a function to benchtests:
===============================

If the name of the function is `foo', then the following procedure should allow
one to add `foo' to the bench tests:

- Append the function name to the bench variable in the Makefile.

- Make a file called `foo-inputs` to provide the definition and input for the
  function.  The file should have some directives telling the parser script
  about the function and then one input per line.  Directives are lines that
  have a special meaning for the parser and they begin with two hashes '##'.
  The following directives are recognized:

  - args: This should be assigned a colon separated list of types of the input
    arguments.  This directive may be skipped if the function does not take any
    inputs.  One may identify output arguments by nesting them in <>.  The
    generator will create variables to get outputs from the calling function.
  - ret: This should be assigned the type that the function returns.  This
    directive may be skipped if the function does not return a value.
  - includes: This should be assigned a comma-separated list of headers that
    need to be included to provide declarations for the function and types it
    may need (specifically, this includes using "#include <header>").
  - include-sources: This should be assigned a comma-separated list of source
    files that need to be included to provide definitions of global variables
    and functions (specifically, this includes using "#include "source").
    See pthread_once-inputs and pthreads_once-source.c for an example of how
    to use this to benchmark a function that needs state across several calls.
  - init: Name of an initializer function to call to initialize the benchtest.
  - name: See following section for instructions on how to use this directive.

  Lines beginning with a single hash '#' are treated as comments.  See
  pow-inputs for an example of an input file.

Multiple execution units per function:
=====================================

Some functions have distinct performance characteristics for different input
domains and it may be necessary to measure those separately.  For example, some
math functions perform computations at different levels of precision (64-bit vs
240-bit vs 768-bit) and mixing them does not give a very useful picture of the
performance of these functions.  One could separate inputs for these domains in
the same file by using the `name' directive that looks something like this:

  ##name: 240bits

All inputs after the ##name: 240bits directive and until the next `name'
directive (or the end of file) are part of the "240bits" benchmark and
will be output separately in benchtests/bench.out.  See the pow-inputs file
for an example of what such a partitioned input file would look like.

It is also possible to measure latency and reciprocal throughput of a
(partial) trace extracted from a real workload.  In this case the whole trace
is iterated over multiple times rather than repeating every input multiple
times.  This can be done via:

  ##name: workload-<name>

where <name> is simply used to distinguish between different traces in the
same file.  To create such a trace, you can simply extract using printf()
values uses for a specific application, or generate random values in some
interval.  See the expf-inputs file for an example of this workload mechanism.

Benchmark Sets:
==============

In addition to standard benchmarking of functions, one may also generate
custom outputs for a set of functions.  This is currently used by string
function benchmarks where the aim is to compare performance between
implementations at various alignments and for various sizes.

To add a benchset for `foo':

- Add `foo' to the benchset variable.
- Write your bench-foo.c that prints out the measurements to stdout.
- On execution, a bench-foo.out is created in $(objpfx) with the contents of
  stdout.

Reading String Benchmark Results:
================================

Some of the string benchmark results are now in JSON to make it easier to read
in scripts.  Use the benchtests/compare_strings.py script to show the results
in a tabular format, generate graphs and more. Run

    benchtests/scripts/compare_strings.py -h

for usage information.