Commit Graph

194 Commits

Author SHA1 Message Date
Siddhesh Poyarekar
df26ea5359 Implement benchmarking script in python
Implemented the benchmark script in python since it is much cleaner
and simpler to maintain.
2014-03-21 17:32:50 +05:30
Roland McGrath
c8f8fa1504 Work around binutils bugs in 2.23 and older
binutils versions up through at least 2.23 have some bugs that cause
STV_HIDDEN symbols to appear in .dynsyms.
2014-03-18 22:04:04 +05:30
Joseph Myers
d6fe5e582d Do not terminate default test runs on test failure.
This patch is an updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00198.html> and
<https://sourceware.org/ml/libc-alpha/2014-03/msg00180.html>.

Normal practice for software testsuites is that rather than
terminating immediately when a test fails, they continue running and
report at the end on how many tests passed or failed.

The principle behind the glibc testsuite stopping on failure was
probably that the expected state is no failures and so any failure
indicates a problem such as miscompilation.  In practice, while this
is fairly close to true for native testing on x86_64 and x86 (kernel
bugs and race conditions can still cause intermittent failures), it's
less likely to be the case on other platforms, and so people testing
glibc run the testsuite with "make -k" and then examine the logs to
determine whether the failures are what they expect to fail on that
platform, possibly with some automation for the comparison.

This patch switches the glibc testsuite to the normal convention of
not stopping on failure - unless you use stop-on-test-failure=y, in
which case it behaves essentially as it did before (and does not
generate overall test summaries on failure).  Instead, the summary
tests.sum may contain tests that FAILed.  At the end of the test run,
any FAIL or ERROR lines from tests.sum are printed, and then it exits
with error status if there were any such lines.  In addition, build
failures will also cause the test run to stop - this has the
justification that those *do* indicate serious problems that should be
promptly fixed and aren't generally hard to fix (but apart from that,
avoiding the build stopping on those failures seems harder).

Note that unlike the previous patches in this series, this *does*
require people with automation around testing glibc to change their
processes - either to start using tests.sum / xtests.sum to track
failures and compare them with expectations (with or without also
using "make -k" and examining "make" logs to identify build failures),
or else to use stop-on-test-failure=y and ignore the new tests.sum /
xtests.sum mechanism.  (If all you check is the exit status from "make
check", no changes are needed unless you want to avoid test runs
continuing after the first failure.)

Tested x86_64.

	* scripts/evaluate-test.sh: Handle fourth argument to determine
	whether test run should stop on failure.
	* Makeconfig (stop-on-test-failure): New variable.
	(evaluate-test): Pass fourth argument to evaluate-test.sh based on
	$(stop-on-test-failure).
	* Makefile (tests): Give a summary of results from testing and
	exit with failure status if they include an ERROR or FAIL.
	(xtests): Likewise.
	* manual/install.texi (Configuring and compiling): Mention
	stop-on-test-failure=y.
	* INSTALL: Regenerated.
2014-03-14 21:02:40 +00:00
Roland McGrath
22dbc19dbb Get rid of Versions.def source file 2014-03-14 11:39:56 -07:00
Joseph Myers
265d52abcb Generate overall summary of test results.
This patch, an updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00197.html>, makes
testsuite runs generate an overall summary of test results.

A new script merge-test-results.sh deals both with collecting results
within a directory to a file with all the results from that directory,
and collecting the results from subdirectories into a single overall
file (there's not much in common between the two modes of operation of
the script, but it seemed silly to have two separate scripts for
this).  Within a directory, missing results produce UNRESOLVED lines;
at top level, missing results for a whole directory produce an ERROR
line (since toplevel can't identify what the specific missing tests
are in this case).

Note that this does not change the rules for when "make" considers
there has been an error, or terminates, so unexpected failures will
still cause make to terminate, or, with -k, mean the commands for
"tests" don't get run because of failure of a dependency.

Tested x86_64, including that the summary does in fact reflect all the
tests with .test-result files.

	* scripts/merge-test-results.sh: New file.
	* Makefile (tests-special-notdir): New variable.
	(tests): Run merge-test-results.sh.
	(xtests): Likewise.
	* Rules (tests-special-notdir): New variable.
	(xtests-special-notdir): Likewise.
	(tests): Run merge-test-results.sh
	(xtests): Likewise.
2014-03-07 03:25:57 +00:00
Joseph Myers
f8c17e79fa Support expected failures in .test-result files.
This patch, an updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00195.html>, makes it
possible for .test-result files for individual tests to contain XPASS
and XFAIL rather than PASS and FAIL in cases where failure is
expected.  This replaces the marking of two individual tests with "-"
to cause them to be expected at makefile level to fail;
evaluate-test.sh will ensure it exits with status 0 for an expected
failure.

Tested x86_64.

	* scripts/evaluate-test.sh: Take new argument indicating whether
	failure is expected.
	* Makeconfig (evaluate-test): Pass argument to evaluate-test.sh
	indicating whether failure is expected.
	* conform/Makefile (test-xfail-run-conformtest): New variable.
	($(objpfx)run-conformtest.out): Don't expect to fail at makefile
	level.
	* posix/Makefile (test-xfail-annexc): New variable.
	($(objpfx)annexc.out): Don't expect to fail at makefile level.
2014-02-27 03:25:27 +00:00
Tomas Dohnalek
591aeaf7a9 Generate .test-result files for ordinary tests.
This patch, an updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00193.html>, starts
the process of generating explicit PASS or FAIL status for individual
glibc tests.  It's based on Tomas Dohnalek's patch
<https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>, but is
deliberately more minimal: it doesn't try to cover any tests outside
of $(tests) / $(xtests) (that's for a later patch), nor does it put
the result together in an overall summary file (again, a later patch):
it just generates the .test-result files.

Thus, this patch keeps the overall logic for when a testsuite run
finishes completely unchanged: a test failing will terminate the run.
I think we *should* move to a more conventional approach where plain
"make check" does not terminate for an individual test failure, unless
e.g. you say "make stop-on-test-failure=y check", but that sort of
policy change is best done as a separate patch once the infrastructure
is in place to generate summary files for completed test runs (which
will entirely consist of PASS and XFAIL lines if the testsuite run
reaches the point of generating them, until such a policy change is
made).

Tested x86_64.

2014-02-14  Tomas Dohnalek  <tdohnale@redhat.com>
	    Joseph Myers  <joseph@codesourcery.com>

	* Makeconfig (test-name): New variable.
	(evaluate-test): Likewise.
	* Makerules (do-test-clean): Remove .test-result files.
	(common-mostlyclean): Likewise.
	* Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules.
	* scripts/evaluate-test.sh: New file.
2014-02-15 01:04:57 +00:00
Allan McRae
ddb3687f2a scripts/update-copyrights: adjust configure input file suffix 2014-01-01 22:10:54 +10:00
Allan McRae
88726d48af Update remaining copyright dates
Update copyright years that are not handled by scripts/update-copyright.
2014-01-01 22:02:55 +10:00
Allan McRae
d4697bc93d Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
Joseph Myers
21fea2e228 Update texinfo.tex, config.guess, config.sub from upstream.
This patch updates various miscellaneous files we take from upstream
GNU sources (texinfo.texi, config.guess, config.sub - various others
haven't changed upstream since we last updated them) to their current
upstream versions.

Tested x86_64.

	* manual/texinfo.tex: Update to version 2013-11-26.10 with
	trailing whitespace removed.
	* scripts/config.guess: Update to version 2013-11-29.
	* scripts/config.sub: Update to version 2013-10-01.
2013-12-19 17:36:10 +00:00
Siddhesh Poyarekar
f447922094 benchtests: Append volatile keyword to type instead of prepending
`volatile int` means the same as 'int volatile', but that's not the
case for 'volatile char *' and 'char * volatile'.  We won't need a
'char volatile *' or other complicated semantics for now.
2013-12-06 09:02:19 +05:30
Siddhesh Poyarekar
9298ecba15 Accept output arguments to benchmark functions
This patch adds the ability to accept output arguments to functions
being benchmarked, by nesting the argument type in <> in the args
directive.  It includes the sincos implementation as an example, where
the function would have the following args directive:

  ## args: double:<double *>:<double *>

This simply adds a definition for a static variable whose pointer gets
passed into the function, so it's not yet possible to pass something
more complicated like a pre-allocated string or array.  That would be
a good feature to add if a function needs it.

The values in the input file will map only to the input arguments.  So
if I had a directive like this for a function foo:

  ## args: int:<int *>:int:<int *>

and I have a value list like this:

1, 2
3, 4
5, 6

then the function calls generated would be:

foo (1, &out1, 2, &out2);
foo (3, &out1, 4, &out2);
foo (5, &out1, 6, &out2);
2013-12-05 10:12:59 +05:30
Siddhesh Poyarekar
ecaf142d3d benchtests: skip over blank lines in benchmark input files 2013-12-04 18:20:32 +05:30
Torvald Riegel
40fefba1b5 benchtests: Add include-sources directive.
This adds the "include-sources" directive to scripts/bench.pl.  This
allows for including source code (vs including headers, which might get
a different search path) after the inclusion of any headers.
2013-10-10 14:45:30 +03:00
Siddhesh Poyarekar
a357259bf8 Add more directives to benchmark input files
This patch adds some more directives to the benchmark inputs file,
moving functionality from the Makefile and making the code generation
script a bit cleaner.  The function argument and return types that
were earlier added as variables in the makefile and passed to the
script via command line arguments are now the 'args' and 'ret'
directive respectively.  'args' should be a colon separated list of
argument types (skipped if the function doesn't accept any arguments)
and 'ret' should be the return type.

Additionally, an 'includes' directive may have a comma separated list
of headers to include in the source.  For example, the pow input file
now looks like this:

42.0, 42.0
1.0000000000000020, 1.5

I did this to unclutter the benchtests Makefile a bit and eventually
eliminate dependency of the tests on the Makefile and have tests
depend on their respective include files only.
2013-10-07 11:51:25 +05:30
Roland McGrath
12086fb483 Sort sysd-rules-patterns by descending pattern length. 2013-06-17 09:55:49 -07:00
Roland McGrath
346d65b33a Rewrite sysd-rules generation using an awk script. 2013-06-17 09:55:21 -07:00
Joseph Myers
9e54314bb0 Update miscellaneous scripts from upstream. 2013-06-06 19:02:09 +00:00
Siddhesh Poyarekar
48a18de1e1 Prevent optimizing out of benchmark function call
Resolves: #15424

The compiler would optimize the benchmark function call out of the
loop and call it only once, resulting in blazingly fast times for some
benchmarks (notably atan, sin and cos).  Mark the inputs as volatile
so that the code is forced to read again from the input for each
iteration.
2013-05-17 19:10:33 +05:30
Siddhesh Poyarekar
f0ee064b7d Allow multiple input domains to be run in the same benchmark program
Some math functions have distinct performance characteristics in
specific domains of inputs, where some inputs return via a fast path
while other inputs require multiple precision calculations, that too
at different precision levels.  The way to implement different domains
was to have a separate source file and benchmark definition, resulting
in separate programs.

This clutters up the benchmark, so this change allows these domains to
be consolidated into the same input file.  To do this, the input file
format is now enhanced to allow comments with a preceding # and
directives with two # at the begining of a line.  A directive that
looks like:

tells the benchmark generation script that what follows is a different
domain of inputs.  The value of the 'name' directive (in this case,
foo) is used in the output.  The two input domains are then executed
sequentially and their results collated separately.  with the above
directive, there would be two lines in the result that look like:

func(): ....
func(foo): ...
2013-04-30 14:17:57 +05:30
Siddhesh Poyarekar
d569c6eeb4 Maintain runtime of each benchmark at ~10 seconds
The idea to run benchmarks for a constant number of iterations is
problematic.  While the benchmarks may run for 10 seconds on x86_64,
they could run for about 30 seconds on powerpc and worse, over 3
minutes on arm.  Besides that, adding a new benchmark is cumbersome
since one needs to find out the number of iterations needed for a
sufficient runtime.

A better idea would be to run each benchmark for a specific amount of
time.  This patch does just that.  The run time defaults to 10 seconds
and it is configurable at command line:

  make BENCH_DURATION=5 bench
2013-04-30 14:10:20 +05:30
Andreas Schwab
aaa8cb4b43 Add support for rtld directory different from slib directory 2013-03-19 12:07:26 +01:00
Siddhesh Poyarekar
8cfdb7e056 Framework for performance benchmarking of functions
See benchtests/Makefile to know how to use it.
2013-03-15 12:30:03 +05:30
Richard Henderson
7776238604 Fix file modes 2013-02-27 10:01:53 -08:00
Richard Henderson
b5977bf267 Sync config.guess and config.sub with upstream 2013-02-27 09:46:51 -08:00
Joseph Myers
e6898b8d92 Update miscellaneous copyright dates. 2013-01-02 19:43:40 +00:00
Joseph Myers
568035b787 Update copyright notices with scripts/update-copyrights. 2013-01-02 19:05:09 +00:00
Joseph Myers
f4cf5f2d8b Add script to update copyright notices and reformat some to facilitate its use. 2013-01-01 16:29:10 +00:00
Andreas Schwab
e3c6aa3a58 Properly handle indirect functions in ABI check on powerpc64 2012-11-28 16:19:08 +01:00
Pino Toscano
a93f9cbc27 check-local-headers: ignore Mach kernel headers 2012-11-16 18:55:19 +01:00
Andreas Schwab
a542b3894d Make cross-test-ssh.sh compatible with a remote POSIX sh 2012-11-03 08:06:12 +01:00
Andreas Schwab
4da224a2d1 check-local-headers: ignore c++ headers 2012-11-01 18:28:24 +01:00
Chris Metcalf
9f45bfe790 check-local-headers: Ignore <arch> headers
The tile architecture's Linux port installs headers in an
<arch> directory; these headers are in part shared with glibc.
Ignore these headers for check-local-headers like we ignore
all the other Linux headers.
2012-10-30 14:18:17 -04:00
Joseph Myers
df381762dc Add cross-test-ssh.sh. 2012-10-25 19:17:45 +00:00
Joseph Myers
23c31b7695 Update config.guess and config.sub. 2012-10-05 16:00:07 +00:00
Alexandre Oliva
b6c5ec0728 * scripts/check-local-headers.sh: Exclude sys/sdt.h and
sys/sdt-config.h.
2012-10-02 23:40:52 -03:00
H.J. Lu
3d9b46b350 Add "shopt -s nullglob" to check-local-headers.sh 2012-09-07 14:41:14 -07:00
H.J. Lu
d22e28b070 Use LD_SO to set $ld_so_name/$ld_so_version 2012-08-29 06:45:37 -07:00
Joseph Myers
0f48659e36 Move localplt baselines to sysdeps directories. 2012-07-20 19:20:34 +00:00
Joseph Myers
bd29910a8a Move c++-types baselines to sysdeps directories. 2012-07-17 14:44:06 +00:00
Carlos O'Donell
feca660a79 Regenerate libc.pot.
Enhance scripts/list-sources.sh to search glibc-ports
for translatable strings.
2012-06-21 07:58:27 -07:00
Richard Henderson
63f1549e0e alpha: Handle ST_OTHER in objdump output
* scripts/abilist.awk: Accept 8 fields.  Handle Alpha functions
        marked to avoid plt entry.
2012-05-30 11:05:51 -07:00
H.J. Lu
0ab0291b84 Convert WORDSIZE[32|64]/ld entries to abi-variants 2012-05-30 08:33:26 -07:00
H.J. Lu
cfba4fda12 Add x32 C++ type data 2012-05-24 11:18:30 -07:00
David S. Miller
6693d69429 Adjust sparc localplt entries.
* scripts/data/localplt-sparc-linux-gnu.data: Add '?' markers
	to quad soft-float symbols whose references are compiler
	generated.
	* scripts/data/localplt-sparc64-linux-gnu.data: Likewise.
2012-05-06 20:15:50 -07:00
Andreas Jaeger
7ac30cc5f0 Move FAQ to wiki
The FAQ is now at http://sourceware.org/glibc/wiki/FAQ and not
anymore part of the repository.
2012-05-03 09:46:57 +02:00
Roland McGrath
90fe4186b3 Do check-localplt test using readelf rather than a build-time C program. 2012-05-01 13:30:35 -07:00
Roland McGrath
82397ed6ea Do check-execstack test using readelf rather than a build-time C program. 2012-05-01 13:27:52 -07:00
Roland McGrath
82a79e7d18 Do check-textrel test using readelf rather than a build-time C program. 2012-05-01 13:27:11 -07:00