glibc/scripts
Joseph Myers d6fe5e582d Do not terminate default test runs on test failure.
This patch is an updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00198.html> and
<https://sourceware.org/ml/libc-alpha/2014-03/msg00180.html>.

Normal practice for software testsuites is that rather than
terminating immediately when a test fails, they continue running and
report at the end on how many tests passed or failed.

The principle behind the glibc testsuite stopping on failure was
probably that the expected state is no failures and so any failure
indicates a problem such as miscompilation.  In practice, while this
is fairly close to true for native testing on x86_64 and x86 (kernel
bugs and race conditions can still cause intermittent failures), it's
less likely to be the case on other platforms, and so people testing
glibc run the testsuite with "make -k" and then examine the logs to
determine whether the failures are what they expect to fail on that
platform, possibly with some automation for the comparison.

This patch switches the glibc testsuite to the normal convention of
not stopping on failure - unless you use stop-on-test-failure=y, in
which case it behaves essentially as it did before (and does not
generate overall test summaries on failure).  Instead, the summary
tests.sum may contain tests that FAILed.  At the end of the test run,
any FAIL or ERROR lines from tests.sum are printed, and then it exits
with error status if there were any such lines.  In addition, build
failures will also cause the test run to stop - this has the
justification that those *do* indicate serious problems that should be
promptly fixed and aren't generally hard to fix (but apart from that,
avoiding the build stopping on those failures seems harder).

Note that unlike the previous patches in this series, this *does*
require people with automation around testing glibc to change their
processes - either to start using tests.sum / xtests.sum to track
failures and compare them with expectations (with or without also
using "make -k" and examining "make" logs to identify build failures),
or else to use stop-on-test-failure=y and ignore the new tests.sum /
xtests.sum mechanism.  (If all you check is the exit status from "make
check", no changes are needed unless you want to avoid test runs
continuing after the first failure.)

Tested x86_64.

	* scripts/evaluate-test.sh: Handle fourth argument to determine
	whether test run should stop on failure.
	* Makeconfig (stop-on-test-failure): New variable.
	(evaluate-test): Pass fourth argument to evaluate-test.sh based on
	$(stop-on-test-failure).
	* Makefile (tests): Give a summary of results from testing and
	exit with failure status if they include an ERROR or FAIL.
	(xtests): Likewise.
	* manual/install.texi (Configuring and compiling): Mention
	stop-on-test-failure=y.
	* INSTALL: Regenerated.
2014-03-14 21:02:40 +00:00
..
abi-versions.awk
abilist.awk Properly handle indirect functions in ABI check on powerpc64 2012-11-28 16:19:08 +01:00
begin-end-check.pl
bench.pl Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
check-c++-types.sh Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
check-execstack.awk Do check-execstack test using readelf rather than a build-time C program. 2012-05-01 13:27:52 -07:00
check-local-headers.sh Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
check-localplt.awk Add flexibility to localplt-*.data files, using an awk script rather than diff to check the results. 2012-04-24 14:48:15 -07:00
check-textrel.awk Do check-textrel test using readelf rather than a build-time C program. 2012-05-01 13:27:11 -07:00
config-uname.sh
config.guess Update texinfo.tex, config.guess, config.sub from upstream. 2013-12-19 17:36:10 +00:00
config.sub Update texinfo.tex, config.guess, config.sub from upstream. 2013-12-19 17:36:10 +00:00
cpp
cross-test-ssh.sh Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
documented.sh
evaluate-test.sh Do not terminate default test runs on test failure. 2014-03-14 21:02:40 +00:00
firstversions.awk Fix bug in firstversions.awk version range handling. 2012-01-28 12:02:44 -05:00
gen-as-const.awk
gen-libc-abis
gen-sorted.awk Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
install-sh Update miscellaneous scripts from upstream. 2013-06-06 19:02:09 +00:00
lib-names.awk Convert WORDSIZE[32|64]/ld entries to abi-variants 2012-05-30 08:33:26 -07:00
list-sources.sh Regenerate libc.pot. 2012-06-21 07:58:27 -07:00
localplt.awk Do check-localplt test using readelf rather than a build-time C program. 2012-05-01 13:30:35 -07:00
merge-test-results.sh Generate overall summary of test results. 2014-03-07 03:25:57 +00:00
mkinstalldirs Update miscellaneous scripts from upstream. 2013-06-06 19:02:09 +00:00
move-if-change Update miscellaneous scripts from upstream. 2013-06-06 19:02:09 +00:00
output-format.sed
rellns-sh Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00
rpm2dynsym.sh
soversions.awk Convert WORDSIZE[32|64]/ld entries to abi-variants 2012-05-30 08:33:26 -07:00
sysd-rules.awk Sort sysd-rules-patterns by descending pattern length. 2013-06-17 09:55:49 -07:00
test-installation.pl Update remaining copyright dates 2014-01-01 22:02:55 +10:00
update-copyrights scripts/update-copyrights: adjust configure input file suffix 2014-01-01 22:10:54 +10:00
versionlist.awk Get rid of Versions.def source file 2014-03-14 11:39:56 -07:00
versions.awk Update copyright notices with scripts/update-copyrights 2014-01-01 22:00:23 +10:00