The feature was removed from the bots a while ago. It was
superseeded by the flaky-test detection which reruns tests.
Remaining tests still marked as flaky most certainly pass
since a while.
Referencing all the bugs whose expectations lines get
removed by this.
BUG=v8:3838,v8:3525,v8:3125
LOG=n
Review URL: https://codereview.chromium.org/1802983002
Cr-Commit-Position: refs/heads/master@{#34804}
test262 "negative" test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.
Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class.
It invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.
Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.
BUG=v8:4803
R=machenbach
LOG=Y
Review URL: https://codereview.chromium.org/1766503002
Cr-Commit-Position: refs/heads/master@{#34763}
Changes include:
- better test coverage for builds with snapshot
- write snapshot blobs to buffer instead of test serialization files
- renamed tests
R=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1777213002
Cr-Commit-Position: refs/heads/master@{#34657}
This adds sanitizer-coverage compilation, test-runner
features and post processing.
Sanitizer coverage is expected to be used together with
asan.
During test runner execution, the produced sancov files
are disambiguated and match the pattern:
<executable name>.test.<test id>.sancov.
Two additional scripts are added for merging raw sancov
files and for generating json data containing all
instrumented lines + all covered lines from merged sancov
files. Both scripts use multiprocessing for speed.
The json data will later be uploaded to google storage
for further use, e.g. to show coverage data in rietveld.
Sancov documentation:
http://clang.llvm.org/docs/SanitizerCoverage.html
BUG=chromium:568949
LOG=n
NOTRY=true
TEST=python -m unittest sancov_formatter_test
TEST=python -m unittest sancov_merger_test
Review URL: https://codereview.chromium.org/1737263003
Cr-Commit-Position: refs/heads/master@{#34578}
This returns a proper test result (failure) when files are
missing for setting up the command.
Review URL: https://codereview.chromium.org/1696713002
Cr-Commit-Position: refs/heads/master@{#33946}
The current support for try-catch in the interpreter can handle most of
the cases appearing in our test suite. Also the flag in question did not
detect try-finally constructs. This removes the flag and instead extends
the test expectations.
R=rmcilroy@chromium.org
BUG=v8:4674
LOG=n
Review URL: https://codereview.chromium.org/1631593003
Cr-Commit-Position: refs/heads/master@{#33494}
This removes the above flag definition. The flag is no longer needed as
the default implementation is more than capable of faking presence of
handling of try-catch and try-finally constructs by now.
R=rmcilroy@chromium.org
BUG=v8:4674
LOG=n
Review URL: https://codereview.chromium.org/1603063003
Cr-Commit-Position: refs/heads/master@{#33402}
Now that we support eval in Ignition, remove the fallback for eval checks
and make the flag only fallback on catch blocks.
BUG=v8:4280,v8:4676
LOG=N
Review URL: https://codereview.chromium.org/1595223004
Cr-Commit-Position: refs/heads/master@{#33384}
Adds --turbo to the set of flags run with ignition (making the
configuration pure Ignition+TurboFan except for fallbacks to
full-codegen for catch / eval). Also changes the default
--ignition-filter to allow everything, rather than omit everything.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1537433003
Cr-Commit-Position: refs/heads/master@{#33090}
This will allow moving the test outcome check beyond the
multi-process boundary in a follow up. It'll allow wrapping
more complex test jobs like predicable mode on the multi-
process side, which will make the code easier to maintain.
BUG=
Review URL: https://codereview.chromium.org/1469833002
Cr-Commit-Position: refs/heads/master@{#32373}
This will make sure that message tests cover both the parser and preparser
paths, just as we do for parsing-related cctests.
BUG=v8:4372
LOG=n
Review URL: https://codereview.chromium.org/1469383004
Cr-Commit-Position: refs/heads/master@{#32307}
This loads all test suites and status files to catch subtle
syntax errors. It also checks basic status file integrity
and common mistakes.
NOTRY=true
Review URL: https://codereview.chromium.org/1475663002
Cr-Commit-Position: refs/heads/master@{#32271}
The feature sometimes prevents subsequent swarming bots to
delete the work directory on windows.
The data file is not kept on swarming bots anyways, therefore
this switches off the feature completely.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1468933002
Cr-Commit-Position: refs/heads/master@{#32168}
Adds a blacklist of tests which are currently unsupported or broken in Ignition to
the mjsunit and test262 test status.
Also removes --ignition-script-filter flag, and adds a
--ignition_fallback_on_eval_and_catch flag which fallsback to fullcodegen for
functions which call eval or contain a catch block.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1420963009
Cr-Commit-Position: refs/heads/master@{#31864}
The flake detection is done on the infra-side according to
the contents of the json test results. We don't want the
runner to fail after flakes.
This was controlled on the infra side by accepting any exit
codes so far. After the swarming switch, this is more
difficult, because the runner is wrapped by the swarming
collect script. There, failing exit codes can mean many
things, including network failures. Therefore, we now
force exit code 0 with test failures if those failures
are reported in the formal test results json.
The infrastructure will take care of reporting the flakes
and failures accordingly.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1416373005
Cr-Commit-Position: refs/heads/master@{#31740}
Adds an ignition variant to the test runner and adds support to test262 for
filtering such that only test scripts (not the test harness) get run by
the interpreter.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1379093002
Cr-Commit-Position: refs/heads/master@{#31123}
Reason for revert:
Prime suspect in breakage of V8 Linux -- no snap
Original issue's description:
> [swarming] Isolate v8 testing.
>
> Add gyp support and isolates for default test suites.
> Add two default isolates, one (default) for using the
> test suite collection we call "default" on the bots. One
> (developer_default) for also supporting the way developers
> call the driver (i.e. without argument, which includes
> the unittests).
>
> BUG=chromium:535160
> LOG=n
>
> Committed: https://crrev.com/9bd83f58f29ab0c7c5b71b00bcb1df3a9e641f05
> Cr-Commit-Position: refs/heads/master@{#31081}
TBR=tandrii@chromium.org,jochen@chromium.org,maruel@chromium.org,machenbach@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=chromium:535160
Review URL: https://codereview.chromium.org/1370993008
Cr-Commit-Position: refs/heads/master@{#31084}
Add gyp support and isolates for default test suites.
Add two default isolates, one (default) for using the
test suite collection we call "default" on the bots. One
(developer_default) for also supporting the way developers
call the driver (i.e. without argument, which includes
the unittests).
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1380593002
Cr-Commit-Position: refs/heads/master@{#31081}
Swarming won't support an incremental data file. By just
sorting the lowest hanging fruits to the beginning we
already get a big bang for the buck (>80% of the improvement
we get otherwise).
This will require semi-regular manual updates of the
slowest tests.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1385443002
Cr-Commit-Position: refs/heads/master@{#31061}
[run-tests.py]
- adding more detailed information about the flags
- show more detailed error message on failing Popen commands
BUG=
Review URL: https://codereview.chromium.org/1369343002
Cr-Commit-Position: refs/heads/master@{#30997}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Original review: https://codereview.chromium.org/1348653003/
This reverts commit 5f44a91059.
NOTRY=true
Review URL: https://codereview.chromium.org/1356613002
Cr-Commit-Position: refs/heads/master@{#30798}
Reason for revert:
mozilla tests are failing on Windows
Original issue's description:
> [test] Fix cctest path separators on Windows
>
> Now run-tests.py understands "suite/foo/bar" with forward slashes for
> command-line test selection on all test suites on all platforms.
>
> Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
> that behavior is sacrificed here in favor of unification. For the cctest
> suite, OTOH, it wasn't possible on Windows to select specific tests at all.
>
> Committed: https://crrev.com/b36cfdb39ae648b49a1396c4f669df9b1f57996c
> Cr-Commit-Position: refs/heads/master@{#30794}
TBR=machenbach@google.com,machenbach@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1349163002
Cr-Commit-Position: refs/heads/master@{#30795}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Review URL: https://codereview.chromium.org/1348653003
Cr-Commit-Position: refs/heads/master@{#30794}
This information can be used on the buildbot side to only
rebuild the failing target for bisection.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1313353006
Cr-Commit-Position: refs/heads/master@{#30617}
Use the same path separator for test cases on the cmd-line
on all platforms as in testcase.GetLabel, which is used to
report failures.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1281453003
Cr-Commit-Position: refs/heads/master@{#30045}
This allows variants to be named on test failures (follow
up) and then to be used in the test runner for a repro.
This also speeds up variant iteration for test262 and fixes
a bug with variants for benchmarks.
BUG=chromium:511215
NOTREECHECKS=true
LOG=n
Review URL: https://codereview.chromium.org/1245623005
Cr-Commit-Position: refs/heads/master@{#29899}
There are many test names in the v8 code base that prefix
others, which makes it hard to only run those tests.
BUG=chromium:511215
LOG=n
NOTRY=true
Review URL: https://codereview.chromium.org/1251363002
Cr-Commit-Position: refs/heads/master@{#29809}
