This patch changes the test262 infrastructure to pass individual flags,
specified in the status file, for tests for experimental features, rather
than passing --harmony for all runs. With this change, it should be
easier to run test262 tests in automation when developing new features.
The new workflow would be, when adding a flag, include the flag in the
test expectations file, and when removing the flag, remove the lines from
the test expectations file. This way, the status file does not have to
change when staging or unstaging, and you get the benefit of the automated
tests before staging starts.
R=adamk
CQ_INCLUDE_TRYBOTS=master.tryserver.v8:v8_linux_noi18n_rel_ng
Review-Url: https://codereview.chromium.org/2601393002
Cr-Commit-Position: refs/heads/master@{#42249}
This patch provides improved infrastructure for developing test262 tests
together with V8. It has three parts:
- The test262 test runner is updated to look for local versions of tests
in the /test/test262/local-tests directory, which mirrors
/test/test262/data. Additional tests can be added there and are run
together with tests from upstream. Upstream tests can be locally
updated by using the same name in local-tests; if a same-named test
exists, then only the local version will be run. The local-tests
directory is in the V8 repository, unlike the contents of the data
directory, so tests can be added in the same patch as something else.
- The tool /test/test262/upstream-local-tests.sh is added to create
a patch against the test262 respository based on a patch which changes
the local-tests directory.
- The tool /test/test262/prune-local-tests.sh is added to remove
redundant local tests on a test262 roll.
See design doc:
https://docs.google.com/document/d/16bj7AIDgZLv4WOsUEzQ5NzcEN9_xo095e88Pz8FC5rA/edit
Review-Url: https://codereview.chromium.org/2611793002
Cr-Commit-Position: refs/heads/master@{#42117}
- Make testcfg.py ignore files ending in "_FIXTURE.js", which are not supposed
to be tested standalone but only as imports to other modules.
- Refine test262.status to match where we are with the modules implementation.
BUG=v8:1569
R=adamk@chromium.org
Review-Url: https://codereview.chromium.org/2400713004
Cr-Commit-Position: refs/heads/master@{#40134}
The _ParseException function in testcfg.py made incorrect assumptions about how
exceptions are printed (I believe it expected a trace like that produced by
Error).
R=littledan@chromium.org
BUG=v8:5136
Review-Url: https://codereview.chromium.org/2386103010
Cr-Commit-Position: refs/heads/master@{#40078}
This removes the test-download method for all but
promises-aplus. They all only contain legacy code for
deleting old archives.
Only test262 needs to prepare sources on swarming, which is
moved to a new method, called unconditionally.
All references to --download-data and --download-data-only
in the infrastructure can be removed after this.
BUG=
Review-Url: https://codereview.chromium.org/2227613002
Cr-Commit-Position: refs/heads/master@{#38439}
TC-39 recently decided to remove the Python-based testing harness from the
Test262 project [1]. The code has been duplicated in a standalone project;
update V8's dependencies to fetch from that new location. This is based on
an earlier patch by Mike Pennisi.
[1] 2b9722db9b/es7/2016-05/may-25.md
BUG=v8:5078
Review-Url: https://codereview.chromium.org/2131743002
Cr-Commit-Position: refs/heads/master@{#37985}
New incoming test262 tests check what happens on detached ("neutered")
ArrayBuffers. This patch makes the test262 infrastructure define
detaching an ArrayBuffer in terms of %ArrayBufferNeuter, passing the
--allow-natives-syntax flag, when it is needed.
BUG=v8:4193
LOG=N
R=adamk,machenbach
Review URL: https://codereview.chromium.org/1897203003
Cr-Commit-Position: refs/heads/master@{#35676}
This causes an incoming test262 test to pass, as part of the next
test262 roll.
R=adamk,machenbach
BUG=v8:1569
LOG=N
Review URL: https://codereview.chromium.org/1896293003
Cr-Commit-Position: refs/heads/master@{#35667}
test262 "negative" test expectations list which exception is thrown. The ES2017
draft specification is very specific about which exception class is thrown
from which path, and V8 works hard to be correct with respect to that spec.
Previously, the test262 test runner would accept any nonzero status code,
such as from a crash, or a FAIL printed out, for a negative test. This
patch makes negative tests check for the right answer using a quick-and-dirty
parsing of the exception printing from d8 to find the exception class.
It invokes d8 in a way to get a status code of 0 from thrown exceptions
so that 'negative' tests aren't actually implemented by negating the output.
Amazingly, this didn't catch any test262 failures, but I verified the extra
checking interactively by changing a negative test to expect a different type
and saw it fail.
BUG=v8:4803
R=machenbach
LOG=Y
Review URL: https://codereview.chromium.org/1766503002
Cr-Commit-Position: refs/heads/master@{#34763}
This experimentally implements taring/untaring the test data
for test262 on the v8-side before test isolation and when
running the tests.
It archives on demand only if the tar is outdated compared
to the contained files. This comes with a cost of ~1s extra
to run gyp on linux and ~6s extra on windows. Ninja is
lightning fast afterwards in detecting changes. Also, we
archive only when test_isolation_mode is set and when
the test262_run target is required.
The archiving itself costs ~30s on all platforms. But as the
files will change seldom this shouldn't have a big impact.
Extraction on the test runner side is below 2s on mac and
linux. The speedup is enormous. Around 5 minutes were spent
on download on swarming slaves before, which is now only
a few seconds. So total test time for release (no variants),
e.g. goes from 8 to 3 minutes.
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1713993002
Cr-Commit-Position: refs/heads/master@{#34155}
Adds a blacklist of tests which are currently unsupported or broken in Ignition to
the mjsunit and test262 test status.
Also removes --ignition-script-filter flag, and adds a
--ignition_fallback_on_eval_and_catch flag which fallsback to fullcodegen for
functions which call eval or contain a catch block.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1420963009
Cr-Commit-Position: refs/heads/master@{#31864}
This patch brings test262 up to the latest version on GitHub. Some
highlights from new failures are:
- Reflect
- Default parameters
- Case conversion edge cases
R=adamk
Review URL: https://codereview.chromium.org/1391793002
Cr-Commit-Position: refs/heads/master@{#31165}
Add a flag to explicitly filter scripts in ignition and use it for the test262
variant. The previous approach of overloading ignition-filter meant that only
top-level code was getting compiled through ignition.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1396493002
Cr-Commit-Position: refs/heads/master@{#31164}
This makes it explicit when the --ignition-filter pattern should be
applied to the script name instead of the function name by using a
proper "s:{name}" pattern. It also hardcodes it to be a prefix match
instead of an exact match, because that is all we need for test262.
R=rmcilroy@chromium.org
Review URL: https://codereview.chromium.org/1389353002
Cr-Commit-Position: refs/heads/master@{#31153}
Adds an ignition variant to the test runner and adds support to test262 for
filtering such that only test scripts (not the test harness) get run by
the interpreter.
BUG=v8:4280
LOG=N
Review URL: https://codereview.chromium.org/1379093002
Cr-Commit-Position: refs/heads/master@{#31123}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Original review: https://codereview.chromium.org/1348653003/
This reverts commit 5f44a91059.
NOTRY=true
Review URL: https://codereview.chromium.org/1356613002
Cr-Commit-Position: refs/heads/master@{#30798}
Reason for revert:
mozilla tests are failing on Windows
Original issue's description:
> [test] Fix cctest path separators on Windows
>
> Now run-tests.py understands "suite/foo/bar" with forward slashes for
> command-line test selection on all test suites on all platforms.
>
> Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
> that behavior is sacrificed here in favor of unification. For the cctest
> suite, OTOH, it wasn't possible on Windows to select specific tests at all.
>
> Committed: https://crrev.com/b36cfdb39ae648b49a1396c4f669df9b1f57996c
> Cr-Commit-Position: refs/heads/master@{#30794}
TBR=machenbach@google.com,machenbach@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1349163002
Cr-Commit-Position: refs/heads/master@{#30795}
Now run-tests.py understands "suite/foo/bar" with forward slashes for
command-line test selection on all test suites on all platforms.
Previously, file-based suites like mjsunit also accepted "mjsunit/foo\bar";
that behavior is sacrificed here in favor of unification. For the cctest
suite, OTOH, it wasn't possible on Windows to select specific tests at all.
Review URL: https://codereview.chromium.org/1348653003
Cr-Commit-Position: refs/heads/master@{#30794}
