Using test generators meant that we had to remove the progress indicator since
the total number of tests weren't known before-hand.
This CL implements a progress indicator using test number estimations.
cctest and unittests progress indicator is accurate, however estimating
means the progress will terminate over 100% in big test suites and sometimes
under 100%.
R=machenbach@chromium.org
CC=sergiyb@chromium.org,yangguo@chromium.org
Bug: v8:8769
Change-Id: I40ca5b40f9b1223376d33707f0945900ea98cea3
Reviewed-on: https://chromium-review.googlesource.com/c/1460471
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59538}
This is a reland of 81eec150f6
Original change's description:
> Reland "[test] refactor testsuite configuration"
>
> This is a reland of 7f92ad0ab6
>
> Original change's description:
> > [test] refactor testsuite configuration
> >
> > Every testsuite configuration consist of at least 30% code duplication.
> >
> > The code age ranges from 10 years old to 5 years old. Implementing anything that
> > touches the testsuite code becomes a technical fight to the death.
> >
> > This CL removes all the duplication by refactoring the common functionality.
> >
> > This CL contains structural changes without any logical changes % small bug
> > fixes.
> >
> > R=machenbach@chromium.org
> > CC=yangguo@chromium.org,sergiyb@chromium.org
> >
> > Bug: v8:8174, v8:8769
> > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#59361}
>
> Bug: v8:8174, v8:8769
> Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
> Reviewed-on: https://chromium-review.googlesource.com/c/1454485
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Tamer Tas <tmrts@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59369}
Bug: v8:8174, v8:8790
Change-Id: I38ab9d37bca76057441a970f26e2102e4387a857
Reviewed-on: https://chromium-review.googlesource.com/c/1454724
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59387}
This reverts commit 81eec150f6.
Reason for revert: windows mozilla test failures
Original change's description:
> Reland "[test] refactor testsuite configuration"
>
> This is a reland of 7f92ad0ab6
>
> Original change's description:
> > [test] refactor testsuite configuration
> >
> > Every testsuite configuration consist of at least 30% code duplication.
> >
> > The code age ranges from 10 years old to 5 years old. Implementing anything that
> > touches the testsuite code becomes a technical fight to the death.
> >
> > This CL removes all the duplication by refactoring the common functionality.
> >
> > This CL contains structural changes without any logical changes % small bug
> > fixes.
> >
> > R=machenbach@chromium.org
> > CC=yangguo@chromium.org,sergiyb@chromium.org
> >
> > Bug: v8:8174, v8:8769
> > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#59361}
>
> Bug: v8:8174, v8:8769
> Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
> Reviewed-on: https://chromium-review.googlesource.com/c/1454485
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Tamer Tas <tmrts@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59369}
TBR=machenbach@chromium.org,sergiyb@chromium.org,tmrts@chromium.org
Change-Id: I8f5650b5f46be299c004e2fa8b708fa2c17a4dc2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:8174, v8:8769
Reviewed-on: https://chromium-review.googlesource.com/c/1454607
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59370}
This is a reland of 7f92ad0ab6
Original change's description:
> [test] refactor testsuite configuration
>
> Every testsuite configuration consist of at least 30% code duplication.
>
> The code age ranges from 10 years old to 5 years old. Implementing anything that
> touches the testsuite code becomes a technical fight to the death.
>
> This CL removes all the duplication by refactoring the common functionality.
>
> This CL contains structural changes without any logical changes % small bug
> fixes.
>
> R=machenbach@chromium.org
> CC=yangguo@chromium.org,sergiyb@chromium.org
>
> Bug: v8:8174, v8:8769
> Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59361}
Bug: v8:8174, v8:8769
Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
Reviewed-on: https://chromium-review.googlesource.com/c/1454485
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59369}
This reverts commit 7f92ad0ab6.
Reason for revert: https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Win32/19148
Original change's description:
> [test] refactor testsuite configuration
>
> Every testsuite configuration consist of at least 30% code duplication.
>
> The code age ranges from 10 years old to 5 years old. Implementing anything that
> touches the testsuite code becomes a technical fight to the death.
>
> This CL removes all the duplication by refactoring the common functionality.
>
> This CL contains structural changes without any logical changes % small bug
> fixes.
>
> R=machenbach@chromium.org
> CC=yangguo@chromium.org,sergiyb@chromium.org
>
> Bug: v8:8174, v8:8769
> Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59361}
TBR=machenbach@chromium.org,sergiyb@chromium.org,tmrts@chromium.org,v8-reviews@chromium.org
Change-Id: I473f0d4c6b9c0239923b8c03699dbc38b7f85030
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:8174, v8:8769
Reviewed-on: https://chromium-review.googlesource.com/c/1454599
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59362}
Every testsuite configuration consist of at least 30% code duplication.
The code age ranges from 10 years old to 5 years old. Implementing anything that
touches the testsuite code becomes a technical fight to the death.
This CL removes all the duplication by refactoring the common functionality.
This CL contains structural changes without any logical changes % small bug
fixes.
R=machenbach@chromium.orgCC=yangguo@chromium.org,sergiyb@chromium.org
Bug: v8:8174, v8:8769
Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
Reviewed-on: https://chromium-review.googlesource.com/c/1445881
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59361}
This shares logic for finding additional resources in JS source code.
Previously the logic was implemented for mjsunit, now it will be used
across all d8-based test cases.
This'll enable adding those test suites for Android testing.
Bug: chromium:866862
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I7c89ba141483aaf692a03c0e168edb61bbb7b010
Reviewed-on: https://chromium-review.googlesource.com/1193873
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Georg Neis <neis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55503}
This is a reland of 0db74d4974.
Original change's description:
> [test] Random seed processor
>
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
>
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}
Bug: v8:6917
Change-Id: I1ea376a4abffce5ab65f4834ea7e6d6011765ffa
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894204
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50978}
This reverts commit 0db74d4974.
Reason for revert: https://chromium-swarm.appspot.com/task?id=3b609f9976bac610&refresh=10&show_raw=1
Original change's description:
> [test] Random seed processor
>
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
>
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}
TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com
Change-Id: I2d96ea328cda2d09b01ff455e47c77d567fafe00
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894522
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50970}
1. --total-timeout-sec now available for ./run-tests.py. It can be
useful with infinite seed stressing
2. random seed dropped from the context. Now JSON progress indicator
gets it from the list of command args.
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
Reviewed-on: https://chromium-review.googlesource.com/888524
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50964}
- All testcase/testsuite/variant generator subclasses renamed to
just TestCase/TestSuite/VariantGenerator since they're private
implementation.
- All `testcase` variables renamed to `test` to not conflict with
a module name.
- No more two statements in the same line.
- Removed some unused testsuite methods.
Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I7710f3419f738a5f9ddca73765dd2cad2e35b952
Reviewed-on: https://chromium-review.googlesource.com/823964
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50076}
- create testcase subclass for each test
- move get_command, get_source from suite to test
- promises-aplus tests are broken
- moving expected outcomes etc. is still in progress
Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I006e64ecf51dda95c41dff88c68b24f17a638566
Reviewed-on: https://chromium-review.googlesource.com/798331
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50065}
Some refactor moved from
https://chromium-review.googlesource.com/c/v8/v8/+/798331.
Bug: v8:6917
Change-Id: I8cae6cfca7a0d7d8e234052c0ab0bfe252355e60
Reviewed-on: https://chromium-review.googlesource.com/819550
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50020}
The status-file flags and the flags from the test case's source code
must always overwrite extra flags set by bots.
Bug: v8:6924
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I0e2aabb69da7cfb8ba6c1c79bd3851462071a6ac
Reviewed-on: https://chromium-review.googlesource.com/732656
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49001}
This prepares for pulling chromium's build as dependency for
gn. After this, the files in build and gypfiles need to stay
in sync until chromium is updated.
BUG=chromium:474921
LOG=n
Review-Url: https://codereview.chromium.org/1848553003
Cr-Commit-Position: refs/heads/master@{#35898}
This moves the last remaining JS file based tests out of the "preparser"
suite. The tests in question all are expected to parse normally and not
throw any exception. This also deprecates the ability of the test suite
to run anything else outside Python templated tests.
R=adamk@chromium.org
TEST=preparser
Review URL: https://codereview.chromium.org/1782173005
Cr-Commit-Position: refs/heads/master@{#34753}
This converts another test case that is expected to throw a TypeError
but no SyntaxError to have better test coverage (exact message is being
checked now).
R=machenbach@chromium.org
TEST=message,preparser
Review URL: https://codereview.chromium.org/1786623002
Cr-Commit-Position: refs/heads/master@{#34715}
This converts existing "preparser" tests that expect a certain exception
message to be produced into "message" tests. Thereby we get much better
coverage because the former test suite degraded by now to just check
whether each test case threw or not, the exception message was not being
checked at all.
This also deprecates the ability of "preparser" to specify that single
test cases based on JS files are expected to throw, "messages" is far
superior, use that test suite instead.
R=machenbach@chromium.org
TEST=message,preparser
Review URL: https://codereview.chromium.org/1784013003
Cr-Commit-Position: refs/heads/master@{#34713}
Reason for revert:
Prime suspect in breakage of V8 Linux -- no snap
Original issue's description:
> [swarming] Isolate v8 testing.
>
> Add gyp support and isolates for default test suites.
> Add two default isolates, one (default) for using the
> test suite collection we call "default" on the bots. One
> (developer_default) for also supporting the way developers
> call the driver (i.e. without argument, which includes
> the unittests).
>
> BUG=chromium:535160
> LOG=n
>
> Committed: https://crrev.com/9bd83f58f29ab0c7c5b71b00bcb1df3a9e641f05
> Cr-Commit-Position: refs/heads/master@{#31081}
TBR=tandrii@chromium.org,jochen@chromium.org,maruel@chromium.org,machenbach@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=chromium:535160
Review URL: https://codereview.chromium.org/1370993008
Cr-Commit-Position: refs/heads/master@{#31084}
Add gyp support and isolates for default test suites.
Add two default isolates, one (default) for using the
test suite collection we call "default" on the bots. One
(developer_default) for also supporting the way developers
call the driver (i.e. without argument, which includes
the unittests).
BUG=chromium:535160
LOG=n
Review URL: https://codereview.chromium.org/1380593002
Cr-Commit-Position: refs/heads/master@{#31081}
This allows variants to be named on test failures (follow
up) and then to be used in the test runner for a repro.
This also speeds up variant iteration for test262 and fixes
a bug with variants for benchmarks.
BUG=chromium:511215
NOTREECHECKS=true
LOG=n
Review URL: https://codereview.chromium.org/1245623005
Cr-Commit-Position: refs/heads/master@{#29899}
