Commit Graph

26 Commits

Author SHA1 Message Date
Liviu Rau
b3477fdd01 [test] Refactor testrunner (4)
- Removed duplication and unnecessary indirection from all suites testcfgs.
 - Introduced a more comprehensive context to cover both command context and other platform specific concerns.
 - Propagated above context to TestLoader to allow for test counting command execution on all platforms.
 - Wrapped original pool with another class to give it a new interface and allow injecting different implementations in the future.
 - Consolidated progress indicators under a single processor in the pipeline.
 - Consolidated result retention requirements calculation outside of pipeline chain.
 - Refactored LoaderProc and got it under tests.
 - Added some more tests for the standard runner.
 - Extracted BuildConfig class.


Bug: v8:12785
Change-Id: I87be040e91f792a983662bb5a10d55b36a14ea7f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3701595
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81770}
2022-07-18 09:52:24 +00:00
Mathias Bynens
5c2c5ce133 [test] Remove module pragma in favor of .mjs
d8 treats files with the .mjs extension as modules instead of
classic scripts. Thus, the `// MODULE` pragma and its corresponding
logic in test runners can be removed in favor of explicitly adding
the extension.

Bug: v8:7950, v8:9395, v8:9406
Also-By: tmrts@chromium.org
Change-Id: Ic74328dc5c5f176bb4bdf6d74bdd4d3966279ba5
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1675958
Commit-Queue: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Auto-Submit: Mathias Bynens <mathias@chromium.org>
Cr-Commit-Position: refs/heads/master@{#62421}
2019-06-27 13:25:48 +00:00
Tamer Tas
df630e67fd Reland "Reland "[test] refactor testsuite configuration""
This is a reland of 81eec150f6

Original change's description:
> Reland "[test] refactor testsuite configuration"
>
> This is a reland of 7f92ad0ab6
>
> Original change's description:
> > [test] refactor testsuite configuration
> >
> > Every testsuite configuration consist of at least 30% code duplication.
> >
> > The code age ranges from 10 years old to 5 years old. Implementing anything that
> > touches the testsuite code becomes a technical fight to the death.
> >
> > This CL removes all the duplication by refactoring the common functionality.
> >
> > This CL contains structural changes without any logical changes % small bug
> > fixes.
> >
> > R=machenbach@chromium.org
> > CC=yangguo@chromium.org,sergiyb@chromium.org
> >
> > Bug: v8:8174, v8:8769
> > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#59361}
>
> Bug: v8:8174, v8:8769
> Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
> Reviewed-on: https://chromium-review.googlesource.com/c/1454485
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Tamer Tas <tmrts@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59369}

Bug: v8:8174, v8:8790
Change-Id: I38ab9d37bca76057441a970f26e2102e4387a857
Reviewed-on: https://chromium-review.googlesource.com/c/1454724
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59387}
2019-02-06 09:02:09 +00:00
Tamer Tas
ea4412ad33 Revert "Reland "[test] refactor testsuite configuration""
This reverts commit 81eec150f6.

Reason for revert: windows mozilla test failures

Original change's description:
> Reland "[test] refactor testsuite configuration"
> 
> This is a reland of 7f92ad0ab6
> 
> Original change's description:
> > [test] refactor testsuite configuration
> > 
> > Every testsuite configuration consist of at least 30% code duplication.
> > 
> > The code age ranges from 10 years old to 5 years old. Implementing anything that
> > touches the testsuite code becomes a technical fight to the death.
> > 
> > This CL removes all the duplication by refactoring the common functionality.
> > 
> > This CL contains structural changes without any logical changes % small bug
> > fixes.
> > 
> > R=machenbach@chromium.org
> > CC=yangguo@chromium.org,sergiyb@chromium.org
> > 
> > Bug: v8:8174, v8:8769
> > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#59361}
> 
> Bug: v8:8174, v8:8769
> Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
> Reviewed-on: https://chromium-review.googlesource.com/c/1454485
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Tamer Tas <tmrts@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59369}

TBR=machenbach@chromium.org,sergiyb@chromium.org,tmrts@chromium.org

Change-Id: I8f5650b5f46be299c004e2fa8b708fa2c17a4dc2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:8174, v8:8769
Reviewed-on: https://chromium-review.googlesource.com/c/1454607
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59370}
2019-02-05 15:20:46 +00:00
Tamer Tas
81eec150f6 Reland "[test] refactor testsuite configuration"
This is a reland of 7f92ad0ab6

Original change's description:
> [test] refactor testsuite configuration
> 
> Every testsuite configuration consist of at least 30% code duplication.
> 
> The code age ranges from 10 years old to 5 years old. Implementing anything that
> touches the testsuite code becomes a technical fight to the death.
> 
> This CL removes all the duplication by refactoring the common functionality.
> 
> This CL contains structural changes without any logical changes % small bug
> fixes.
> 
> R=machenbach@chromium.org
> CC=yangguo@chromium.org,sergiyb@chromium.org
> 
> Bug: v8:8174, v8:8769
> Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59361}

Bug: v8:8174, v8:8769
Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
Reviewed-on: https://chromium-review.googlesource.com/c/1454485
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59369}
2019-02-05 14:53:05 +00:00
Tamer Tas
97068800fe Revert "[test] refactor testsuite configuration"
This reverts commit 7f92ad0ab6.

Reason for revert: https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Win32/19148

Original change's description:
> [test] refactor testsuite configuration
>
> Every testsuite configuration consist of at least 30% code duplication.
>
> The code age ranges from 10 years old to 5 years old. Implementing anything that
> touches the testsuite code becomes a technical fight to the death.
>
> This CL removes all the duplication by refactoring the common functionality.
>
> This CL contains structural changes without any logical changes % small bug
> fixes.
>
> R=​machenbach@chromium.org
> CC=​yangguo@chromium.org,sergiyb@chromium.org
>
> Bug: v8:8174, v8:8769
> Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
> Reviewed-on: https://chromium-review.googlesource.com/c/1445881
> Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59361}

TBR=machenbach@chromium.org,sergiyb@chromium.org,tmrts@chromium.org,v8-reviews@chromium.org

Change-Id: I473f0d4c6b9c0239923b8c03699dbc38b7f85030
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:8174, v8:8769
Reviewed-on: https://chromium-review.googlesource.com/c/1454599
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59362}
2019-02-05 12:06:18 +00:00
Tamer Tas
7f92ad0ab6 [test] refactor testsuite configuration
Every testsuite configuration consist of at least 30% code duplication.

The code age ranges from 10 years old to 5 years old. Implementing anything that
touches the testsuite code becomes a technical fight to the death.

This CL removes all the duplication by refactoring the common functionality.

This CL contains structural changes without any logical changes % small bug
fixes.

R=machenbach@chromium.org
CC=yangguo@chromium.org,sergiyb@chromium.org

Bug: v8:8174, v8:8769
Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
Reviewed-on: https://chromium-review.googlesource.com/c/1445881
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59361}
2019-02-05 11:37:51 +00:00
Michael Achenbach
6206a3e362 [test] Share resource-fetching logic with all d8 test cases
This shares logic for finding additional resources in JS source code.
Previously the logic was implemented for mjsunit, now it will be used
across all d8-based test cases.

This'll enable adding those test suites for Android testing.

Bug: chromium:866862
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I7c89ba141483aaf692a03c0e168edb61bbb7b010
Reviewed-on: https://chromium-review.googlesource.com/1193873
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Georg Neis <neis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55503}
2018-08-29 17:51:43 +00:00
Michal Majewski
804cc9f385 [test] Remove context usage from test processors
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: Id5a7b41502dfd4be43496b1edb958522a8eb0c31
Reviewed-on: https://chromium-review.googlesource.com/895588
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51023}
2018-02-01 10:35:02 +00:00
Michal Majewski
d3083b46ee [test] Move options to test config
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I254d2e545709029346f585b02a9edf91d3f27893
Reviewed-on: https://chromium-review.googlesource.com/893321
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50996}
2018-01-31 13:46:29 +00:00
Michal Majewski
5ede3cb533 Reland "[test] Random seed processor"
This is a reland of 0db74d4974.

Original change's description:
> [test] Random seed processor
> 
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
> 
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}

Bug: v8:6917
Change-Id: I1ea376a4abffce5ab65f4834ea7e6d6011765ffa
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894204
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50978}
2018-01-31 09:53:34 +00:00
Michael Achenbach
ba30988cbc Revert "[test] Random seed processor"
This reverts commit 0db74d4974.

Reason for revert: https://chromium-swarm.appspot.com/task?id=3b609f9976bac610&refresh=10&show_raw=1

Original change's description:
> [test] Random seed processor
> 
> 1. --total-timeout-sec now available for ./run-tests.py. It can be
> useful with infinite seed stressing
> 2. random seed dropped from the context. Now JSON progress indicator
> gets it from the list of command args.
> 
> Bug: v8:6917
> Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
> Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
> Reviewed-on: https://chromium-review.googlesource.com/888524
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#50964}

TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com

Change-Id: I2d96ea328cda2d09b01ff455e47c77d567fafe00
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Reviewed-on: https://chromium-review.googlesource.com/894522
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50970}
2018-01-30 22:05:03 +00:00
Michal Majewski
0db74d4974 [test] Random seed processor
1. --total-timeout-sec now available for ./run-tests.py. It can be
useful with infinite seed stressing
2. random seed dropped from the context. Now JSON progress indicator
gets it from the list of command args.

Bug: v8:6917
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I73e535bc8face9b913c696b8d5e3a246fa231004
Reviewed-on: https://chromium-review.googlesource.com/888524
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50964}
2018-01-30 15:50:49 +00:00
Michal Majewski
03165cb93b [test] Store expected outcomes in the testcase.
Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I69d15801b79bf7aa846582367e9f3037b6612431
Reviewed-on: https://chromium-review.googlesource.com/829033
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50163}
2017-12-18 14:51:20 +00:00
Michal Majewski
c8102945af [test] Code cleanup in testcfg.py
- All testcase/testsuite/variant generator subclasses renamed to
  just TestCase/TestSuite/VariantGenerator since they're private
  implementation.
- All `testcase` variables renamed to `test` to not conflict with
  a module name.
- No more two statements in the same line.
- Removed some unused testsuite methods.

Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I7710f3419f738a5f9ddca73765dd2cad2e35b952
Reviewed-on: https://chromium-review.googlesource.com/823964
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#50076}
2017-12-13 13:36:20 +00:00
Michal Majewski
08a05c2bff [test] Move data access from suite to test.
- create testcase subclass for each test
- move get_command, get_source from suite to test
- promises-aplus tests are broken
- moving expected outcomes etc. is still in progress

Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I006e64ecf51dda95c41dff88c68b24f17a638566
Reviewed-on: https://chromium-review.googlesource.com/798331
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50065}
2017-12-13 09:14:47 +00:00
Michal Majewski
4f4654a013 [test] Deduplicate parsing source file flags.
Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I12d62e28b8e22820d4358d0166fa5db5e09b8bc3
Reviewed-on: https://chromium-review.googlesource.com/819630
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50033}
2017-12-12 13:12:18 +00:00
Michal Majewski
22eac2e987 [test] Code cleanup in test suites.
Some refactor moved from
https://chromium-review.googlesource.com/c/v8/v8/+/798331.

Bug: v8:6917
Change-Id: I8cae6cfca7a0d7d8e234052c0ab0bfe252355e60
Reviewed-on: https://chromium-review.googlesource.com/819550
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50020}
2017-12-12 08:54:52 +00:00
Michal Majewski
0a12eaea45 Remove env field from testcase.
Bug: v8:6917
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I09fb05ac6d4b9b1223118494ce2c89e3ab5de109
Reviewed-on: https://chromium-review.googlesource.com/771870
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#49415}
2017-11-16 15:26:06 +00:00
Michael Achenbach
0a35d6f985 [test] Let flags from test case have precedence
The status-file flags and the flags from the test case's source code
must always overwrite extra flags set by bots.

Bug: v8:6924
Cq-Include-Trybots: master.tryserver.v8:v8_linux_noi18n_rel_ng
Change-Id: I0e2aabb69da7cfb8ba6c1c79bd3851462071a6ac
Reviewed-on: https://chromium-review.googlesource.com/732656
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49001}
2017-10-27 13:22:39 +00:00
machenbach
2d41b5f835 Revert of [test] add --no-harness option to debugger tests. (patchset #1 id:1 of https://codereview.chromium.org/2831083003/ )
Reason for revert:
Should define its own NO_HARNESS_PATTERN. See comments.

Original issue's description:
> [test] add --no-harness option to debugger tests.
>
> Review-Url: https://codereview.chromium.org/2831083003
> Cr-Commit-Position: refs/heads/master@{#44774}
> Committed: 43c20d4cc5

TBR=caitp@igalia.com,yangguo@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.

Review-Url: https://codereview.chromium.org/2871593002
Cr-Commit-Position: refs/heads/master@{#45163}
2017-05-08 15:06:42 +00:00
yangguo
43c20d4cc5 [test] add --no-harness option to debugger tests.
Review-Url: https://codereview.chromium.org/2831083003
Cr-Commit-Position: refs/heads/master@{#44774}
2017-04-21 12:56:53 +00:00
yangguo
bcb73f6219 [inspector] add scope type for modules.
R=jgruber@chromium.org, kozyatinskiy@chromium.org
BUG=v8:5530

Review-Url: https://codereview.chromium.org/2568083002
Cr-Commit-Position: refs/heads/master@{#41765}
2016-12-16 14:28:56 +00:00
jgruber
0cb3897179 [debugger] Basic scope functionality and exception events in wrapper
BUG=v8:5530

Review-Url: https://codereview.chromium.org/2487673002
Cr-Commit-Position: refs/heads/master@{#40840}
2016-11-08 14:54:10 +00:00
jgruber
83b560b0e5 [debugger] Various break-related functionality in test wrapper
This CL adds simple implementation of break and stepping-related functionality
as required by the debug-step.js test. This includes

* stepOver, stepInto, stepOut
* setBreakPoint
* clearBreakPoint
* evaluate

Some of these, e.g. setBreakPoint are not fully implemented for all cases but
only for the ones we need right now.

One interesting result of this is that using the inspector protocol is roughly
14x slower for debug-step.js (14s instead of 0.5s). One cause of this seems to
be iteration over all object properties in toProtocolValue, which is used to
serialize JS objects before being sent over the wire (e.g. FrameMirrors).  This
is something that should be fixed at some point. In the meantime, the test now
runs 100 instead of 1000 iterations.

BUG=v8:5530

Review-Url: https://codereview.chromium.org/2447073007
Cr-Commit-Position: refs/heads/master@{#40636}
2016-10-28 08:18:34 +00:00
yangguo
2f135d464c [debugger] basic test infrastructure for new debugger test api.
This introduces:
- a way in d8 to send messages to the inspector and receive responses.
- a new test suite where existing debugger tests should migrate to.

R=jgruber@chromium.org, kozyatinskiy@chromium.org, machenbach@chromium.org
BUG=v8:5530

Review-Url: https://chromiumcodereview.appspot.com/2425973002
Cr-Commit-Position: refs/heads/master@{#40487}
2016-10-21 06:38:05 +00:00