Commit Graph

152 Commits

Author SHA1 Message Date
Liviu Rau
9aa28daafe [test] Refactor testrunner (5)
- Unify old Pool interface with the new context related interface
 - Add single threaded execution pool
 - Defer task killing back to OS context
 - Defer process listing in indicators back to OS context

Bug: v8:12785
Cq-Include-Trybots: luci.v8.try:v8_numfuzz_dbg_ng,v8_numfuzz_ng,v8_numfuzz_tsan_ng,v8_android_arm64_n5x_rel_ng
Change-Id: I8ffe01c5d567411203f69ecc451c718ff35d81c9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3781347
Reviewed-by: Alexander Schulze <alexschulze@chromium.org>
Commit-Queue: Liviu Rau <liviurau@google.com>
Cr-Commit-Position: refs/heads/main@{#82371}
2022-08-11 05:55:52 +00:00
Liviu Rau
b3477fdd01 [test] Refactor testrunner (4)
- Removed duplication and unnecessary indirection from all suites testcfgs.
 - Introduced a more comprehensive context to cover both command context and other platform specific concerns.
 - Propagated above context to TestLoader to allow for test counting command execution on all platforms.
 - Wrapped original pool with another class to give it a new interface and allow injecting different implementations in the future.
 - Consolidated progress indicators under a single processor in the pipeline.
 - Consolidated result retention requirements calculation outside of pipeline chain.
 - Refactored LoaderProc and got it under tests.
 - Added some more tests for the standard runner.
 - Extracted BuildConfig class.


Bug: v8:12785
Change-Id: I87be040e91f792a983662bb5a10d55b36a14ea7f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3701595
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81770}
2022-07-18 09:52:24 +00:00
Liviu Rau
9733dacc5c [test] Refactor testrunner (3)
- removed some 'feature envy' instances:
  - created a AugmentedOptions class to take care of non trivial option look-ups
  - moved some calculation closer the the actual data
- promoted parameter that was passed around a lot (options) to filed in the offending class
- similar object creation looks a bit more similar
- CommandContext provides a wrapper that ensures resource de-allocation
- separate tests from standard and num_fuzz runners
- added a couple of more tests to improve coverage

This is still in flux. But further changes risk creating a disconnect between the original implementation and further refactorings.

Bug: v8:12785
Change-Id: I0ec2e9c6a81604a5cd1d4a80982dd3329c1b48db
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3686411
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81148}
2022-06-14 12:49:47 +00:00
Tobias Tebbi
77ba98ef32 [turboshaft] add support for all JS machine-level operators
In particular, this CL adds support for:
- exception handling
- source positions
- OSR
- various numeric operations and conversions

Since the test suite now passes with `--turboshaft`, this also adds a
new variant for Turboshaft and enables it on some bots.

Bug: v8:12783
Change-Id: Ia2dd2e16f56fc955d49e51f86d050218e70cb575
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3669251
Reviewed-by: Darius Mercadier <dmercadier@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Commit-Queue: Tobias Tebbi <tebbi@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81074}
2022-06-10 15:17:20 +00:00
Liviu Rau
b5a7ca848c [test] Refactor testrunner (2)
- moved test data closer to tests
- removed the coverage related code
- refactored to remove boilerplate from test code

Bug: v8:12785
Change-Id: I1013d29d8ff2c3ecb786c294ae3b3ab6decdca20
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3683610
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80921}
2022-06-02 16:49:28 +00:00
Liviu Rau
0281df0421 [test] Refactor testrunner (1)
Bringing test closer together

Bug: v8:12785
Change-Id: I2eed79fd3b1bda210b1f58b341422fc0c6435bfd
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3678210
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80887}
2022-06-01 10:37:21 +00:00
Camillo Bruni
b29ed5a59a [tools] run-tests.py
List all variants for the --variant help text

Change-Id: I249d8140b19e13dc3eceedaade2b856b1fdb1567
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3663088
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80884}
2022-06-01 09:17:58 +00:00
Michael Achenbach
b086206161 Reland "[tools] Clean up py2 code"
This is a reland of commit 1289704aae

Mac-arm64 problem fixed by:
https://crrev.com/c/3550199

Original change's description:
> [tools] Clean up py2 code
>
> Bug: chromium:1292013
> Change-Id: Ic2c3a197005a2136bb0eda4cbb36d8eb57f42a7c
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3523047
> Reviewed-by: Liviu Rau <liviurau@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#79507}

Bug: chromium:1292013
Change-Id: Iadf0ccf94c82012088b76a866296c8e008dff02f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3550274
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#79624}
2022-03-25 12:14:26 +00:00
Michael Achenbach
f7aac0cf9f Revert "[tools] Clean up py2 code"
This reverts commit 1289704aae.

Reason for revert:
https://ci.chromium.org/ui/p/v8/builders/ci/V8%20Mac%20-%20arm64%20-%20release/8735/overview

Original change's description:
> [tools] Clean up py2 code
>
> Bug: chromium:1292013
> Change-Id: Ic2c3a197005a2136bb0eda4cbb36d8eb57f42a7c
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3523047
> Reviewed-by: Liviu Rau <liviurau@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#79507}

Bug: chromium:1292013
Change-Id: I48c38209220b775d0caa45f487ed163f78333e0c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3532228
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Cr-Commit-Position: refs/heads/main@{#79508}
2022-03-17 10:07:38 +00:00
Michael Achenbach
1289704aae [tools] Clean up py2 code
Bug: chromium:1292013
Change-Id: Ic2c3a197005a2136bb0eda4cbb36d8eb57f42a7c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3523047
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#79507}
2022-03-17 09:16:57 +00:00
Jakob Gruber
0a6c1a778a Remove the turboprop implementation
Bug: v8:12552
Change-Id: I99e4d8e8aeba5460f11e54cc1b2bcaea98a5276d
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3400964
Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Tobias Tebbi <tebbi@chromium.org>
Commit-Queue: Jakob Gruber <jgruber@chromium.org>
Cr-Commit-Position: refs/heads/main@{#78698}
2022-01-20 12:31:39 +00:00
Victor Gomes
c5c60391e4 [tools] Use absolute imports in testrunner
It also updates the scripts to support Python3

Bug: chromium:1245634
Change-Id: Iffe29bacfd788575b35da6449d5830fc665da7a8
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3194259
Commit-Queue: Victor Gomes <victorgomes@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#77155}
2021-09-30 07:54:10 +00:00
Al Muthanna Athamina
eae0a9a390 Add infra-staging flag to test runner
Add infra-staging flag to test runner
which adds the no-fail flag. This will
be used to see the accuracy of numfuzz
builders when we ignore exit code 1.

Bug: v8:11826
Change-Id: I6684331efe9c801d02716d94cb16e8ba816d9c68
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3110196
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Almothana Athamneh <almuthanna@chromium.org>
Cr-Commit-Position: refs/heads/main@{#76409}
2021-08-20 15:44:52 +00:00
Mythri A
ef7d657960 [sparkplug] Add support to flush only baseline code
Add support to flush only baseline code. FLAG_flush_baseline_code
controls if baseline code is flushed or not and FLAG_flush_bytecode
controls if bytecode is flushed or not. With this CL it is possible
to control if we want to flush only bytecode / only baseline code / both.
This also lets us have different heuristics for bytecode and baseline
code flushing.

Bug: v8:11947
Change-Id: Ibdfb9d8be7e7d54196db7890541fa0b5d84f037e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3060481
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Commit-Queue: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#76075}
2021-08-04 08:22:18 +00:00
Mythri A
06697f7a82 Rename stress_flush_bytecode to stress_flush_code
stress_flush_bytecode controls stress flushing of both bytecode and
baseline code. So rename the flag to better reflect its functionality

Bug: v8:11947
Change-Id: Ie6c124a476c3a7c6eabd1d75de030ee15fe78e32
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3062567
Commit-Queue: Mythri Alle <mythria@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Reviewed-by: Omer Katz <omerkatz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#76043}
2021-08-02 19:02:57 +00:00
Michael Achenbach
63d5919b85 [test] Limit running heavy tests sequentially to swarming
This limits the feature of running Memory-hungry tests in sequence to
the GCE bots in swarming. There, the positive memory impact is large
due to the overall lower system memory and the negative runtime impact
is small due to the smaller number of cores.

Bug: v8:11928
Change-Id: Ib577b455d5cccbb3c9855526becbdbf822259bea
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2991630
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#75406}
2021-06-28 09:11:00 +00:00
Michael Achenbach
ee56a9863e [test] Run heavy tests sequentially
This adds a new status file indicator "HEAVY" to mark tests with high
resource demands. There will be other tests running in parallel,
but only a limited number of other heavy tests. The limit is
controlled with a new parameter --max-heavy-tests and defaults to 1.

The change also marks a variety of tests as heavy that recently had
flaky timeouts. Heavy also implies slow, hence heavy tests are
executed at the beginning with a higher timeout like other slow tests.

The implementation is encapsulated in the test-processor chain. A
new processor buffers heavy tests in a queue and adds buffered tests
only if other heavy tests have ended their computation.

Bug: v8:5861
Change-Id: I89648ad0030271a3a5af588ecc9c43285b728d6d
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2905767
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#74712}
2021-05-21 12:59:49 +00:00
Leszek Swirski
15803e5fa9 [test] Replace turboprop_as_toptier with always_sparkplug
In the 'extra' variant, replace turboprop_as_toptier (which is mostly
covered by turboprop) with always_sparkplug, to increase Sparkplug's
coverage on tests.

Bug: v8:11420
Change-Id: I43b7009d184a03cb163d78f1f8bd05233ef7c34e
Fixed: v8:11686
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2862769
Auto-Submit: Leszek Swirski <leszeks@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#74324}
2021-05-03 12:33:38 +00:00
Mythri A
7ea641455a [turboprop] Add a new test variant for turboprop-as-toptier
Bug: v8:9684
Change-Id: Ie8c684998b9811c85ab385037d13604ac838b962
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2637225
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#72249}
2021-01-22 11:19:59 +00:00
Michael Achenbach
373a9a8cfc [test] Switch to flattened json output
This flattens the json output to one result record as a dict. In
the past several records with different arch/mode combinations
could be run, but this is deprecated since several releases.

We also drop storing the arch/mode information in the record as it
isn't used on the infra side for anything.

This was prepared on the infra side by:
https://crrev.com/c/2453562

Bug: chromium:1132088
Change-Id: I944514dc00a671e7671bcdbcaa3a72407476d7ad
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2456987
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#70402}
2020-10-08 13:05:11 +00:00
Michael Achenbach
aede8c7795 [test] Derive simulator status flag from build product
Guessing the architecture on the testing machine is not required to
derive if the simulator is supposed to run. The architecture check
doesn't work on some platforms.

We derive this now by checking if we have compiled a simulator build:
target_cpu != v8_target_cpu.

Bug: chromium:1110824
Change-Id: Id30a647f0610f21efb00d68ad1602e62dcd2c65c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2395563
Reviewed-by: Jakob Kummerow <jkummerow@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#69719}
2020-09-07 15:10:23 +00:00
Ng Zhi An
8e4df90b8c [Py3] Get tools/testrunner closer to Py3
This modernizes python code without breaking Py2 compat.

Ran with command:

futurize --stage1 -w tools/testrunner

Bug: v8:9871
Change-Id: Ie23333cbd923197be0bffcad5041056e00990042
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2252554
Commit-Queue: Zhi An Ng <zhin@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68496}
2020-06-23 21:09:06 +00:00
Camillo Bruni
0961376575 [tools] Print variants in test runner
Print ALL_VARIANTS and VARIANT_ALIASES when passing a wrong
--variants argument to tools/run-tests.py.

Change-Id: I6d4278633dd11990d0ace1c93f544213fbfc156e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2139579
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#67039}
2020-04-07 12:36:15 +00:00
Michael Achenbach
c12f395da7 [test] Don't use --testing-d8-test-runner on NumFuzz
The extra checks done with --testing-d8-test-runner might not hold
when using NumFuzz. This refactors the test runner and allows passing
implementation specific flags. The --testing-d8-test-runner flag is
now only passed in the standard-runner not in the numfuzz
implementation.

Bug: v8:10220
Change-Id: I83cac57a948c98c34f2d84f41d719e0434e25ee7
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2064217
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66341}
2020-02-19 14:40:19 +00:00
Georg Neis
ed0dce9d2f [tests] Exercise Turbofan's instruction scheduling
Turbofan's instruction scheduling is currently only enabled for
mksnapshot and has thus little test coverage. This CL introduces two
new test variants, "instruction_scheduling" and
"stress_instruction_scheduling", and activates them on a selection of
bots.

Change-Id: I5917fc781e289377c58f584c770c91e31765b2de
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1899778
Auto-Submit: Georg Neis <neis@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Stanton <mvstanton@chromium.org>
Commit-Queue: Georg Neis <neis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64807}
2019-11-06 14:21:54 +00:00
Liviu Rau
c5783e27c7 Remove nosnap configs and references
Bug: v8:9856
Change-Id: Ia7dd11c3e603b7880a90fb89087d3118409b432f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1864650
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Simon Zünd <szuend@chromium.org>
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64404}
2019-10-21 07:28:38 +00:00
Joshua Litt
f25edf22e7 [d8] Terminate execution instead of omitting quit in d8 asan
Bug: v8:4653
Change-Id: I2b2e0e12dc7c3734dd554aa6dd5ed71c90a77758
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1806796
Reviewed-by: Yang Guo <yangguo@chromium.org>
Commit-Queue: Joshua Litt <joshualitt@chromium.org>
Cr-Commit-Position: refs/heads/master@{#63911}
2019-09-20 16:38:19 +00:00
Jakob Kummerow
b823bf1ba6 [test][cleanup] Revive --time, speed up some tests
This reimplements the "--time" option of run-tests.py to print the
20 slowest tests, on top of json_test_results infrastructure just
like the bots do it.
Additionally this CL speeds up a bunch of slow tests.

Bug: v8:9396
Change-Id: I40797d2c8c3bfdd310b72f15cd1a035844b7c6f3
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1803635
Commit-Queue: Jakob Kummerow <jkummerow@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#63786}
2019-09-16 11:24:11 +00:00
Ross McIlroy
db3cc4a247 [Test] Add TurboProp test variant.
BUG=v8:9684

Change-Id: Ifdcb8497fbbae157df270e87451639779f8a78c6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1784917
Commit-Queue: Ross McIlroy <rmcilroy@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#63626}
2019-09-09 16:49:24 +00:00
Sam Clegg
436fcaa032 Fix some python nits in tools/testrunner/standard_runner.py
I noticed the indentation was off in one function, but also fixed
all the other flake8 issues in this file.

Change-Id: I2303ed87da7154484a872315f8355f57621514c4
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1697054
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sam Clegg <sbc@chromium.org>
Cr-Commit-Position: refs/heads/master@{#62669}
2019-07-12 10:17:31 +00:00
Clemens Hammacher
01821ab3b6 [test] Stress wasm code gc in "stress_background_compile"
The "stress_background_compile" variant runs on all our bots. We
combine it with testing wasm code GC (which kind of fits into
background compile stressing) to get more coverage for that. Both
features are orthogonal, so we can test both at the same time without
loosing any coverage.

R=machenbach@chromium.org
CC=rmcilroy@chromium.org

Bug: v8:8217
Change-Id: Ib17decd4869978ff98e302694fa73d70ceec120e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1588472
Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61151}
2019-05-02 09:56:05 +00:00
Clemens Hammacher
b76ccb9471 [wasm][gc] Stress wasm code gc in --gc-stress mode
This makes the gc-stress bots set the '--stress-wasm-code-gc' flag.
Note that this also implicitly enables wasm code gc on these bots.

R=mstarzinger@chromium.org, machenbach@chromium.org

Cq-Include-Trybots: luci.v8.try:v8_linux_gc_stress_dbg
Cq-Include-Trybots: luci.v8.try:v8_mac64_gc_stress_dbg
Bug: v8:8217
Change-Id: Idc4fb996f50632d5621200e138f3ddc572ed9b05
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1585721
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61089}
2019-04-29 14:32:46 +00:00
Michael Achenbach
98cde3f400 [test] Expose test-framework name in test results
This will allow to distinguish between the standard runner and the num fuzzer
on the infra side when generating flako command lines. The value could be
inferred, but it'd be more confusing.

Bug: v8:8971
Change-Id: I78f5104135d1c7fd7d98bceb4b17897e79421455
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1564050
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60769}
2019-04-11 11:34:53 +00:00
Michael Achenbach
73bbdbfdf6 [test] Clean-up: Fix indentation and quotation
This has no logic changes intended.

Bug: v8:8971
Change-Id: I6d35096cd5e0ea189006ba7a5663606f101ccab9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1564051
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60766}
2019-04-11 11:03:33 +00:00
Clemens Hammacher
31b49aeeb0 [test] Test more wasm variants
Since wasm optimizations are independent of JS optimizations, we can
just combine the "no_liftoff" variant with the existing "stress"
variant (which has the similar "--always-opt" option for JS), and add
a "Liftoff only" variant as part of "nooptimization".

This gives more coverage to find bugs like https://crrev.com/c/1543354
more easy.

R=mstarzinger@chromium.org, machenbach@chromium.org

Change-Id: I81bb22074c59dcb650a05252da43a4170cd467ac
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1559740
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Michael Starzinger <mstarzinger@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60714}
2019-04-09 13:41:52 +00:00
Clemens Hammacher
ac8c78e01d [cleanup] Replace underscore by minus in d8 flags
Reading --verify_predictable makes me sad, whereas --verify-predictable
makes me happy. This CL introduces more happiness.

R=machenbach@chromium.org

Bug: v8:8834
Change-Id: Id51a75f32e6d5a2f87aed81e058a8b6dff189758
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1550399
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60599}
2019-04-03 12:32:08 +00:00
Tamer Tas
8c9b8127f2 [testrunner] remove duplicate word from prompt
R=clemensh@chromium.org
CC=machenbach@chromium.org,sergiyb@chromium.org

Bug: v8:8728
Change-Id: I37d10867a805647c98660699233c5c2b23429eae
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1503377
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60038}
2019-03-05 15:51:30 +00:00
Michael Achenbach
d5ef815ca7 [test] Pull exit-code logic to shared base runner
This makes the test runner and numfuzz share the same exit code behavior on
errors. This is needed as they also share the same infrastructure logic
to collect swarming tasks.

Bug: chromium:937228
Change-Id: I155b37c7b10dd22959a4dcf30bbd0321c452236b
Reviewed-on: https://chromium-review.googlesource.com/c/1495987
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59976}
2019-03-01 14:24:56 +00:00
Vadim Gorbachev (bmsdave)
6cad46b4f3 Preparing v8 to use with python3 /tools/testrunner
There are now less that 400 days until the end of life
of Python 2(aka _legacy_ Python) https://pythonclock.org/ .
The code compatibility check for python2 and python3
used the following tools: futurize, flake8
You can see the reports here: https://travis-ci.com/bmsdave/v8/builds

This CL was uploaded by git cl split.

Bug: v8:8594
Change-Id: I2a90aaecb270f03aed1c0fc92da1a0e2621b0eb2
Reviewed-on: https://chromium-review.googlesource.com/c/1470101
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59678}
2019-02-19 09:23:11 +00:00
Tamer Tas
5b957fa169 [testrunner] explain how the testrunner progress percentage is calculated
Progress indicator calculates the percentage using the estimated amount of
tests.

When base tests produce more tests or when testrunner filters some tests, the
percentage terminates over 100% or under it.

This CL adds an informative message about how the percentage behaves.

R=machenbach@chromium.org
CC=yangguo@chromium.org,sergiyb@chromium.org

Bug: v8:8728
Change-Id: I91cafd2579ea1894ac347ff7483c307cd46c545d
Reviewed-on: https://chromium-review.googlesource.com/c/1477056
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59660}
2019-02-18 13:42:37 +00:00
Tamer Tas
baeb4e324d [testrunner] enable the progress indicator
Using test generators meant that we had to remove the progress indicator since
the total number of tests weren't known before-hand.

This CL implements a progress indicator using test number estimations.

cctest and unittests progress indicator is accurate, however estimating
means the progress will terminate over 100% in big test suites and sometimes
under 100%.

R=machenbach@chromium.org
CC=​sergiyb@chromium.org,yangguo@chromium.org

Bug: v8:8769
Change-Id: I40ca5b40f9b1223376d33707f0945900ea98cea3
Reviewed-on: https://chromium-review.googlesource.com/c/1460471
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59538}
2019-02-12 16:03:17 +00:00
Tamer Tas
ad5b6d7dcb [testrunner] enable variant sharding
Testrunner runs variants of a test sequentially without taking sharding into
account. A slow test with slow variants slows down the whole test run no matter
the sharding configuration.

This CL implements a test hashing algorithm and variant sharding for test
variants.

R=machenbach@chromium.org
CC=yangguo@chromium.org,sergiyb@chromium.org

Bug: v8:8174
Change-Id: I15f8c547fa2f361fb6c53bf8d5df055d3df38d3e
Reviewed-on: https://chromium-review.googlesource.com/c/1458016
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59530}
2019-02-12 13:40:33 +00:00
Michael Achenbach
b7b43b7dc0 [test] Run jitless on all bots
Bug: v8:8778
Change-Id: I384ad4387743d534a79ebad03130e8a688cc4631
Reviewed-on: https://chromium-review.googlesource.com/c/1449691
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59332}
2019-02-04 12:04:08 +00:00
Tamer Tas
3f83accb09 [testrunner] load tests concurrently into test execution processor
loading every test up-front into the processing queue costs about 224MB for a
x64 testsuite run.

This CL eliminates that overhead by utilizing generators and threading.

LoadingProc now loads test after receiving the results of the loaded tests.

R=machenbach@chromium.org
CC=​yangguo@chromium.org,sergiyb@chromium.org

Bug: v8:8174,v8:8731
Change-Id: Ifee79c3e213da568f092de0f1623016174e9410c
Reviewed-on: https://chromium-review.googlesource.com/c/1439240
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59223}
2019-01-31 08:19:06 +00:00
Michael Achenbach
4d2a3cc9a5 [test] Add interpreted-regexp testing variant
This adds the new runtime flag as a variant and runs it on a subset of
builders corresponding to the "extra" testing set.

Currently failing tests are skipped in the new variant.

After https://crrev.com/c/1433777 this costs only little additional
resources.

Bug: v8:8678
Change-Id: Ibd0e38872814d11252e55a7c6a58d313aa84ebe3
Reviewed-on: https://chromium-review.googlesource.com/c/1433774
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59080}
2019-01-24 19:11:22 +00:00
Michael Achenbach
1c0b17b03f Revert "[testrunner] load tests concurrently into test execution processor"
This reverts commit 25457c60a7.

Reason for revert: https://crbug.com/v8/8731

Original change's description:
> [testrunner] load tests concurrently into test execution processor
> 
> loading every test up-front into the processing queue costs about 224MB for a
> x64 testsuite run.
> 
> This CL eliminates that overhead by utilizing generators and threading.
> 
> LoadingProc now loads test after receiving the results of the loaded tests.
> 
> R=​machenbach@chromium.org
> CC=​​yangguo@chromium.org,sergiyb@chromium.org
> 
> Bug: v8:8174
> Change-Id: I8f4e6de38430c54fe126e4504b52851866769efb
> Reviewed-on: https://chromium-review.googlesource.com/c/1420678
> Commit-Queue: Tamer Tas <tmrts@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59056}

TBR=machenbach@chromium.org,tmrts@chromium.org

Change-Id: I1e074a031dced367a32a93827b9e863b0331340f
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:8174
Reviewed-on: https://chromium-review.googlesource.com/c/1433792
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59065}
2019-01-24 13:56:04 +00:00
Michael Achenbach
588373412c [test] Reduce testing 'trusted' variant to arm builders
Previously, trusted (or no-mitigations) has been tested on a subset of builders
from all platforms. This reduces it to arm-sim and native Android devices.

Change-Id: I90066686e6a92db4a944025538e01a117f324421
Reviewed-on: https://chromium-review.googlesource.com/c/1433777
Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59062}
2019-01-24 12:42:41 +00:00
Tamer Tas
25457c60a7 [testrunner] load tests concurrently into test execution processor
loading every test up-front into the processing queue costs about 224MB for a
x64 testsuite run.

This CL eliminates that overhead by utilizing generators and threading.

LoadingProc now loads test after receiving the results of the loaded tests.

R=machenbach@chromium.org
CC=​yangguo@chromium.org,sergiyb@chromium.org

Bug: v8:8174
Change-Id: I8f4e6de38430c54fe126e4504b52851866769efb
Reviewed-on: https://chromium-review.googlesource.com/c/1420678
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59056}
2019-01-24 11:19:12 +00:00
Tamer Tas
a8b8a4408b [testrunner] remove the up-front test counter
Moving to a lazy test loading approach makes counting the total number of tests
non-trivial.

For CI runs, we output the total number of tests after the run.
For users, progress indicator signals the status of the run.

R=machenbach@chromium.org
CC=​yangguo@chromium.org,sergiyb@chromium.org

Bug: v8:8174
Change-Id: I0731964515aac60a3629acee6c7243433a2b4e04
Reviewed-on: https://chromium-review.googlesource.com/c/1420677
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#58944}
2019-01-21 07:34:53 +00:00
Tamer Tas
d6c915189f [testrunner] use selection sort generator instead of timsort for test cases
V8 testrunner is loading every test it has to run into memory greedily in order
to sort by slowness of the test case. The memory and CPU overhead for loading
the test-suites are non-trivial.

This CL restructures it by changing the sorting method.

R=machenbach@chromium.org
CC=​​sergiyb@chromium.org,yangguo@chromium.org

Bug: v8:8174
Change-Id: I08331182147b92cf4ac54823eea0e2b472f51e84
Reviewed-on: https://chromium-review.googlesource.com/c/1406684
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#58821}
2019-01-15 11:27:19 +00:00