We define a TestFailedError exception and raise it when we can reliably detect
that a test has crashed. All other exceptions are treated as infra failures and
are captured by the try-catch clause in MainWrapper function.
This also fixes all tests in run_perf_test.py, run_tests_test.py and makes sure
that both are run on any changes in tools directory.
R=machenbach@chromium.org
Bug: chromium:899028
Change-Id: I283bc87b31c814be476bebe9fdda414975494183
Reviewed-on: https://chromium-review.googlesource.com/c/1303293
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57134}
Originally timestamps were added in crrev.com/c/1159361 to ease debugging. Since
then, however, timestamps were introduce to the LogDog viewer and are rendered
in a separate column, making it easier to scan text for important messages
starting with '>>>'. We do keep the log level, but make sure that it always
takes a fixed width such that the actual messages are aligned vertically.
R=machenbach@chromium.org
No-Try: true
Change-Id: I3a741069824aa25e674aca79da03a725017cb956
Reviewed-on: https://chromium-review.googlesource.com/c/1291370
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56834}
This will allow us to determine whether tests are not printing anything or
whether our infrastructure is failing to retrieve the stdout.
TBR=machenbach@google.com
No-Try: true
Bug: chromium:891314
Change-Id: I0786e77b4db9f247e02a25924acd6d6773c0db9f
Reviewed-on: https://chromium-review.googlesource.com/c/1282962
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56701}
This is a reland of 4c0943424c
Original change's description:
> [test] Add logic to run tests on Android
>
> This adds a new command abstraction for running commands on Android
> using dockered devices on swarming.
>
> The new abstraction handles pushing all required files to the device.
> The logic used for pushing and running is reused from the perf runner.
>
> This adds only the mjsunit test suite. Others will be handled in
> follow up CLs. The suite logic is enhanced with auto-detection of files
> to be pushed to devices, for e.g. load or import statements.
>
> Some test cases need an extra resource section for specifying required
> files.
>
> Remaining failing tests are marked in the status files for later
> triage.
>
> Bug: chromium:866862
> Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
> Reviewed-on: https://chromium-review.googlesource.com/1150153
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55041}
Bug: chromium:866862
Change-Id: Icf7e04c75d4abeab7254d10ba21240e46b0022ae
Reviewed-on: https://chromium-review.googlesource.com/1170643
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55059}
All status messages will be printed into STDERR and only the results will
continue to be written to STDOUT as documentation suggests. Additionally the
following changes are made:
- on Buildbot each log entry will have a timestamp to make it easy to measure
time between status messages printed by test runner, users will continue to
see just the message itself
- when devil (or any other module) logs error, they will not be interleaved
with our own output since logging module is thread-safe
- critical logs replaced with warning since they do not immediately stop
the test runner
- logging.exception is used for all exceptions to get formatted stack trace
TBR=machenbach@chromium.org
No-Try: true
Bug: chromium:838864
Change-Id: I197fcc3bc5890130ce8b35c3e21237f9614fc468
Reviewed-on: https://chromium-review.googlesource.com/1159361
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54851}
This is a reland of 40d66d8bf8
The fix disambiguates duplicate symbols in the generated embedded
builtins file.
Original change's description:
> [build] Make separate snapshot for trusted variant
>
> This enables side-by-side snapshots with and without untrusted-code
> mitigations. It'll be the default in all V8 stand-alone builds
> with external startup data. Internal snapshots are not supported.
>
> The files snapshot_blob.bin and snapshot_blob_trusted.bin will be
> bundled with V8 on swarming and the correct file is loaded dependent
> on the --untrusted-code-mitigations runtime flag.
>
> Likewise we embed two snapshots for builtins.
>
> Side-by-side snapshots won't be supported in Chromium.
>
> Bug: v8:7441
> Change-Id: I2949ddfd5773649946b1c8e74751d48ad1d9c524
> Reviewed-on: https://chromium-review.googlesource.com/960004
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Jakob Gruber <jgruber@chromium.org>
> Reviewed-by: Yang Guo <yangguo@chromium.org>
> Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#52028}
Bug: v8:7441
Change-Id: I626171d4e07389f0453b4d0a698e2772fd37e8c5
Reviewed-on: https://chromium-review.googlesource.com/968623
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52055}
This reverts commit 40d66d8bf8.
Reason for revert:
https://build.chromium.org/p/client.v8/builders/V8%20Mac64/builds/20910
Original change's description:
> [build] Make separate snapshot for trusted variant
>
> This enables side-by-side snapshots with and without untrusted-code
> mitigations. It'll be the default in all V8 stand-alone builds
> with external startup data. Internal snapshots are not supported.
>
> The files snapshot_blob.bin and snapshot_blob_trusted.bin will be
> bundled with V8 on swarming and the correct file is loaded dependent
> on the --untrusted-code-mitigations runtime flag.
>
> Likewise we embed two snapshots for builtins.
>
> Side-by-side snapshots won't be supported in Chromium.
>
> Bug: v8:7441
> Change-Id: I2949ddfd5773649946b1c8e74751d48ad1d9c524
> Reviewed-on: https://chromium-review.googlesource.com/960004
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Jakob Gruber <jgruber@chromium.org>
> Reviewed-by: Yang Guo <yangguo@chromium.org>
> Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#52028}
TBR=rmcilroy@chromium.org,machenbach@chromium.org,yangguo@chromium.org,mstarzinger@chromium.org,jgruber@chromium.org
Change-Id: I5b18f7aff7c05a17842fc68d785bb617eeca4a41
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:7441
Reviewed-on: https://chromium-review.googlesource.com/968622
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52029}
This enables side-by-side snapshots with and without untrusted-code
mitigations. It'll be the default in all V8 stand-alone builds
with external startup data. Internal snapshots are not supported.
The files snapshot_blob.bin and snapshot_blob_trusted.bin will be
bundled with V8 on swarming and the correct file is loaded dependent
on the --untrusted-code-mitigations runtime flag.
Likewise we embed two snapshots for builtins.
Side-by-side snapshots won't be supported in Chromium.
Bug: v8:7441
Change-Id: I2949ddfd5773649946b1c8e74751d48ad1d9c524
Reviewed-on: https://chromium-review.googlesource.com/960004
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52028}
Modify "run_perf.py --filter Array/" so that it will let match "Array" but
not "ArrayLiteralSpread". Previously --filter Array/ would match nothing.
Change-Id: I1f9c677e3558fc2256a71306af5fc1a8394ac698
Reviewed-on: https://chromium-review.googlesource.com/895453
Commit-Queue: Dan Elphick <delphick@chromium.org>
Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51098}
This is a reland of 3b06511052
Original change's description:
> Reland "[test] Creating command before execution phase."
>
> This is a reland of 98cc9e862f
> Original change's description:
> > [test] Creating command before execution phase.
> >
> > Immutable command class with shell, flags and
> > environment.
> >
> > Command creation moved from worker to the main
> > process. Because of that there is no need to send
> > test cases beyond process boundaries and load test
> > suites in worker processes.
> >
> > Bug: v8:6917
> > Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> > Reviewed-on: https://chromium-review.googlesource.com/791710
> > Commit-Queue: Michał Majewski <majeski@google.com>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#49746}
>
> Bug: v8:6917
> Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
> Reviewed-on: https://chromium-review.googlesource.com/800370
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michał Majewski <majeski@google.com>
> Cr-Commit-Position: refs/heads/master@{#49756}
Bug: v8:6917
Change-Id: Ia39010a0a0f63537ad12490dfab17897d70d4930
Reviewed-on: https://chromium-review.googlesource.com/806034
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#49830}
This reverts commit 3b06511052.
Reason for revert: Broke fuzzers:
https://build.chromium.org/p/client.v8.clusterfuzz/builders/V8%20Deopt%20Fuzzer/builds/384
Original change's description:
> Reland "[test] Creating command before execution phase."
>
> This is a reland of 98cc9e862f
> Original change's description:
> > [test] Creating command before execution phase.
> >
> > Immutable command class with shell, flags and
> > environment.
> >
> > Command creation moved from worker to the main
> > process. Because of that there is no need to send
> > test cases beyond process boundaries and load test
> > suites in worker processes.
> >
> > Bug: v8:6917
> > Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> > Reviewed-on: https://chromium-review.googlesource.com/791710
> > Commit-Queue: Michał Majewski <majeski@google.com>
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#49746}
>
> Bug: v8:6917
> Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
> Reviewed-on: https://chromium-review.googlesource.com/800370
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michał Majewski <majeski@google.com>
> Cr-Commit-Position: refs/heads/master@{#49756}
TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: v8:6917
Change-Id: I4938642c4396366be1e13daf6998c4b8538b688b
Reviewed-on: https://chromium-review.googlesource.com/804254
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49805}
This is a reland of 98cc9e862f
Original change's description:
> [test] Creating command before execution phase.
>
> Immutable command class with shell, flags and
> environment.
>
> Command creation moved from worker to the main
> process. Because of that there is no need to send
> test cases beyond process boundaries and load test
> suites in worker processes.
>
> Bug: v8:6917
> Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> Reviewed-on: https://chromium-review.googlesource.com/791710
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#49746}
Bug: v8:6917
Change-Id: I49c29a8db813c47909f2cc45070ac7721a447c7a
Reviewed-on: https://chromium-review.googlesource.com/800370
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michał Majewski <majeski@google.com>
Cr-Commit-Position: refs/heads/master@{#49756}
This reverts commit 98cc9e862f.
Reason for revert: Breaks test isolation:
https://build.chromium.org/p/client.v8/builders/V8%20Linux%20-%20builder/builds/29746
Original change's description:
> [test] Creating command before execution phase.
>
> Immutable command class with shell, flags and
> environment.
>
> Command creation moved from worker to the main
> process. Because of that there is no need to send
> test cases beyond process boundaries and load test
> suites in worker processes.
>
> Bug: v8:6917
> Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
> Reviewed-on: https://chromium-review.googlesource.com/791710
> Commit-Queue: Michał Majewski <majeski@google.com>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#49746}
TBR=machenbach@chromium.org,sergiyb@chromium.org,majeski@google.com
Change-Id: I44b99468d18fd093833f4185dad067a9eeaf2bc1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:6917
Reviewed-on: https://chromium-review.googlesource.com/800292
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49747}
Immutable command class with shell, flags and
environment.
Command creation moved from worker to the main
process. Because of that there is no need to send
test cases beyond process boundaries and load test
suites in worker processes.
Bug: v8:6917
Change-Id: Ib6a44278095b4f7141eb9b96802fe3e8117678a6
Reviewed-on: https://chromium-review.googlesource.com/791710
Commit-Queue: Michał Majewski <majeski@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49746}
The runs are now called as primary (no suffix) and secondary. This is in
preparation to adding secondary builds on CI, which will run tests on the latest
released stable V8 binary (aka as ref builds).
R=machenbach@chromium.org
Bug: chromium:783763
Change-Id: Ie6560012887bd5bb0d948bc8d34a9256d922137c
Reviewed-on: https://chromium-review.googlesource.com/781941
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49560}
Added a flag, process_size, to collect the maximum memory size used by
a test run.
Bug:
Change-Id: I5d659ff03bfcf7ab4a868aafbda3303ffcb9fa78
Reviewed-on: https://chromium-review.googlesource.com/542415
Commit-Queue: Mircea Trofin <mtrofin@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#46713}
This is a highly requested feature!
Bug: v8:6276
Change-Id: I17b606ae0ff8fa9dfdd0fa74fd1f7ad0dd3fc4f8
Reviewed-on: https://chromium-review.googlesource.com/488044
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#44916}
Inspired by bsuite/compare-baseline.py and tools/perf-to-html.py. The main difference
from the latter is that the new script does not require json files preprocessing and it
is able to compare multiple json files at once (just like bsuite/compare-baseline.py).
This CL also makes tools/run_perf.py relative path friendly.
BUG=
Change-Id: I7be796fa920f8360703806f32767b992eec0f9c2
Reviewed-on: https://chromium-review.googlesource.com/456286
Reviewed-by: Michael Stanton <mvstanton@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#43982}
For results processor to work, we need to maintain the assumption that the cwd is equal to the currently running benchmark directory.
NOTRY=true
TBR=clemensh@chromium.org
Change-Id: I585c301dea846194652645973470786ad43aa280
Reviewed-on: https://chromium-review.googlesource.com/444784
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#43280}
This adds the possibility to specify a python script for post-processing stdout.
This also adds some system tests for testing the new feature.
NOTRY=true
Change-Id: I0383afb3e23513629508feeb639ed2dfce56b54a
Reviewed-on: https://chromium-review.googlesource.com/443449
Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#43257}
For benchmarks with multiple measurements (e.g. runtime, compile
time, generated code size), in case of error, we'd get only the
last portion of the name ("Compile"), which is ambiguous and makes
debugging dificult. This change includes the full name, e.g.
"Unity/Physics/Compile".
BUG=
Review-Url: https://codereview.chromium.org/2615623003
Cr-Commit-Position: refs/heads/master@{#42078}
I plan to drive these options from the infra recipe.
Raising the priority of the benchmarked process, and fixing
its affinity to some other CPU than 0 improves stability of
measurements.
BUG=
Review-Url: https://codereview.chromium.org/1915303002
Cr-Commit-Position: refs/heads/master@{#35912}
Reason for revert:
[Sheriff] Still fails on android.
Original issue's description:
> reland [tools] add --pretty switch to run_perf.py
>
> This CL improves running our internal benchmarks locally by adding the
> --pretty option to tools/run_perf.py. With the flag enabled we print
> the run-time of each benchmark directly and avoid the json output at
> the end.
>
> NOTRY=true
>
> Committed: https://crrev.com/b543c40d78c3aa8283a7d9fccb6c9e26a9a81f90
> Cr-Commit-Position: refs/heads/master@{#34023}
TBR=cbruni@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1700953002
Cr-Commit-Position: refs/heads/master@{#34033}
This CL improves running our internal benchmarks locally by adding the
--pretty option to tools/run_perf.py. With the flag enabled we print
the run-time of each benchmark directly and avoid the json output at
the end.
NOTRY=true
Review URL: https://codereview.chromium.org/1681283004
Cr-Commit-Position: refs/heads/master@{#34023}
Reason for revert:
[Sheriff] Breaks android runs. Will look later why.
Original issue's description:
> [tools] add --pretty switch to run_perf.py
>
> This CL improves running our internal benchmarks locally by adding the
> --pretty option to tools/run_perf.py. With the flag enabled we print
> the run-time of each benchmark directly and avoid the json output at
> the end.
>
> NOTRY=true
>
> Committed: https://crrev.com/83f69507ab1b9380b56758b747d4f3fabc849e49
> Cr-Commit-Position: refs/heads/master@{#33981}
TBR=cbruni@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1696293002
Cr-Commit-Position: refs/heads/master@{#34017}
This CL improves running our internal benchmarks locally by adding the
--pretty option to tools/run_perf.py. With the flag enabled we print
the run-time of each benchmark directly and avoid the json output at
the end.
NOTRY=true
Review URL: https://codereview.chromium.org/1681283004
Cr-Commit-Position: refs/heads/master@{#33981}
Improved flexibility for the perf runner, by adding option to
specify precisely shell binary.
NOTRY=true
Review URL: https://codereview.chromium.org/1659483003
Cr-Commit-Position: refs/heads/master@{#33649}
Enhance the perf runner to run with the profiler and print the summary for each d8 run. This automates running the profiler with multiple benchmarks.
BUG=None
LOG=N
TBR=machenbach@chromium.org
Review URL: https://codereview.chromium.org/1327033003
Cr-Commit-Position: refs/heads/master@{#30720}
