- Mark run_perf.py executable
- Add more user-friendly option --d8-path aliase
- Add --repeat alias that matches the go/crossbench flag
- Handle symlinks for d8-directory using pathlib
- Only print timeout message if the result has timed_out == True
- Add .DS_Store to gitignore
Change-Id: Ia0fb0b926632af4b520d3aaf447e5bd35723816e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4205910
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Reviewed-by: Alexander Schulze <alexschulze@chromium.org>
Cr-Commit-Position: refs/heads/main@{#85555}
- Removed duplication and unnecessary indirection from all suites testcfgs.
- Introduced a more comprehensive context to cover both command context and other platform specific concerns.
- Propagated above context to TestLoader to allow for test counting command execution on all platforms.
- Wrapped original pool with another class to give it a new interface and allow injecting different implementations in the future.
- Consolidated progress indicators under a single processor in the pipeline.
- Consolidated result retention requirements calculation outside of pipeline chain.
- Refactored LoaderProc and got it under tests.
- Added some more tests for the standard runner.
- Extracted BuildConfig class.
Bug: v8:12785
Change-Id: I87be040e91f792a983662bb5a10d55b36a14ea7f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3701595
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81770}
In the previous refactoring --filter was accidentally filtering out
paths that matched <regexp>. This CL restores the original behavior
where only the matching paths are kept.
Bug: v8:12821, v8:11113
Change-Id: I7e7d7b793107fbf9b4944b1674874150803f4bb4
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3623539
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80426}
We usually run benchmarks in multiple variants: default, future, noopt
This is currently only achieved by copying the run-perf json file and
changing the flags at the top-level (or copy whole subsections).
Using "variants" we can duplicate the tests at the current level with
different values and easily create benchmarks that differ only in v8
flags.
Drive-by-fix:
- Add Node.__iter__ and log the whole config graph in debug mode
- Add GraphConfig.__str__ method for better debugging
- Rename TraceConfig to LeafTraceConfig
- Rename RunnableTraceConfig to RunnableLeafTraceConfig
- Make --filter accept a regexp to better filter out variants
Bug: v8:12821, v8:11113
Change-Id: I56a2ba2dd24da15c7757406e9961746219cd8061
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3596128
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80307}
This fixes all Python3 problems in scripts and tests running via
v8_presubmit.py. It includes:
- Test runner
- Release tools
- Perf runner
- Torque formatter
- V8's main presubmit
On bots, v8_presubmit is run with vpython, hence we also add
the required dependencies. After the Python3 migration, most
of the transitional code in this CL can be removed again.
Bug: chromium:1293709,chromium:1292016
Change-Id: Ic25e5965948b212c047e9d5194d2a4b6db1fa91b
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3432213
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#78936}
This prepares moving the build directory on bots to out/build. For a
smooth transition, the performance runner will dynamically check
for the build in several locations.
This prepares:
https://crrev.com/c/2426643
NOTREECHECKS=true
Bug: chromium:1132088
Change-Id: Ia12fcdedec0f4ac2bfe087e8154c0acb8771a43f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2431364
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#70158}
When an overall timeout is reached, swarming sends a SIGTERM to
terminate the test runner. The test runner has a signal handler on the
main process to terminate all workers gracefully.
Additionally, every worker process installs a signal handler for
terminating ongoing tests wrapped by command.Command.
Also, command.Command is used on the main process to list tests for
cctest and gtest executables, which led to overriding the test runner's
main signal handler.
This CL disables using signal handlers in commands by default and only
explicitly enables it in safe source locations.
Bug: v8:8292
Change-Id: Ifceadaff75bdd2b77e761498bccbe00b6a3e265c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2002528
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#65815}
perfrunner returns a failure if the build timeouts at any point even if it's
successful after retries. It tries to surface up the timeout issue. Due to this,
some bots stay red consistently, and confuses the sheriffs.
This CL masks the timeouts if the suite succeeds in the end.
TBR=verwaest@chromium.org,sergiyb@chromium.org
Bug: v8:9494
Change-Id: I8e107e80dfaa51095501bb2e855d9fbbe4023da9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1702612
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Auto-Submit: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#62735}
The script does not have the shebang and apparently running it without 'python'
on the command line produces some strange errors such as script trying to parse
itself as a suite config. Making it non-executable will make it clear that users
should prefix it with 'python' or 'vpython'.
R=machenbach@chromium.org
No-Try: true
No-Tree-Checks: true
Change-Id: I1d4373b8846034cf28b5fd02e60e9ed682330706
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1605942
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61607}
When retries succeed, the perfrunner masks the timeouts.
This CL exposes those failures.
R=sergiyb@chromium.org
Bug: chromium:841700
Change-Id: Ia69a191e84fd994bd25deb2d7019f7acf11db45c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1602695
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61384}
Overall, total test runtime that was wasted due to timeouts is 3420 seconds in
the last 2 weeks. Even with 4 retries, assuming all of them time out, needed
additional capacity is under 2 hours per week. Based on this analysis, I think
it's safe to land this CL.
Note that this is not intended as a long-term solution of the timeout problem,
but rather a temporary solution to prevent ongoing errors. Proper investigation
and correct long-term solution are still needed and tracked in the bug.
R=machenbach@chromium.org, tmrts@chromium.org
Bug: chromium:841700
Change-Id: Id16e6b784fa85bb9e28ed8c6b267b583636e2dc1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1593342
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61224}
This is needed to allow deriving timeout and near_timeout runnables on the
recipe side and drop corresponding fields in the output.
We also remove some unused code that was removed in previous refactoring CLs.
R=tmrts@chromium.org
No-Try: true
No-Tree-Checks: true
Bug: chromium:841700
Change-Id: Ib7cec9baceff994fb9b32be09ba4a4079ebff2c8
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1588417
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61095}
This also makes processing immediate, i.e. outputs are parsed into results as
soon as test run is finished, which will allow us to implement logic that checks
whether we have enough runs based on already-accumulated results.
Since we process each output immediately, we do not need Measurement class any
longer and its ConsumeOutput is now integrated directly into TraceConfig.
Similarly AccumulateResults is replaced with RunnableConfig.ProcessOutput as we
do not accumulate results any longer.
R=machenbach@chromium.org
No-Try: true
No-Tree-Checks: true
Bug: chromium:880724
Change-Id: I0fc4846024c43258c10ba8d568312aa4746d746f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1584325
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61048}
This is part of the refactoring needed to implement a feature that allows
re-running benchmarks until needed confidence is reached.
R=machenbach@chromium.org
No-Try: true
No-Tree-Checks: true
Bug: chromium:880724
Change-Id: I45f584a3503ecf567f4c2661a302a74fc5e516af
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1581605
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#61037}
The --run-count-multiplier flag was used to run an experiment and is not needed
anymore. It will be superceeded by another parameter that will run benchmarks as
many times as needed until needed confidence is achieved.
R=machenbach@chromium.org
No-Try: true
No-Tree-Checks: true
Bug: chromium:880724
Change-Id: Iedd3d0f900a838f97cfa3339f582f6d02cce3e93
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1578504
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60980}
This is dead code as no benchmarks seem to be using it and removing this
additional logic will reduce maintenance cost for the perf infrastructure.
R=machenbach@chromium.org, tmrts@chromium.org
No-Try: true
No-Tree-Checks: true
Change-Id: I952af069d4d18d63624510b4c461fa1891703db9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1570008
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60947}
This is part of the refactoring to allow exporting more information about test
execution to the recipes and upload this information to ChromePerf.
This fixes running secondary even after primary run fails, which will allow us
to differentiate between test and infra failures as latter ones will also affect
refbuilds and re-runs without patch.
R=machenbach@chromium.org, tmrts@chromium.org
Bug: chromium:841700
Change-Id: I29ce49d2f8c5e73158f1d41a73c51f2b35929f36
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1570006
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60946}
This is part of the refactoring to allow exporting more information about
test execution to the recipes and upload this information to ChromePerf.
R=machenbach@chromium.org,tmrts@chromium.org
No-Try: true
No-Tree-Checks: true
Bug: chromium:841700
Change-Id: Iab400e8922231d8eac91a6fa22ce8f45053f7ac6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1569442
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60944}
We've originally added them in https://crrev.com/c/1159361 and then removed
again in https://crrev.com/c/1291370 since they were implemented in LogDog
Viewer. However, looks like logs from the swarming tasks are all dumped into
step stdout at the same time and thus logdog assigns idential timestamp to all
lines making it fairly useless.
TBR=machenbach@chromium.org
No-Try: true
No-Tree-Checks: true
Bug: chromium:841700
Change-Id: I49bfb23f8007e4a0facdfea6a09cde911f8e5a51
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1569432
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60875}
This is not fixing the root cause of the flakiness, but prevents us from loosing
data in the short to medium term as investigation proved to be difficult and
will likely take even more time.
R=machenbach@chromium.org, tmrts@chromium.org
Bug: chromium:872257
Change-Id: Id5fbd0a00058f8612089ee4d6a858193924bd868
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1564204
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60814}
- Remove all relative imports from mock and os
- Fix mocking in a few tests to prevent cross-test side-effects
- Add run_perf_test.py to v8_presubmit.py
- The vpython config was not added since root .vpython already includes
coverage and mock libraries
- Convert all double-quoted strings to single-quoted (PS8->PS9)
R=sergiyb@chromium.org
Bug: chromium:123456
Change-Id: I7b3a08dc5d950b0f51cc7a5eb3a012ea953ca824
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1564206
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#60810}
There are now less that 400 days until the end of life
of Python 2(aka _legacy_ Python) https://pythonclock.org/ .
The code compatibility check for python2 and python3
used the following tools: futurize, flake8
You can see the reports here: https://travis-ci.com/bmsdave/v8/builds
This CL was uploaded by git cl split.
Bug: v8:8594
Change-Id: I661c52a70527e8ddde841fee6d4dcba282b4a938
Reviewed-on: https://chromium-review.googlesource.com/c/1470123
Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59675}
This reverts commit 6e03d7ee42.
Reason for revert: This breaks the Android bot:
https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Android%20Arm64%20-%20N5X/2933
It's quite hidden due to https://crbug.com/v8/8731 but all shards
time out. It looks like to to this change, testing takes
dramatically longer, maybe the pushing takes now much longer than
before. If we want decide for this, the builder needs to get
many more shards.
Original change's description:
> Reland "[tools] Push files using high-level device.PushChangedFiles method"
>
> This is a reland of d045f66682
>
> Original change's description:
> > [tools] Push files using high-level device.PushChangedFiles method
> >
> > R=machenbach@chromium.org
> >
> > No-Try: true
> > Bug: chromium:893593
> > Change-Id: I11cce7694eb7755ccee42c9a342fc1aa22663d85
> > Reviewed-on: https://chromium-review.googlesource.com/c/1382468
> > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#58407}
>
> Bug: chromium:893593
> Change-Id: I88a7143b3f31d87d266b89221f81efe831ea3823
> Reviewed-on: https://chromium-review.googlesource.com/c/1443055
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#59221}
TBR=machenbach@chromium.org,tandrii@chromium.org,sergiyb@chromium.org,bpastene@chromium.org,jbudorick@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: chromium:893593
Change-Id: Ifea307b5de8f39b660966fc6bef54601df91d841
Reviewed-on: https://chromium-review.googlesource.com/c/1450119
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59305}
We define a TestFailedError exception and raise it when we can reliably detect
that a test has crashed. All other exceptions are treated as infra failures and
are captured by the try-catch clause in MainWrapper function.
This also fixes all tests in run_perf_test.py, run_tests_test.py and makes sure
that both are run on any changes in tools directory.
R=machenbach@chromium.org
Bug: chromium:899028
Change-Id: I283bc87b31c814be476bebe9fdda414975494183
Reviewed-on: https://chromium-review.googlesource.com/c/1303293
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57134}
Originally timestamps were added in crrev.com/c/1159361 to ease debugging. Since
then, however, timestamps were introduce to the LogDog viewer and are rendered
in a separate column, making it easier to scan text for important messages
starting with '>>>'. We do keep the log level, but make sure that it always
takes a fixed width such that the actual messages are aligned vertically.
R=machenbach@chromium.org
No-Try: true
Change-Id: I3a741069824aa25e674aca79da03a725017cb956
Reviewed-on: https://chromium-review.googlesource.com/c/1291370
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56834}
This will allow us to determine whether tests are not printing anything or
whether our infrastructure is failing to retrieve the stdout.
TBR=machenbach@google.com
No-Try: true
Bug: chromium:891314
Change-Id: I0786e77b4db9f247e02a25924acd6d6773c0db9f
Reviewed-on: https://chromium-review.googlesource.com/c/1282962
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56701}
This is a reland of 4c0943424c
Original change's description:
> [test] Add logic to run tests on Android
>
> This adds a new command abstraction for running commands on Android
> using dockered devices on swarming.
>
> The new abstraction handles pushing all required files to the device.
> The logic used for pushing and running is reused from the perf runner.
>
> This adds only the mjsunit test suite. Others will be handled in
> follow up CLs. The suite logic is enhanced with auto-detection of files
> to be pushed to devices, for e.g. load or import statements.
>
> Some test cases need an extra resource section for specifying required
> files.
>
> Remaining failing tests are marked in the status files for later
> triage.
>
> Bug: chromium:866862
> Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
> Reviewed-on: https://chromium-review.googlesource.com/1150153
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55041}
Bug: chromium:866862
Change-Id: Icf7e04c75d4abeab7254d10ba21240e46b0022ae
Reviewed-on: https://chromium-review.googlesource.com/1170643
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55059}
