This makes output and test-case suppressions independent of the used
comparison configs and architecture. Such fine-grained suppressions
were only needed during the inception of differential fuzzing, but
by now, most remaining suppressions are implemented in d8 behind
a flag.
This prepares for running with more than two comparison configs in a
follow up.
No-Try: true
Bug: chromium:1100114
Change-Id: I072769adb3ef7c6e6c43459aa23ac906f461b307
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270095
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68579}
It is obsolete to filter out error-message differences since the
time we pass --correctness-fuzzer-suppressions to d8, which already
stubs all messages:
https://cs.chromium.org/chromium/src/v8/src/execution/messages.cc?l=1031
No-Try: true
Bug: chromium:1100114
Change-Id: Iac42a8e2a32f9bae4034f79eaff429bf3ee41724
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270024
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68577}
This simplifies the lengthy main method by extracting some code and
by replacing the scattered returns with exceptions.
We introduce two exceptions for early bail-out. This enables helper
methods on multiple layers. The early bail-out on time-out is
moved to the point where it is detected.
Previously on timeout and crash we also printed out the step number.
Clusterfuzz doesn't parse this, it was only for statistical purposes,
and the latest version of the experimental workbench only parses
crashes and timeouts, not the step in which they happened. Hence,
this CL removes those step numbers.
Except the change described in the last paragraph, this CL doesn't
intend to change behavior.
No-Try: true
Bug: chromium:1100114
Change-Id: Ie8c18f183e4fc538577f3eb49aaf6df1acd1e4e1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270547
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68576}
If we pass flags that make runs very slow, also the validity checks
might time out. Previously this wasn't checked and output was just
cut off.
This also tightens the timeout on validity checks as they are
expected to run very fast.
No-Try: true
Bug: chromium:1098646
Change-Id: Iea9a932be86e84040b72a2311aaa1d44100b3378
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2262915
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68510}
This fixes a bug when an encoded character appears in the difference
string. Python3 doesn't require any encoding.
TBR=tmrts@chromium.org
No-Try: true
Bug: chromium:1095964
Change-Id: I49c66b5b9c105ad64d3a7839d0eb5df97ff5f404
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2249660
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68384}
This prepares using ochang_js_fuzzer with foozzie. The fuzzer uses
tests from CrashTests in the corpus. This leads to a loop when
used with differential fuzzing, as foozzie dedupes failures based
on the original file path. Foozzie finds a new failure for the
existing failure in CrashTests, for which clusterfuzz creates a new
crash test and so on.
This subsumes all failures from CrashTests under the same key.
Once such a failure is reported, a developer can add it to a
mapping in foozzie.py, after which the global key can be used
again by clusterfuzz to report another failure.
No-Try: true
Bug: chromium:1044942
Change-Id: I801a23faeb0c672d6ad64b4100c463f53e36cbc2
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2214837
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68053}
This prevents bug flooding based on differences from calling
%GetOptimizationStatus in correctness tests. It is supposed to
be suppressed with --allow-natives-for-differential-fuzzing.
This ensures early bail-out in case the flag is forgotten at
some point. The v8_sanity_checks.js file is executed before
each correctness test case for this purpose.
NOTRY=true
Bug: chromium:1044942
Change-Id: I74a836a82562604b35e94e5e123a2a8bff939423
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2184294
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#67618}
A new realm doesn't contain the mocks and suppressions on the global
object for correctness fuzzing. We replace Realm.eval with eval to
keep exercising the code.
Bug: chromium:1071133
Change-Id: Iffe82d37bf08829fc5937c17c2089277403e71dc
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2153206
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#67187}
Yet another corner case how non-deterministic timestamps slipped into
the tests.
Bug: chromium:1064900
Change-Id: I33e8b4c8141b3854b7eca5d7ad9b45b6f5130d9e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2120584
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66881}
This adds tests for the mock logic used in differential fuzzing. The
tests uncovered a couple of issues in the mock files that are also
fixed.
This also does some minor code clean up in the mock code.
Bug: chromium:1044942
Change-Id: I5b67f70f8b104bb681548f742ab863395a88360f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2043843
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66304}
This reduces the pointer-compression experiment by 3% as it is in
production since a while.
This also uses turbofan when comparing across architectures as tests
run faster and likely find the same or more relevant bugs.
No-Try: true
Change-Id: I35ec97188a101a7cb43454903f7bf9765e57893c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2059995
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66290}
Pass --no-lazy-feedback-allocation in all second runs depending
on a probability. Also combine with --interrupt-budget=100.
This also allows adding several extra flags behind one probability.
The tests are improved to ensure valid flags and configs.
No-Try: true
Bug: v8:10215
Change-Id: I2766ef5044cd8c7096f6b76f39b60b568f550bde
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2059991
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66289}
This will allow sharing the configs between Python and JavaScript
based fuzzers.
No-Try: true
Bug: chromium:1044942
Change-Id: Idd47ad31430cfed0629dbb5b9dd5fd2ef8c764f6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2051951
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66243}
We ensure now that fuzzers don't produce the f.arguments pattern and
instead replace it with a random variable.
No-Try: true
Bug: chromium:1044942, chromium:1020573
Change-Id: I899985f1a238f36d10fb22f76d93b7d5f444eab1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2051944
Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66238}
Un-ignore some files that only failed in obsolete comparisons.
We don't compare eager anymore and console functions are mocked out
in d8 by now. The exponentiation operator bug doesn't repro anymore.
No-Try: true
Bug: chromium:1044942
Change-Id: I0a572836bc9a4a6aa4736447f638a522ff8e8168
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2050400
Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66233}
The assumtion that V8 has no output differences within a single line
before a stack overflow, didn't hold. The prefix of e.g. console.info
can lead to a difference in a recursive call.
This change makes foozzie's output capping before a crash work on the
level of characters instead of lines to fix this.
No-Try: true
Bug: chromium:1050942
Change-Id: I13f747caf4f5848d40c31bd4232811285bab3c17
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2049844
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66217}
The bug appears to be fixed.
No-Try: true
Bug: chromium:693426
Change-Id: Ibd029c906dca87a3fd0c3e00be0b795e87390bd4
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2037439
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66134}
This will allow uploading repro test cases to clusterfuzz for
already suppressed known issues. This will allow tracking if those
issues still reproduce and that suppressions don't become stale.
No-Try: true
Bug: chromium:1044942
Change-Id: I997f11293c51836b97d143b0fea992055b39955e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2036083
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66114}
Respective differences were fixed and the suppressions didn't fire
anymore according to the statistics.
No-Try: true
Bug: chromium:664068, chromium:667678
Change-Id: I5e1aa802e0b0288a73d0f236c154289ab45314a0
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2035878
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66100}
Crashes in the presence of RangeError happen often during differential
fuzzing. Until now we have ignored such cases completely.
After this change we compare as much output as possible when one or
both runs have crashed, dramatically increasing the coverage.
No-Try: true
Bug: chromium:1048099
Change-Id: I923c10e9064b5dc6cae1e39a254e221d2867e0e7
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2030914
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66085}
This adds two flags for whitelisting natives on fuzzers:
--allow-natives-for-fuzzing:
Enables a small subset of runtime functions and replaces others with
undefined.
--allow-natives-for-differential-fuzzing
Restricts the allowed runtime functions even further, excluding the
ones that break differential fuzzing.
Bug: chromium:1044942
Change-Id: I890bd4a6ff7c2e483dd74155290a7ace06f85239
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2020773
Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66068}
This is a cleanup to remove unneeded flags after these changes (in
https://crrev.com/c/1988548):
* --future does not imply --wasm-tier-up any more, and
* --wasm-tier-up does not imply --liftoff any more.
Instead, now
* --wasm-tier-up is enabled by default,
* --wasm-tier-up has no effect if --liftoff is not set, and
* --future implies --liftoff.
R=ahaas@chromium.org
Bug: chromium:1040061
Change-Id: I5d04ee1f1d84ddcd0654df0e0a4c6298f80aee9e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1993280
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#65666}
Some test cases stress all global functions and hang when calling
readline.
No-Try: true
Bug: chromium:1034285
Change-Id: I0eaf64437c0806cf3df0a5306aecea4367763cad
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1967381
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#65460}
Some fuzz test cases set Math.min to a different function. Let the fake
typed array constructors remember the original Math.min.
No-Try: true
Bug: chromium:1024815
Change-Id: Ia389439fe6757f2497e22187042c5cdf1d1c649f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1958050
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#65383}
This adds a regresson test case for the revert reason of:
https://crrev.com/c/1906378
The test data is tidied up by keeping the different fake d8s in
separate build directories like it would be in production.
A new test simulates an architecture difference and ensures we
pass the architecture mocks in all runs.
No-Try: true
Bug: chromium:1023091
Change-Id: Ic33c426ba8eb9c4b6b0fbb66d43c0859dc2edfcd
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1918248
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#65140}
This adds an abstraction for command-line arguments for each of the
two comparison runs done in correctness fuzzing. No functional
changes intended.
No-Try: true
Bug: chromium:1023091
Change-Id: I9421715c4904416b9aaf53848954a5248c79ffd9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1906372
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64979}
This is a reland of 1d493d31ce
Original change's description:
> [foozzie] Refactor command abstraction
>
> This moves code for running d8 into its own class. No functional
> changes intended.
>
> No-Try: true
> Bug: chromium:1023091
> Change-Id: I7cbfeebd2911dc758322f89cf93666550f2956d9
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1906378
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Tamer Tas <tmrts@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#64928}
Bug: chromium:1023091
Change-Id: I7df6e12084e20510a400ce209827c2bba8325f86
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1914209
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64966}
This moves code for running d8 into its own class. No functional
changes intended.
No-Try: true
Bug: chromium:1023091
Change-Id: I7cbfeebd2911dc758322f89cf93666550f2956d9
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1906378
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64928}
We used the same random seed for all test cases of a fuzz session
for transitioning from choosing the flags on V8 side.
Since the grace period for stable bisection is over, we now use
the same random number generator throughout the fuzz session which
leads to a wider range of differently chosen flags.
TBR=tmrts@chromium.org
No-Try: true
Bug: chromium:813833
Change-Id: I07b9fe5de378c01344afd486bfd85fcbf0fcd8d2
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1906377
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64910}
This adds a fake toolchain for pointer compression, used for
correctness fuzzing. The toolchain enables us to have an extra build
with inverse pointer-compression defaults side-by-side.
The extra build is used similarly to existing x64/x86 comparisons,
except that we now compare builds with different compile-time flags.
Change-Id: I75491371262204b86eaa006ca8d04848f49121ac
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1829275
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64123}
