tldr: This adds an on-demand comparison with x64 when a difference to
a non-x64 build is detected.
Normally foozzie compares the baseline build (just ignition), with
two secondary builds. One, the default, always uses the shipping
configuration, the second passes additional flags. Both can use a
different architecture than the baseline build as well.
Differences between ignition and turbofan are then often detected
independent of the architectures used, but reported several times
(for each compared architecture).
This makes the reporting more specific, by running another build on
demand that uses the baseline architecture, but otherwise the same
configuration that showed a difference. If it shows the difference as
well, the baseline architecture is used for the report.
As a result only pure architecture differences will be reported with
an architecture other than x64.
This also adds some minor refactorings to reduce the code complexity
when looping over comparisons.
For testing this, the fake-d8s are extended with different behavior
for different flags passed. We add two test cases for testing:
x64 vs. ia32 with difference in x64 and ia32
x64 vs. ia32 with difference only in ia32
Bug: chromium:1196633
No-Try: true
Test: tools/clusterfuzz/v8_foozzie_test.py
Change-Id: Ic470ae8f0b37fb1628b32e4fafc0c39377e16f8c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2897099
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#74612}
Previously we ran baseline (e.g. ignition) and one random secondary
comparison configuration (e.g. turbofan) from the list of experiments.
But Clusterfuzz imposes limitations on the total amount of fuzz tests.
Therefore this change enables more throughput by always running the
default configuration (ignition_turbofan like V8 is shipped)
additionally to the baseline and the secondary configuration.
This, hence, doubles the number of comparisons we run, with less than
50% additional runtime, since the slow baseline configuration is only
run once.
The experiments table is updated accordingly. Explicit entries running
ignition_turbofan are removed (as it always runs now), instead some
of the other configurations are increased in their relative
percentage. We also get a few new configurations that didn't run
before (e.g. forcing the slow path on x86).
No-Try: true
Bug: chromium:1100114
Change-Id: I69b2a41d78c06e556b309743a2aace1053c22f91
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270307
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68607}
This makes output and test-case suppressions independent of the used
comparison configs and architecture. Such fine-grained suppressions
were only needed during the inception of differential fuzzing, but
by now, most remaining suppressions are implemented in d8 behind
a flag.
This prepares for running with more than two comparison configs in a
follow up.
No-Try: true
Bug: chromium:1100114
Change-Id: I072769adb3ef7c6e6c43459aa23ac906f461b307
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270095
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68579}
It is obsolete to filter out error-message differences since the
time we pass --correctness-fuzzer-suppressions to d8, which already
stubs all messages:
https://cs.chromium.org/chromium/src/v8/src/execution/messages.cc?l=1031
No-Try: true
Bug: chromium:1100114
Change-Id: Iac42a8e2a32f9bae4034f79eaff429bf3ee41724
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270024
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68577}
This simplifies the lengthy main method by extracting some code and
by replacing the scattered returns with exceptions.
We introduce two exceptions for early bail-out. This enables helper
methods on multiple layers. The early bail-out on time-out is
moved to the point where it is detected.
Previously on timeout and crash we also printed out the step number.
Clusterfuzz doesn't parse this, it was only for statistical purposes,
and the latest version of the experimental workbench only parses
crashes and timeouts, not the step in which they happened. Hence,
this CL removes those step numbers.
Except the change described in the last paragraph, this CL doesn't
intend to change behavior.
No-Try: true
Bug: chromium:1100114
Change-Id: Ie8c18f183e4fc538577f3eb49aaf6df1acd1e4e1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2270547
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68576}
This prepares using ochang_js_fuzzer with foozzie. The fuzzer uses
tests from CrashTests in the corpus. This leads to a loop when
used with differential fuzzing, as foozzie dedupes failures based
on the original file path. Foozzie finds a new failure for the
existing failure in CrashTests, for which clusterfuzz creates a new
crash test and so on.
This subsumes all failures from CrashTests under the same key.
Once such a failure is reported, a developer can add it to a
mapping in foozzie.py, after which the global key can be used
again by clusterfuzz to report another failure.
No-Try: true
Bug: chromium:1044942
Change-Id: I801a23faeb0c672d6ad64b4100c463f53e36cbc2
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2214837
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Maya Lekova <mslekova@chromium.org>
Cr-Commit-Position: refs/heads/master@{#68053}
Pass --no-lazy-feedback-allocation in all second runs depending
on a probability. Also combine with --interrupt-budget=100.
This also allows adding several extra flags behind one probability.
The tests are improved to ensure valid flags and configs.
No-Try: true
Bug: v8:10215
Change-Id: I2766ef5044cd8c7096f6b76f39b60b568f550bde
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2059991
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66289}
The assumtion that V8 has no output differences within a single line
before a stack overflow, didn't hold. The prefix of e.g. console.info
can lead to a difference in a recursive call.
This change makes foozzie's output capping before a crash work on the
level of characters instead of lines to fix this.
No-Try: true
Bug: chromium:1050942
Change-Id: I13f747caf4f5848d40c31bd4232811285bab3c17
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2049844
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66217}
This will allow uploading repro test cases to clusterfuzz for
already suppressed known issues. This will allow tracking if those
issues still reproduce and that suppressions don't become stale.
No-Try: true
Bug: chromium:1044942
Change-Id: I997f11293c51836b97d143b0fea992055b39955e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2036083
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66114}
Crashes in the presence of RangeError happen often during differential
fuzzing. Until now we have ignored such cases completely.
After this change we compare as much output as possible when one or
both runs have crashed, dramatically increasing the coverage.
No-Try: true
Bug: chromium:1048099
Change-Id: I923c10e9064b5dc6cae1e39a254e221d2867e0e7
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2030914
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#66085}
This adds a regresson test case for the revert reason of:
https://crrev.com/c/1906378
The test data is tidied up by keeping the different fake d8s in
separate build directories like it would be in production.
A new test simulates an architecture difference and ensures we
pass the architecture mocks in all runs.
No-Try: true
Bug: chromium:1023091
Change-Id: Ic33c426ba8eb9c4b6b0fbb66d43c0859dc2edfcd
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1918248
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#65140}
We used the same random seed for all test cases of a fuzz session
for transitioning from choosing the flags on V8 side.
Since the grace period for stable bisection is over, we now use
the same random number generator throughout the fuzz session which
leads to a wider range of differently chosen flags.
TBR=tmrts@chromium.org
No-Try: true
Bug: chromium:813833
Change-Id: I07b9fe5de378c01344afd486bfd85fcbf0fcd8d2
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1906377
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64910}
This adds a fake toolchain for pointer compression, used for
correctness fuzzing. The toolchain enables us to have an extra build
with inverse pointer-compression defaults side-by-side.
The extra build is used similarly to existing x64/x86 comparisons,
except that we now compare builds with different compile-time flags.
Change-Id: I75491371262204b86eaa006ca8d04848f49121ac
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1829275
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#64123}
Currently, probabilities for extra flags are calculated in the correctness
fuzzer harness, which makes the RNG fragile when bisecting backwards, when
the script's config changes during bisection.
This adds the possibility to pass extra flags on command line to the
script. After a grace period, we will migrate the flag calculation to
clusterfuzz.
NOTRY=true
Bug: chromium:813833
Change-Id: I515181847474515089b847f8aaffc7c6560d9390
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1675945
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#62359}
We have too many dupes in the no-ic comparisons. We'll increase the
experiment size again once bugs are fixed.
TBR=jarin@chromium.org
NOTRY=true
Bug: chromium:961709
Change-Id: Ic946100b45fd73e1bee59f188a766384836bcdcf
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1660624
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#62212}
This lets foozzie call d8 with sanity output before doing the actual
correctness comparisons. This will make clusterfuzz dedupe cases on
the difference found in the sanity checks.
Also adding missing OWNERS file.
NOTRY=true
Bug: chromium:933076
Change-Id: I4229183726064cc0ad76da8fe432e1dbb601a7ba
Reviewed-on: https://chromium-review.googlesource.com/c/1491221
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Tamer Tas <tmrts@chromium.org>
Cr-Commit-Position: refs/heads/master@{#59938}
Instead of having a separate liftoff config, which is tested against
the default (which currently means tier-up from liftoff to turbofan),
just choose reasonable liftoff configs for the existing configs.
'ignition' now implies pure liftoff execution.
'ignition_turbo_opt' always compiles with turbofan.
Other configs use the default (tier up).
R=machenbach@chromium.org
Bug: chromium:824098, v8:6600
Change-Id: I92c008fc1b1fa54d3161fb5695a095127d6ac263
Reviewed-on: https://chromium-review.googlesource.com/1141731
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54548}
This adds 5% testing of 'ignition' vs 'liftoff', which tests Turbofan vs
Liftoff for wasm code, and tests Ignition vs Turbofan for javascript
code.
It also adds 3% testing of 'liftoff' (x64) vs 'liftoff' (ia32), which
does standard x64 vs ia32 testing for javascript code.
R=machenbach@chromium.org
Bug: chromium:824098, v8:6600
Change-Id: I6a6afae0300efc33f3535541a11695a7bb32dcc5
Reviewed-on: https://chromium-review.googlesource.com/973161
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52111}
We'll soon also host other configurations for general fuzzing, not only
correctness fuzzing in the new tools/clusterfuzz folder.
TBR=yangguo@chromium.org
Bug: chromium:813833
Change-Id: Icd966bfec91cc547522bad5d1a842500b554754f
Reviewed-on: https://chromium-review.googlesource.com/930331
Reviewed-by: Yang Guo <yangguo@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#51480}
