With the old logic, a suppression shows up in the statistics independent if the test cases caused a difference or not. This doesn't give a signal if a suppression is useful. The new logic will help cleaning up suppressions that never apply.
BUG=chromium:673246
NOTRY=true
R=tandrii@chromium.orgTBR=mstarzinger@chromium.org,jarin@chromium.org
Change-Id: Iaebdac475f408f7d2649a34ccaa580c8d91e34a5
Reviewed-on: https://chromium-review.googlesource.com/437264
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#42932}
This adds optional multi-architecture builds, allowing to compile
x86 and x64 in one build. The correctness fuzzer can be configured to
compare the two executables, e.g. to compare x86 to x64 run the
launcher with: --second-d8=clang_x86/d8 in an x64 build.
Configuring the executable's architecture is now simplified and
inferred from the gn build configuration.
Building for clusterfuzz has now a new canonical target that can be
used by the infrastructure (defaults to d8).
The clusterfuzz release builder is now defined to compile
multi-arch builds, which will have an effect as soon as the
infrastructure refers to the new clusterfuzz target.
BUG=chromium:673246
NOTRY=true
TBR=mstarzinger,jarin
Review-Url: https://codereview.chromium.org/2649133010
Cr-Commit-Position: refs/heads/master@{#42884}
The results are too noisy ATM. This switches off validate-asm for
default comparisons. We can add back dedicated jobs later that switch
it on.
BUG=chromium:663714
NOTRY=true
TBR=bradnelson@chromium.org,titzer@chromium.org
Review-Url: https://codereview.chromium.org/2640743004
Cr-Commit-Position: refs/heads/master@{#42467}
The new ignition config will be used as baseline comparison in new
jobs, e.g. against ignition_turbo. We'll keep --validate-asm off
in ignition_turbo for now as it is very chatty.
BUG=chromium:673246
NOTRY=true
Review-Url: https://codereview.chromium.org/2640043002
Cr-Commit-Position: refs/heads/master@{#42464}
Continuation of:
https://codereview.chromium.org/2620343005/
This removes usage of metadata files entirely. Instead we extract
the instrumentation about source files from the test cases.
This also adds extra output of the original source file in the
detailed failure text for easier debugging. The hashes alone
made it hard to reason.
BUG=chromium:673246
NOTRY=true
TBR=tandrii@chromium.org,mbarbella@chromium.org
Review-Url: https://codereview.chromium.org/2634743004
Cr-Commit-Position: refs/heads/master@{#42371}
The fuzz test cases now print the original test paths during execution.
This exploits this extra information and reports a hash of only one
original source file from the section that caused a difference.
The hash size is now limited to 3 to avoid possible duplicate
explosion, in case this doesn't work out as expected.
This prepares for patch 3 of:
https://chromereviews.googleplex.com/550337016/
BUG=chromium:673246
NOTRY=true
TBR=tandrii@chromium.org,mbarbella@chromium.org
Review-Url: https://codereview.chromium.org/2620343005
Cr-Commit-Position: refs/heads/master@{#42305}
Reason for revert:
Setting the BAD_BUILDS_CHECK env variable should
be enough. Lets revert this as the code doesn't
look particularly good. There must be another
problem still that needs investigation.
Original issue's description:
> [foozzie] Pass startup test
>
> This makes sure the harness passes when called without test case.
>
> BUG=chromium:673246
> NOTRY=true
> TBR=tandrii@chromium.org,mbarbella@chromium.org
>
> Review-Url: https://codereview.chromium.org/2623743004
> Cr-Commit-Position: refs/heads/master@{#42183}
> Committed: ede2fc1ff0TBR=tandrii@chromium.org,mbarbella@chromium.org,mmoroz@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=chromium:673246
Review-Url: https://codereview.chromium.org/2628493003
Cr-Commit-Position: refs/heads/master@{#42189}
This makes sure the metadata is found during minimization. Also renames
the test files to fit the naming pattern.
BUG=chromium:673246
NOTRY=true
TBR=tandrii@chromium.org,mbarbella@chromium.org
Review-Url: https://codereview.chromium.org/2622653002
Cr-Commit-Position: refs/heads/master@{#42150}
Initial version of the correctness fuzzer harness for manual testing
and experiments.
For automated usage, some outstanding TODOs are left in the code. E.g.
- Hash source file names in error case
- Bundle script in out directory with executables
- Some suppressions are tied to already fixed bugs. We'll keep it like that for now to test
removing those suppressions in production later.
BUG=chromium:673246
NOTRY=true
Review-Url: https://codereview.chromium.org/2578503003
Cr-Commit-Position: refs/heads/master@{#41789}
