skia2/tools/skqp/README_ALGORITHM.md

SkQP Render Test Algorithm
==========================

The following is a description of the render test validation algorithm that
will be used by the version of SkQP that will be released for Android Q-release.

There is a global macro constant: `SK_SKQP_GLOBAL_ERROR_TOLERANCE`, which
reflects the `gn` variable `skia_skqp_global_error_tolerance`.  This is usually
set to 8.

First, look for a file named `skqp/rendertests.txt` in the
`platform_tools/android/apps/skqp/src/main/assets` directory.  The format of
this file is:  each line contains one render test name, followed by a comma,
followed by an integer.  The integer is the `passing_threshold` for that test.

For each test, we have a `max_image` and a `min_image`.  These are PNG-encoded
images stored in SkQP's APK's asset directory (in the paths `gmkb/${TEST}/min.png`
and `gmkb/${TEST}/max.png`).

The test input is a rendered image.  This will be produced by running one of
the render tests against the either the `vk` (Vulkan) or `gles` (OpenGL ES)
Skia backend.

Here is psuedocode for the error calculation:

    function calculate_pixel_error(pixel_value, pixel_max, pixel_min):
        pixel_error = 0

        for color_channel in { red, green, blue, alpha }:
            value = get_color(pixel_value, color_channel)
            v_max = get_color(pixel_max,   color_channel)
            v_min = get_color(pixel_min,   color_channel)

            if value > v_max:
                channel_error = value - v_max
            elif value < v_min:
                channel_error = v_min - value
            else:
                channel_error = 0
            pixel_error = max(pixel_error, channel_error)

        return max(0, pixel_error - SK_SKQP_GLOBAL_ERROR_TOLERANCE);

    function get_error(rendered_image, max_image, min_image):
        assert(dimensions(rendered_image) == dimensions(max_image))
        assert(dimensions(rendered_image) == dimensions(min_image))

        max_error = 0
        bad_pixels = 0
        total_error = 0

        error_image = allocate_bitmap(dimensions(rendered_image))

        for xy in list_all_pixel_coordinates(rendered_image):
            pixel_error = calculate_pixel_error(rendered_image(xy),
                                                max_image(xy),
                                                min_image(xy))
            if pixel_error > 0:
                for neighboring_xy in find_neighbors(xy):
                    if not inside(neighboring_xy, dimensions(rendered_image)):
                        continue
                    pixel_error = min(pixel_error,
                                      calculate_pixel_error(rendered_image(xy),
                                                            max_image(neighboring_xy),
                                                            min_image(neighboring_xy)))

            if pixel_error > 0:
                max_error = max(max_error, pixel_error)
                bad_pixels += 1
                total_error += pixel_error

                error_image(xy) = linear_interpolation(black, red, pixel_error)
            else:
                error_image(xy) = white

        return ((total_error, max_error, bad_pixels), error_image)

For each render test, there is a threshold value for `total_error`, :
`passing_threshold`.

If `passing_threshold >= 0 && total_error > passing_threshold`, then the test
is a failure and is included in the report.  if `passing_threshold == -1`, then
the test always passes, but we do execute the test to verify that the driver
does not crash.

We generate a report with the following information for each test:

    backend_name,render_test_name,max_error,bad_pixels,total_error

in CSV format in the file `out.csv`.  A HTML report of just the failing tests
is written to the file `report.html`.  This version includes four images for
each test:  `rendered_image`, `max_image`, `min_image`, and `error_image`, as
well as the three metrics: `max_error`, `bad_pixels`, and `total_error`.