A set of scripts that analyzes jobs.json to look for possible holes in our

testing.

+rmistry for reviewing the scripts
+egdaniel for reviewing the Vulkan outpout.

Below are the three types of analysis done so far, showing
Test, Perf, and Vulkan jobs that are not being run:


$ make missing_test_jobs
cpu_or_gpu_value,model
Snapdragon808,Nexus5x


$ make missing_perf_jobs
cpu_or_gpu_value,model
AVX,VMware7.1
Rome,GCE
Snapdragon821,Pixel
SwiftShader,GCE


$ make missing_vulkan_jobs
cpu_or_gpu_value,model
AVX,VMware7.1
AVX2,GCE
AVX2,MacBookPro11.5
AVX2,NUC5i7RYH
AVX512,GCE
AVX512,Golo
Adreno330,Nexus5
Adreno418,Nexus5x
AppleA11,iPhone8
AppleA13,iPhone11
AppleM1,MacMini9.1
IntelBayTrail,NUCDE3815TYKHE
IntelHD2000,ShuttleA
IntelHD4400,NUCD34010WYKH
IntelHD6000,MacBookAir7.2
IntelHD615,MacBook10.1
IntelIris5100,MacMini7.1
IntelIris6100,NUC5i7RYH
IntelUHDGraphics605,Sparky360
Mali400MP2,AndroidOne
MaliT760,GalaxyS6
PowerVRGE8320,TecnoSpark3Pro
PowerVRGT7600,iPhone7
PowerVRGT7800,iPadPro
PowerVRGX6450,iPhone6
RadeonHD8870M,MacBookPro11.5
RadeonVega3,Spin514
Rome,GCE
Snapdragon800,Nexus5
Snapdragon808,Nexus5x
Snapdragon821,Pixel
SwiftShader,GCE
Tegra3,Nexus7

Change-Id: I18c1688fa20c73bfbaf36221596d7784dc0f1212
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/414442
Commit-Queue: Joe Gregorio <jcgregorio@google.com>
Reviewed-by: Ravi Mistry <rmistry@google.com>

infra/bots/analysis/Makefile

@@ -0,0 +1,8 @@
+missing_perf_jobs:
+	./missing.sh '$$Type == "Perf"'
+
+missing_test_jobs:
+	./missing.sh '$$Type == "Test"'
+
+missing_vulkan_jobs:
+	./missing.sh '$$vulkan == "true"'

infra/bots/analysis/README.md

@@ -0,0 +1,23 @@
+# Job Analysis
+
+A set of scripts that analyzes `jobs.json` to look for possible holes in our
+testing.
+
+## Requirements
+
+To run the scripts you need to have both `jq` and `mlr` installed on your
+machine.
+
+    $ sudo apt install jq miller
+
+## Running
+
+The Makefile contains common queries that can be run against the data.
+
+For example, to find all cpu_or_gpu_values that we currently don't run Perf
+tests on you would run:
+
+    $ make missing_perf_jobs
+
+See https://miller.readthedocs.io/en/latest/reference-dsl.html more details on
+the the kinds of queries that can be done against CSV files.

infra/bots/analysis/axis.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# Copyright 2021 Google Inc.
+#
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Creates a CSV file with all the unique values from one or more columns from
+# /tmp/alljobs.csv.
+
+# The only arg is a comma separated list of column names that will also be used
+# as the output filename.
+
+mlr --csv uniq -f $1 /tmp/alljobs.csv | mlr --csv sort -f $1 > /tmp/$1.csv

infra/bots/analysis/create-alljobs.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# Copyright 2021 Google Inc.
+#
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Create a file /tmp/alljobs.csv by parsing jobs.json.
+
+set -e
+
+# Write the starting set of headers.
+echo Type,os,compiler,model,cpu_or_gpu,cpu_or_gpu_value,arch,configuration,test_filter,extra > /tmp/alljobs.csv
+
+# Extract the CSV values from the jobs.json file.
+# The seds parse up the job names into columns, ensuring that every column has a value.
+cat ../jobs.json | jq .[] -r | grep "^[Perf|Test]" | sed "s#-#,#g" | sed "s#All\$#All,none#g" >> /tmp/alljobs.csv
+
+# Add the vulkan column.
+mlr --csv -I put '$vulkan=$extra =~ "Vulkan"' /tmp/alljobs.csv
+
+# Add the metal column.
+mlr --csv -I put '$metal=$extra =~ "Metal"' /tmp/alljobs.csv
+
+# Add the skpbench column.
+mlr --csv -I put '$skpbench=$extra =~ "Skpbench"' /tmp/alljobs.csv
+
+# Validate the output file is a valid CSV file.
+mlr --icsv check /tmp/alljobs.csv

infra/bots/analysis/missing.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+# Copyright 2021 Google Inc.
+#
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+
+# A script to find all cpu_or_gpu_values that don't get run as a job for the
+# given filter passed in as the first argument. The filter is a boolean query
+# against columns in /tmp/alljobs.csv, such as '$vulkan == "true"'.
+set -e
+
+if [ $# -ne 1 ]; then
+    echo "$0 <filter>"
+    echo "Example: $0 '\$vulkan == \"true\"'"
+    exit 1
+fi
+
+FILTER=$1
+
+# Ensure /tmp/alljobs.csv has been created.
+./create-alljobs.sh
+
+# Extract out the list of all cpu_or_gpu_values and associated model name.
+./axis.sh cpu_or_gpu_value,model
+
+# Find all cpus or gpus that we don't Test or Perf with Vulkan by creating a
+# list of all the cpu_or_gpu_values associated with Vulkan tests, and then print
+# all the rows in  /tmp/cpu_or_gpu_value,model.csv that don't match that list.
+#
+# The last join with --np means don't print matches, and --ul specifies to print
+# values that are unmatched on the left hand side of the join, i.e. from the
+# /tmp/cpu_or_gpu_value,model.csv file.
+mlr --csv filter "${FILTER}" /tmp/alljobs.csv  | \
+mlr --csv cut -f cpu_or_gpu_value,model | \
+mlr --csv sort -f cpu_or_gpu_value | \
+mlr --csv uniq -f cpu_or_gpu_value | \
+mlr --csv join -f /tmp/cpu_or_gpu_value,model.csv -j cpu_or_gpu_value --ul --np