SkQP: consolidate cut_release scripts
replace the following scripts: cut_release, get_gold_results.py, goldgetter.py, make_rendertests_list.py, and upload_model with a single program: cut_release. Still depends on three C++ programs: jitter_gms, list_gpu_unit_tests, and make_skqp_model. Change-Id: I28f59bc1f0caedc05d6ce2c4cc11bbd66cfb9784 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/209171 Reviewed-by: Ben Wagner <bungeman@google.com> Commit-Queue: Hal Canary <halcanary@google.com>
This commit is contained in:
Hal Canary
Skia Commit-Bot
parent
067861e148
commit
572d9bba64
@ -8,21 +8,19 @@ Skia][1]. Here is how that process works:
|
||||
|
||||
COMMIT=origin/master
|
||||
|
||||
1. Get the positively triaged results from Gold:
|
||||
Or use the script to find the best one:
|
||||
|
||||
cd SKIA_SOURCE_DIRECTORY
|
||||
git fetch origin
|
||||
COMMIT=$(python tools/skqp/find_commit_with_best_gold_results.py \
|
||||
origin/master ^origin/skqp/dev)
|
||||
|
||||
1. Get the positively triaged results from Gold and generate models:
|
||||
|
||||
cd SKIA_SOURCE_DIRECTORY
|
||||
git fetch origin
|
||||
git checkout "$COMMIT"
|
||||
python tools/skqp/get_gold_results.py "${COMMIT}~10" "$COMMIT"
|
||||
|
||||
This will produce a file `meta_YYYMMMDDD_HHHMMMSS_COMMIT_COMMIT.json` in
|
||||
the current directory.
|
||||
|
||||
2. From a checkout of Skia's master branch, execute:
|
||||
|
||||
cd SKIA_SOURCE_DIRECTORY
|
||||
git checkout "$COMMIT"
|
||||
tools/skqp/cut_release META_JSON_FILE
|
||||
python tools/skqp/cut_release.py HEAD~10 HEAD
|
||||
|
||||
This will create the following files:
|
||||
|
||||
@ -32,9 +30,9 @@ Skia][1]. Here is how that process works:
|
||||
|
||||
These three files can be commited to Skia to create a new commit. Make
|
||||
`origin/skqp/dev` a parent of this commit (without merging it in), and
|
||||
push this new commit to `origin/skqp/dev`:
|
||||
push this new commit to `origin/skqp/dev`, using this script:
|
||||
|
||||
tools/skqp/branch_skqp_dev.sh
|
||||
sh tools/skqp/branch_skqp_dev.sh
|
||||
|
||||
Review and submit the change:
|
||||
|
||||
@ -47,20 +45,18 @@ Skia][1]. Here is how that process works:
|
||||
|
||||
(Optional) Test the SkQP APK:
|
||||
|
||||
adb uninstall org.skia.skqp
|
||||
tools/skqp/test_apk.sh LOCATION/skqp-universal-debug.apk
|
||||
tools/skqp/test_apk.sh (LOCATION)/skqp-universal-debug.apk
|
||||
|
||||
(Once changes land) Upload the SkQP APK.
|
||||
|
||||
tools/skqp/upload_apk LOCATION/skqp-universal-debug.apk
|
||||
tools/skqp/upload_apk HEAD (LOCATION)/skqp-universal-debug.apk
|
||||
|
||||
|
||||
`tools/skqp/cut_release`
|
||||
------------------------
|
||||
`tools/skqp/cut_release.py`
|
||||
---------------------------
|
||||
|
||||
This tool will call `make_gmkb.go` to generate the `m{ax,in}.png` files for
|
||||
each render test. Additionaly, a `models.txt` file enumerates all of the
|
||||
models.
|
||||
This tool will call `make_skqp_model` to generate the `m{ax,in}.png` files for
|
||||
each render test.
|
||||
|
||||
Then it calls `jitter_gms` to see which render tests pass the jitter test.
|
||||
`jitter_gms` respects the `bad_gms.txt` file by ignoring the render tests
|
||||
@ -70,7 +66,7 @@ the file `good.txt`, those that fail in the `bad.txt` file.
|
||||
Next, the `skqp/rendertests.txt` file is created. This file lists the render
|
||||
tests that will be executed by SkQP. These are the union of the tests
|
||||
enumerated in the `good.txt` and `bad.txt` files. If the render test is found
|
||||
in the `models.txt` file and the `good.txt` file, its per-test threshold is set
|
||||
in the `good.txt` file and the model exists, its per-test threshold is set
|
||||
to 0 (a later CL can manually change this, if needed). Otherwise, the
|
||||
threshold is set to -1; this indicated that the rendertest will be executed (to
|
||||
verify that the driver will not crash), but the output will not be compared
|
||||
|
||||
@ -1,26 +0,0 @@
|
||||
#! /bin/sh
|
||||
# Copyright 2018 Google LLC.
|
||||
# Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "Usage: $0 META.JSON" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
set -x
|
||||
set -e
|
||||
META_JSON="$1"
|
||||
cd "$(dirname "$0")/../.."
|
||||
env GIT_SYNC_DEPS_QUIET=1 python tools/git-sync-deps
|
||||
O='out/ndebug'
|
||||
mkdir -p $O
|
||||
bin/gn gen $O --args='cc="clang" cxx="clang++" is_debug=false'
|
||||
ninja -C $O jitter_gms list_gpu_unit_tests make_skqp_model
|
||||
GMKB='platform_tools/android/apps/skqp/src/main/assets/gmkb'
|
||||
python tools/skqp/goldgetter.py "$META_JSON" "$GMKB" $O/make_skqp_model
|
||||
$O/jitter_gms tools/skqp/bad_gms.txt
|
||||
python tools/skqp/make_rendertests_list.py
|
||||
rm 'bad.txt' 'good.txt' "$GMKB"/models.txt
|
||||
sh tools/skqp/upload_model
|
||||
$O/list_gpu_unit_tests \
|
||||
> platform_tools/android/apps/skqp/src/main/assets/skqp/unittests.txt
|
||||
168
tools/skqp/cut_release.py
Executable file
168
tools/skqp/cut_release.py
Executable file
@ -0,0 +1,168 @@
|
||||
#! /usr/bin/env python
|
||||
# Copyright 2018 Google LLC.
|
||||
# Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.
|
||||
|
||||
import json
|
||||
import md5
|
||||
import multiprocessing
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib
|
||||
import urllib2
|
||||
|
||||
from subprocess import check_call, check_output
|
||||
|
||||
assert '/' in [os.sep, os.altsep] and os.pardir == '..'
|
||||
|
||||
ASSETS = 'platform_tools/android/apps/skqp/src/main/assets'
|
||||
BUCKET = 'skia-skqp-assets'
|
||||
|
||||
def make_skqp_model(arg):
|
||||
name, urls, exe = arg
|
||||
tmp = tempfile.mkdtemp()
|
||||
for url in urls:
|
||||
urllib.urlretrieve(url, tmp + '/' + url[url.rindex('/') + 1:])
|
||||
check_call([exe, tmp, ASSETS + '/gmkb/' + name])
|
||||
shutil.rmtree(tmp)
|
||||
sys.stdout.write(name + ' ')
|
||||
sys.stdout.flush()
|
||||
|
||||
def goldgetter(meta, exe):
|
||||
assert os.path.exists(exe)
|
||||
jobs = []
|
||||
for rec in meta:
|
||||
urls = [d['URL'] for d in rec['digests']
|
||||
if d['status'] == 'positive' and
|
||||
(set(d['paramset']['config']) & set(['vk', 'gles']))]
|
||||
if urls:
|
||||
jobs.append((rec['testName'], urls, exe))
|
||||
pool = multiprocessing.Pool(processes=20)
|
||||
pool.map(make_skqp_model, jobs)
|
||||
sys.stdout.write('\n')
|
||||
return set((n for n, _, _ in jobs))
|
||||
|
||||
def gold(first_commit, last_commit):
|
||||
c1, c2 = (check_output(['git', 'rev-parse', c]).strip()
|
||||
for c in (first_commit, last_commit))
|
||||
f = urllib2.urlopen('https://public-gold.skia.org/json/export?' + urllib.urlencode([
|
||||
('fbegin', c1),
|
||||
('fend', c2),
|
||||
('query', 'config=gles&config=vk&source_type=gm'),
|
||||
('pos', 'true'),
|
||||
('neg', 'false'),
|
||||
('unt', 'false')
|
||||
]))
|
||||
j = json.load(f)
|
||||
f.close()
|
||||
return j
|
||||
|
||||
def gset(path):
|
||||
s = set()
|
||||
if os.path.isfile(path):
|
||||
with open(path, 'r') as f:
|
||||
for line in f:
|
||||
s.add(line.strip())
|
||||
return s
|
||||
|
||||
def make_rendertest_list(models, good, bad):
|
||||
assert good.isdisjoint(bad)
|
||||
do_score = good & models
|
||||
no_score = bad | (good - models)
|
||||
to_delete = models & bad
|
||||
for d in to_delete:
|
||||
path = ASSETS + '/gmkb/' + d
|
||||
if os.path.isdir(path):
|
||||
shutil.rmtree(path)
|
||||
results = dict()
|
||||
for n in do_score:
|
||||
results[n] = 0
|
||||
for n in no_score:
|
||||
results[n] = -1
|
||||
return ''.join('%s,%d\n' % (n, results[n]) for n in sorted(results))
|
||||
|
||||
def get_digest(path):
|
||||
m = md5.new()
|
||||
with open(path, 'r') as f:
|
||||
m.update(f.read())
|
||||
return m.hexdigest()
|
||||
|
||||
def upload_cmd(path, digest):
|
||||
return ['gsutil', 'cp', path, 'gs://%s/%s' % (BUCKET, digest)]
|
||||
|
||||
def upload_model():
|
||||
bucket_url = 'gs://%s/' % BUCKET
|
||||
extant = set((u.replace(bucket_url, '', 1)
|
||||
for u in check_output(['gsutil', 'ls', bucket_url]).splitlines() if u))
|
||||
cmds = []
|
||||
filelist = []
|
||||
for dirpath, _, filenames in os.walk(ASSETS + '/gmkb'):
|
||||
for filename in filenames:
|
||||
path = os.path.join(dirpath, filename)
|
||||
digest = get_digest(path)
|
||||
if digest not in extant:
|
||||
cmds.append(upload_cmd(path, digest))
|
||||
filelist.append('%s;%s\n' % (digest, os.path.relpath(path, ASSETS)))
|
||||
tmp = tempfile.mkdtemp()
|
||||
filelist_path = tmp + '/x'
|
||||
with open(filelist_path, 'w') as o:
|
||||
for l in filelist:
|
||||
o.write(l)
|
||||
filelist_digest = get_digest(filelist_path)
|
||||
if filelist_digest not in extant:
|
||||
cmds.append(upload_cmd(filelist_path, filelist_digest))
|
||||
|
||||
pool = multiprocessing.Pool(processes=20)
|
||||
pool.map(check_call, cmds)
|
||||
shutil.rmtree(tmp)
|
||||
return filelist_digest
|
||||
|
||||
def remove(x):
|
||||
if os.path.isdir(x) and not os.path.islink(x):
|
||||
shutil.rmtree(x)
|
||||
if os.path.exists(x):
|
||||
os.remove(x)
|
||||
|
||||
def main(first_commit, last_commit):
|
||||
check_call(upload_cmd('/dev/null', get_digest('/dev/null')))
|
||||
|
||||
os.chdir(os.path.dirname(__file__) + '/../..')
|
||||
remove(ASSETS + '/files.checksum')
|
||||
for d in [ASSETS + '/gmkb', ASSETS + '/skqp', ]:
|
||||
remove(d)
|
||||
os.mkdir(d)
|
||||
|
||||
check_call([sys.executable, 'tools/git-sync-deps'],
|
||||
env=dict(os.environ, GIT_SYNC_DEPS_QUIET='T'))
|
||||
build = 'out/ndebug'
|
||||
check_call(['bin/gn', 'gen', build,
|
||||
'--args=cc="clang" cxx="clang++" is_debug=false'])
|
||||
check_call(['ninja', '-C', build,
|
||||
'jitter_gms', 'list_gpu_unit_tests', 'make_skqp_model'])
|
||||
|
||||
models = goldgetter(gold(first_commit, last_commit), build + '/make_skqp_model')
|
||||
|
||||
check_call([build + '/jitter_gms', 'tools/skqp/bad_gms.txt'])
|
||||
|
||||
with open(ASSETS + '/skqp/rendertests.txt', 'w') as o:
|
||||
o.write(make_rendertest_list(models, gset('good.txt'), gset('bad.txt')))
|
||||
|
||||
remove('good.txt')
|
||||
remove('bad.txt')
|
||||
|
||||
with open(ASSETS + '/skqp/unittests.txt', 'w') as o:
|
||||
o.write(check_output([build + '/list_gpu_unit_tests']))
|
||||
|
||||
with open(ASSETS + '/files.checksum', 'w') as o:
|
||||
o.write(upload_model() + '\n')
|
||||
|
||||
sys.stdout.write(ASSETS + '/files.checksum\n')
|
||||
sys.stdout.write(ASSETS + '/skqp/rendertests.txt\n')
|
||||
sys.stdout.write(ASSETS + '/skqp/unittests.txt\n')
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) != 3:
|
||||
sys.stderr.write('Usage:\n %s C1 C2\n\n' % sys.argv[0])
|
||||
sys.exit(1)
|
||||
main(sys.argv[1], sys.argv[2])
|
||||
@ -1,42 +0,0 @@
|
||||
#! /usr/bin/env python
|
||||
|
||||
# Copyright 2018 Google LLC.
|
||||
# Use of this source code is governed by a BSD-style license that can be
|
||||
# found in the LICENSE file.
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib
|
||||
|
||||
def gold_export_url(first_commit, last_commit):
|
||||
query = [
|
||||
('fbegin', first_commit),
|
||||
('fend', last_commit),
|
||||
('query', 'config=gles&config=vk&source_type=gm'),
|
||||
('pos', 'true'),
|
||||
('neg', 'false'),
|
||||
('unt', 'false')
|
||||
]
|
||||
return 'https://public-gold.skia.org/json/export?' + urllib.urlencode(query)
|
||||
|
||||
def git_rev_parse(rev):
|
||||
return subprocess.check_output(['git', 'rev-parse', rev]).strip()
|
||||
|
||||
def main(args):
|
||||
if len(args) != 2:
|
||||
sys.stderr.write('Usage:\n %s FIRST_COMMIT LAST_COMMIT\n' % __file__)
|
||||
sys.exit(1)
|
||||
c1 = git_rev_parse(args[0])
|
||||
c2 = git_rev_parse(args[1])
|
||||
now = time.strftime("%Y%m%d_%H%M%S", time.gmtime())
|
||||
url = gold_export_url(c1, c2)
|
||||
sys.stdout.write(url + '\n')
|
||||
filename = 'meta_%s_%s_%s.json' % (now, c1[:16], c2[:16])
|
||||
urllib.urlretrieve(url, filename)
|
||||
sys.stdout.write('\n' + filename + '\n')
|
||||
|
||||
if __name__ == '__main__':
|
||||
main(sys.argv[1:])
|
||||
|
||||
@ -1,48 +0,0 @@
|
||||
#! /usr/bin/env python
|
||||
# Copyright 2019 Google LLC.
|
||||
# Use of this source code is governed by a BSD-style license that can be
|
||||
# found in the LICENSE file.
|
||||
|
||||
import json
|
||||
import multiprocessing
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib
|
||||
|
||||
def make_skqp_model(arg):
|
||||
name, urls, dst_dir, exe = arg
|
||||
tmp = tempfile.mkdtemp()
|
||||
for url in urls:
|
||||
urllib.urlretrieve(url, tmp + '/' + url[url.rindex('/') + 1:])
|
||||
subprocess.check_call([exe, tmp, dst_dir + '/' + name])
|
||||
shutil.rmtree(tmp)
|
||||
sys.stdout.write(name + ' ')
|
||||
sys.stdout.flush()
|
||||
|
||||
def main(meta, dst, exe):
|
||||
assert os.path.exists(exe)
|
||||
jobs = []
|
||||
with open(meta, 'r') as f:
|
||||
for rec in json.load(f):
|
||||
urls = [d['URL'] for d in rec['digests']
|
||||
if d['status'] == 'positive' and
|
||||
(set(d['paramset']['config']) & set(['vk', 'gles']))]
|
||||
if urls:
|
||||
jobs.append((rec['testName'], urls, dst, exe))
|
||||
if not os.path.exists(dst):
|
||||
os.mkdir(dst)
|
||||
pool = multiprocessing.Pool(processes=20)
|
||||
pool.map(make_skqp_model, jobs)
|
||||
sys.stdout.write('\n')
|
||||
with open(dst + '/models.txt', 'w') as o:
|
||||
for n, _, _, _ in jobs:
|
||||
o.write(n + '\n')
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) != 4:
|
||||
sys.stderr.write('Usage:\n %s META.JSON DST_DIR MAKE_SKQP_MODEL_EXE\n\n' % sys.argv[0])
|
||||
sys.exit(1)
|
||||
main(sys.argv[1], sys.argv[2], sys.argv[3])
|
||||
@ -1,49 +0,0 @@
|
||||
#! /usr/bin/env python
|
||||
|
||||
# Copyright 2018 Google LLC.
|
||||
# Use of this source code is governed by a BSD-style license that can be
|
||||
# found in the LICENSE file.
|
||||
|
||||
import csv
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
def gset(path):
|
||||
s = set()
|
||||
if os.path.isfile(path):
|
||||
with open(path, 'r') as f:
|
||||
for line in f:
|
||||
s.add(line.strip())
|
||||
return s
|
||||
|
||||
def main():
|
||||
assert '/' in [os.sep, os.altsep]
|
||||
assets = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir,
|
||||
'platform_tools/android/apps/skqp/src/main/assets')
|
||||
models = gset(assets + '/gmkb/models.txt')
|
||||
good = gset('good.txt')
|
||||
bad = gset('bad.txt')
|
||||
assert good.isdisjoint(bad)
|
||||
do_score = good & models
|
||||
no_score = bad | (good - models)
|
||||
to_delete = models & bad
|
||||
for d in to_delete:
|
||||
path = assets + '/gmkb/' + d
|
||||
if os.path.isdir(path):
|
||||
shutil.rmtree(path)
|
||||
results = dict()
|
||||
for n in do_score:
|
||||
results[n] = 0
|
||||
for n in no_score:
|
||||
results[n] = -1
|
||||
skqp = assets + '/skqp'
|
||||
if not os.path.isdir(skqp):
|
||||
os.mkdir(skqp)
|
||||
with open(skqp + '/rendertests.txt', 'w') as o:
|
||||
for n in sorted(results):
|
||||
o.write('%s,%d\n' % (n, results[n]))
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
@ -1,62 +0,0 @@
|
||||
#! /bin/sh
|
||||
|
||||
# Copyright 2018 Google Inc.
|
||||
# Use of this source code is governed by a BSD-style license that can be
|
||||
# found in the LICENSE file.
|
||||
|
||||
set -e
|
||||
|
||||
BASE_DIR='platform_tools/android/apps/skqp/src/main/assets'
|
||||
PATH_LIST='gmkb'
|
||||
BUCKET=skia-skqp-assets
|
||||
|
||||
EXTANT="$(mktemp "${TMPDIR:-/tmp}/extant.XXXXXXXXXX")"
|
||||
gsutil ls gs://$BUCKET/ | sed "s|^gs://$BUCKET/||" > "$EXTANT"
|
||||
|
||||
upload() {
|
||||
MD5=$(md5sum < "$1" | head -c 32)
|
||||
if ! grep -q "$MD5" "$EXTANT"; then
|
||||
URL="gs://${BUCKET}/$MD5"
|
||||
gsutil cp "$1" "$URL" > /dev/null 2>&1 &
|
||||
fi
|
||||
echo $MD5
|
||||
}
|
||||
|
||||
size() { gsutil du -s gs://$BUCKET | awk '{print $1}'; }
|
||||
|
||||
cd "$(dirname "$0")/../../$BASE_DIR"
|
||||
|
||||
rm -f files.checksum
|
||||
|
||||
FILES="$(mktemp "${TMPDIR:-/tmp}/files.XXXXXXXXXX")"
|
||||
|
||||
: > "$FILES"
|
||||
|
||||
COUNT=$(find $PATH_LIST -type f | wc -l)
|
||||
INDEX=1
|
||||
SHARD_COUNT=32
|
||||
|
||||
SIZE=$(size)
|
||||
find $PATH_LIST -type f | sort | while IFS= read -r FILENAME; do
|
||||
printf '\r %d / %d ' "$INDEX" "$COUNT"
|
||||
if ! [ -f "$FILENAME" ]; then
|
||||
echo error [${FILENAME}] >&2;
|
||||
exit 1;
|
||||
fi
|
||||
case "$FILENAME" in *\;*) echo bad filename: $FILENAME >&2; exit 1;; esac
|
||||
MD5=$(upload "$FILENAME")
|
||||
printf '%s;%s\n' "$MD5" "$FILENAME" >> "$FILES"
|
||||
|
||||
if [ $(($INDEX % $SHARD_COUNT)) = 0 ]; then
|
||||
wait
|
||||
fi
|
||||
INDEX=$(( $INDEX + 1))
|
||||
done
|
||||
printf '\rdone \n'
|
||||
upload "$FILES" > files.checksum
|
||||
wait
|
||||
|
||||
D=$(( $(size) - $SIZE ))
|
||||
printf 'Added %d bytes to %s, %d%%\n' $D $BUCKET $(( $D * 100 / $SIZE ))
|
||||
|
||||
rm "$EXTANT"
|
||||
Loading…
Reference in New Issue
Block a user