[infra] Make gold_upload module for direct-from-test uploads.

Bug: skia:11785
Change-Id: I6358e83242483cde3c8a726111fe9ed07fb09bda
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/389708
Reviewed-by: Eric Boren <borenet@google.com>
This commit is contained in:
Weston Tracey 2021-03-29 10:29:40 -04:00
parent 7845b9700e
commit 9bb0b6e2f6
7 changed files with 198 additions and 110 deletions

View File

@ -11,6 +11,7 @@
* [env](#recipe_modules-env)
* [flavor](#recipe_modules-flavor)
* [git](#recipe_modules-git)
* [gold_upload](#recipe_modules-gold_upload)
* [gsutil](#recipe_modules-gsutil)
* [infra](#recipe_modules-infra)
* [run](#recipe_modules-run)
@ -29,6 +30,7 @@
* [env:examples/full](#recipes-env_examples_full)
* [flavor:examples/full](#recipes-flavor_examples_full)
* [git:examples/full](#recipes-git_examples_full)
* [gold_upload:examples/full](#recipes-gold_upload_examples_full)
* [gsutil:examples/full](#recipes-gsutil_examples_full)
* [housekeeper](#recipes-housekeeper)
* [infra](#recipes-infra)
@ -174,6 +176,16 @@ Add Git to PATH
Requires the infra/git and infra/tools/git CIPD packages to be installed
in the 'git' relative path.
### *recipe_modules* / [gold\_upload](/infra/bots/recipe_modules/gold_upload)
[DEPS](/infra/bots/recipe_modules/gold_upload/__init__.py#5): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [flavor](#recipe_modules-flavor), [gsutil](#recipe_modules-gsutil), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
#### **class [GoldUploadApi](/infra/bots/recipe_modules/gold_upload/api.py#11)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&mdash; **def [upload](/infra/bots/recipe_modules/gold_upload/api.py#12)(self):**
Attempt to upload files to Gold.
This module assumes setup has occurred for the vars and flavor modules.
### *recipe_modules* / [gsutil](/infra/bots/recipe_modules/gsutil)
[DEPS](/infra/bots/recipe_modules/gsutil/__init__.py#5): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step], [run](#recipe_modules-run), [vars](#recipe_modules-vars)
@ -347,6 +359,11 @@ Prepare the variables.
[DEPS](/infra/bots/recipe_modules/git/examples/full.py#6): [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/step][recipe_engine/recipe_modules/step], [git](#recipe_modules-git)
&mdash; **def [RunSteps](/infra/bots/recipe_modules/git/examples/full.py#13)(api):**
### *recipes* / [gold\_upload:examples/full](/infra/bots/recipe_modules/gold_upload/examples/full.py)
[DEPS](/infra/bots/recipe_modules/gold_upload/examples/full.py#9): [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step], [flavor](#recipe_modules-flavor), [gold\_upload](#recipe_modules-gold_upload), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
&mdash; **def [RunSteps](/infra/bots/recipe_modules/gold_upload/examples/full.py#21)(api):**
### *recipes* / [gsutil:examples/full](/infra/bots/recipe_modules/gsutil/examples/full.py)
[DEPS](/infra/bots/recipe_modules/gsutil/examples/full.py#9): [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step], [gsutil](#recipe_modules-gsutil), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
@ -458,22 +475,18 @@ benchmark Skia using skpbench.
&mdash; **def [RunSteps](/infra/bots/recipes/sync_and_compile.py#25)(api):**
### *recipes* / [test](/infra/bots/recipes/test.py)
[DEPS](/infra/bots/recipes/test.py#13): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [env](#recipe_modules-env), [flavor](#recipe_modules-flavor), [gsutil](#recipe_modules-gsutil), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
[DEPS](/infra/bots/recipes/test.py#12): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [env](#recipe_modules-env), [flavor](#recipe_modules-flavor), [gold\_upload](#recipe_modules-gold_upload), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
&mdash; **def [RunSteps](/infra/bots/recipes/test.py#209)(api):**
&mdash; **def [RunSteps](/infra/bots/recipes/test.py#161)(api):**
&mdash; **def [test\_steps](/infra/bots/recipes/test.py#78)(api):**
&mdash; **def [test\_steps](/infra/bots/recipes/test.py#30)(api):**
Run the DM test.
&mdash; **def [upload](/infra/bots/recipes/test.py#32)(api):**
### *recipes* / [test\_canvaskit](/infra/bots/recipes/test_canvaskit.py)
[DEPS](/infra/bots/recipes/test_canvaskit.py#9): [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [checkout](#recipe_modules-checkout), [docker](#recipe_modules-docker), [env](#recipe_modules-env), [gsutil](#recipe_modules-gsutil), [infra](#recipe_modules-infra), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
[DEPS](/infra/bots/recipes/test_canvaskit.py#8): [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step], [checkout](#recipe_modules-checkout), [docker](#recipe_modules-docker), [env](#recipe_modules-env), [flavor](#recipe_modules-flavor), [gold\_upload](#recipe_modules-gold_upload), [infra](#recipe_modules-infra), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
&mdash; **def [RunSteps](/infra/bots/recipes/test_canvaskit.py#76)(api):**
&mdash; **def [upload](/infra/bots/recipes/test_canvaskit.py#31)(api):**
&mdash; **def [RunSteps](/infra/bots/recipes/test_canvaskit.py#28)(api):**
### *recipes* / [test\_lottie\_web](/infra/bots/recipes/test_lottie_web.py)
[DEPS](/infra/bots/recipes/test_lottie_web.py#7): [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step], [checkout](#recipe_modules-checkout), [docker](#recipe_modules-docker), [env](#recipe_modules-env), [infra](#recipe_modules-infra), [run](#recipe_modules-run), [vars](#recipe_modules-vars)

View File

@ -0,0 +1,17 @@
# Copyright 2021 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
DEPS = [
'recipe_engine/json',
'recipe_engine/context',
'recipe_engine/file',
'recipe_engine/python',
'recipe_engine/properties',
'recipe_engine/step',
'recipe_engine/time',
'flavor',
'gsutil',
'run',
'vars',
]

View File

@ -0,0 +1,59 @@
# Copyright 2021 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from recipe_engine import recipe_api
import calendar
DM_JSON = 'dm.json'
class GoldUploadApi(recipe_api.RecipeApi):
def upload(self):
"""Attempt to upload files to Gold.
This module assumes setup has occurred for the vars and flavor modules.
"""
revision = self.m.properties['revision']
results_dir = self.m.flavor.host_dirs.dm_dir
# Upload the images. It is preferred that the images are uploaded first
# so they exist whenever the json is processed.
image_dest_path = 'gs://%s/dm-images-v1' % self.m.properties['gs_bucket']
for ext in ['.png']:
files_to_upload = self.m.file.glob_paths(
'find %s images' % ext,
results_dir,
'*%s' % ext,
test_data=['someimage.png'])
# For some reason, glob returns results_dir when it should return nothing.
files_to_upload = [f for f in files_to_upload if str(f).endswith(ext)]
if len(files_to_upload) > 0:
self.m.gsutil.cp('%s images' % ext, results_dir.join('*%s' % ext),
image_dest_path, multithread=True)
summary_dest_path = 'gs://%s' % self.m.properties['gs_bucket']
ref = revision
# Trybot results are siloed by issue/patchset.
if self.m.vars.is_trybot:
summary_dest_path = '/'.join([summary_dest_path, 'trybot'])
ref = '%s_%s' % (str(self.m.vars.issue), str(self.m.vars.patchset))
# Compute the directory to upload results to
now = self.m.time.utcnow()
summary_dest_path = '/'.join([
summary_dest_path,
'dm-json-v1',
str(now.year ).zfill(4),
str(now.month).zfill(2),
str(now.day ).zfill(2),
str(now.hour ).zfill(2),
ref,
self.m.vars.builder_name,
str(int(calendar.timegm(now.utctimetuple())))])
# Directly upload dm.json if it exists.
json_file = results_dir.join(DM_JSON)
# -Z compresses the json file at rest with gzip.
self.m.gsutil.cp('dm.json', json_file,
summary_dest_path + '/' + DM_JSON, extra_args=['-Z'])

View File

@ -0,0 +1,56 @@
[
{
"cmd": [
"python",
"-u",
"import os\nprint os.environ.get('SWARMING_BOT_ID', '')\n"
],
"name": "get swarming bot id",
"~followup_annotations": [
"@@@STEP_LOG_LINE@python.inline@import os@@@",
"@@@STEP_LOG_LINE@python.inline@print os.environ.get('SWARMING_BOT_ID', '')@@@",
"@@@STEP_LOG_END@python.inline@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"glob",
"[START_DIR]/[SWARM_OUT_DIR]",
"*.png"
],
"infra_step": true,
"name": "find .png images",
"~followup_annotations": [
"@@@STEP_LOG_LINE@glob@[START_DIR]/[SWARM_OUT_DIR]/someimage.png@@@",
"@@@STEP_LOG_END@glob@@@"
]
},
{
"cmd": [
"gsutil",
"-m",
"cp",
"[START_DIR]/[SWARM_OUT_DIR]/*.png",
"gs://skia-infra-gm/dm-images-v1"
],
"name": "upload .png images"
},
{
"cmd": [
"gsutil",
"cp",
"-Z",
"[START_DIR]/[SWARM_OUT_DIR]/dm.json",
"gs://skia-infra-gm/trybot/dm-json-v1/2012/05/14/12/1234_7/Test-Android-Clang-Pixel2XL-Some-GPU-arm64-Debug-All/1337000001/dm.json"
],
"name": "upload dm.json"
},
{
"name": "$result"
}
]

View File

@ -0,0 +1,38 @@
# Copyright 2021 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# Recipe which runs the Skia gold_upload tests.
DEPS = [
'gold_upload',
'recipe_engine/path',
'recipe_engine/properties',
'recipe_engine/python',
'recipe_engine/step',
'flavor',
'run',
'vars',
]
def RunSteps(api):
api.vars.setup()
api.flavor.setup('dm')
api.gold_upload.upload()
def GenTests(api):
yield (
api.test('upload_tests') +
api.properties(buildername='Test-Android-Clang-Pixel2XL-Some-GPU-arm64-Debug-All',
repository='https://skia.googlesource.com/skia.git',
gs_bucket='skia-infra-gm',
patch_ref='89/456789/12',
patch_set=7,
patch_issue=1234,
revision='abc123',
path_config='kitchen',
swarm_out_dir='[SWARM_OUT_DIR]')
)

View File

@ -6,14 +6,12 @@
# Recipe module for Skia Swarming test.
import calendar
import json
DEPS = [
'env',
'flavor',
'gsutil',
'recipe_engine/context',
'recipe_engine/file',
'recipe_engine/path',
@ -22,59 +20,13 @@ DEPS = [
'recipe_engine/python',
'recipe_engine/raw_io',
'recipe_engine/step',
'recipe_engine/time',
'gold_upload',
'run',
'vars',
]
DM_JSON = 'dm.json'
def upload(api):
revision = api.properties['revision']
results_dir = api.flavor.host_dirs.dm_dir
# Upload the images. It is preferred that the images are uploaded first
# so they exist whenever the json is processed.
image_dest_path = 'gs://%s/dm-images-v1' % api.properties['gs_bucket']
for ext in ['.png']:
files_to_upload = api.file.glob_paths(
'find %s images' % ext,
results_dir,
'*%s' % ext,
test_data=['someimage.png'])
# For some reason, glob returns results_dir when it should return nothing.
files_to_upload = [f for f in files_to_upload if str(f).endswith(ext)]
if len(files_to_upload) > 0:
api.gsutil.cp('%s images' % ext, results_dir.join('*%s' % ext),
image_dest_path, multithread=True)
summary_dest_path = 'gs://%s' % api.properties['gs_bucket']
ref = revision
# Trybot results are siloed by issue/patchset.
if api.vars.is_trybot:
summary_dest_path = '/'.join([summary_dest_path, 'trybot'])
ref = '%s_%s' % (str(api.vars.issue), str(api.vars.patchset))
# Compute the directory to upload results to
now = api.time.utcnow()
summary_dest_path = '/'.join([
summary_dest_path,
'dm-json-v1',
str(now.year ).zfill(4),
str(now.month).zfill(2),
str(now.day ).zfill(2),
str(now.hour ).zfill(2),
ref,
api.vars.builder_name,
str(int(calendar.timegm(now.utctimetuple())))])
# Directly upload dm.json if it exists.
json_file = results_dir.join(DM_JSON)
# -Z compresses the json file at rest with gzip.
api.gsutil.cp('dm.json', json_file,
summary_dest_path + '/' + DM_JSON, extra_args=['-Z'])
def test_steps(api):
"""Run the DM test."""
do_upload = api.properties.get('do_upload') == 'true'
@ -203,7 +155,7 @@ def test_steps(api):
api.flavor.device_dirs.dm_dir, api.flavor.host_dirs.dm_dir)
# https://bugs.chromium.org/p/chromium/issues/detail?id=1192611
if 'Win' not in api.vars.builder_cfg.get('os', ''):
upload(api)
api.gold_upload.upload()
def RunSteps(api):

View File

@ -4,21 +4,19 @@
# Recipe which runs the Canvaskit tests using docker
import calendar
DEPS = [
'checkout',
'docker',
'env',
'flavor',
'infra',
'recipe_engine/file',
'recipe_engine/json',
'recipe_engine/path',
'recipe_engine/properties',
'recipe_engine/python',
'recipe_engine/step',
'recipe_engine/time',
'gsutil',
'gold_upload',
'run',
'vars',
]
@ -26,55 +24,10 @@ DEPS = [
DOCKER_IMAGE = 'gcr.io/skia-public/gold-karma-chrome-tests:87.0.4280.88_v1'
INNER_KARMA_SCRIPT = 'skia/infra/canvaskit/test_canvaskit.sh'
DM_JSON = 'dm.json'
def upload(api):
revision = api.properties['revision']
results_dir = api.vars.swarming_out_dir
# Upload the images. It is preferred that the images are uploaded first
# so they exist whenever the json is processed.
image_dest_path = 'gs://%s/dm-images-v1' % api.properties['gs_bucket']
for ext in ['.png']:
files_to_upload = api.file.glob_paths(
'find %s images' % ext,
results_dir,
'*%s' % ext,
test_data=['someimage.png'])
# For some reason, glob returns results_dir when it should return nothing.
files_to_upload = [f for f in files_to_upload if str(f).endswith(ext)]
if len(files_to_upload) > 0:
api.gsutil.cp('%s images' % ext, results_dir.join('*%s' % ext),
image_dest_path, multithread=True)
summary_dest_path = 'gs://%s' % api.properties['gs_bucket']
ref = revision
# Trybot results are siloed by issue/patchset.
if api.vars.is_trybot:
summary_dest_path = '/'.join([summary_dest_path, 'trybot'])
ref = '%s_%s' % (str(api.vars.issue), str(api.vars.patchset))
# Compute the directory to upload results to
now = api.time.utcnow()
summary_dest_path = '/'.join([
summary_dest_path,
'dm-json-v1',
str(now.year ).zfill(4),
str(now.month).zfill(2),
str(now.day ).zfill(2),
str(now.hour ).zfill(2),
ref,
api.vars.builder_name,
str(int(calendar.timegm(now.utctimetuple())))])
# Directly upload dm.json if it exists.
json_file = results_dir.join(DM_JSON)
# -Z compresses the json file at rest with gzip.
api.gsutil.cp('dm.json', json_file,
summary_dest_path + '/' + DM_JSON, extra_args=['-Z'])
def RunSteps(api):
api.vars.setup()
api.flavor.setup('dm')
checkout_root = api.path['start_dir']
out_dir = api.vars.swarming_out_dir
@ -119,7 +72,7 @@ def RunSteps(api):
attempts=3,
)
upload(api)
api.gold_upload.upload()
def GenTests(api):
yield (