AuroraMiddleware/skia2
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

[infra] Remove Python2 when running CI tasks

Browse Source 
This is similar to https://skia-review.googlesource.com/c/buildbot/+/538218

BIG CHANGE: If we fail to download uninteresting hashes
(because Gold is down or the script otherwise fails), we
now crash/fail our Test-* tasks. In an early version of
this removal, that lack of failure masked an issue with
the script (urllib2 is not in Python3) and nearly would
have landed. Failing loudly is better, IMO.

This removes the symbolize_stack_trace script, which
stopped working with Python3 for reasons unknown.
If we need the behavior, we can rewrite it.
Relatedly, we removed 4 jobs, the Docker ones, because
we will not get much value out of them as we migrate
towards Bazel and removing the symbolization script
was tricky to get right.

There are a few cleanups around copypasta that I noticed
when combing through the recipes.

Change-Id: I8dfab416e964fd494267800b4ebe216061895f19
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/538636
Reviewed-by: Ravi Mistry <rmistry@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
Reviewed-by: Eric Boren <borenet@google.com>
This commit is contained in:
Kevin Lubick 2022-05-17 15:47:17 -04:00 committed by SkCQ
parent 11c0ca2833
commit 52b040968a
30 changed files with 97 additions and 7115 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
infra/bots/README.recipes.md
View File

@ -138,39 +138,39 @@ PYTHON_VERSION_COMPATIBILITY: PY3
#### **class [EnvApi](/infra/bots/recipe_modules/env/api.py#9)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
### *recipe_modules* / [flavor](/infra/bots/recipe_modules/flavor)
[DEPS](/infra/bots/recipe_modules/flavor/__init__.py#7): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [docker](#recipe_modules-docker), [env](#recipe_modules-env), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
[DEPS](/infra/bots/recipe_modules/flavor/__init__.py#7): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [env](#recipe_modules-env), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
PYTHON_VERSION_COMPATIBILITY: PY3
#### **class [SkiaFlavorApi](/infra/bots/recipe_modules/flavor/api.py#59)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
#### **class [SkiaFlavorApi](/infra/bots/recipe_modules/flavor/api.py#55)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
&mdash; **def [cleanup\_steps](/infra/bots/recipe_modules/flavor/api.py#131)(self):**
&mdash; **def [cleanup\_steps](/infra/bots/recipe_modules/flavor/api.py#125)(self):**
&mdash; **def [copy\_directory\_contents\_to\_device](/infra/bots/recipe_modules/flavor/api.py#87)(self, host_dir, device_dir):**
&mdash; **def [copy\_directory\_contents\_to\_device](/infra/bots/recipe_modules/flavor/api.py#81)(self, host_dir, device_dir):**
&mdash; **def [copy\_directory\_contents\_to\_host](/infra/bots/recipe_modules/flavor/api.py#90)(self, device_dir, host_dir):**
&mdash; **def [copy\_directory\_contents\_to\_host](/infra/bots/recipe_modules/flavor/api.py#84)(self, device_dir, host_dir):**
&mdash; **def [copy\_file\_to\_device](/infra/bots/recipe_modules/flavor/api.py#93)(self, host_path, device_path):**
&mdash; **def [copy\_file\_to\_device](/infra/bots/recipe_modules/flavor/api.py#87)(self, host_path, device_path):**
&mdash; **def [create\_clean\_device\_dir](/infra/bots/recipe_modules/flavor/api.py#99)(self, path):**
&mdash; **def [create\_clean\_device\_dir](/infra/bots/recipe_modules/flavor/api.py#93)(self, path):**
&mdash; **def [create\_clean\_host\_dir](/infra/bots/recipe_modules/flavor/api.py#96)(self, path):**
&mdash; **def [create\_clean\_host\_dir](/infra/bots/recipe_modules/flavor/api.py#90)(self, path):**
&mdash; **def [device\_path\_join](/infra/bots/recipe_modules/flavor/api.py#84)(self, \*args):**
&mdash; **def [device\_path\_join](/infra/bots/recipe_modules/flavor/api.py#78)(self, \*args):**
&mdash; **def [get\_flavor](/infra/bots/recipe_modules/flavor/api.py#60)(self, vars_api, app_name):**
&mdash; **def [get\_flavor](/infra/bots/recipe_modules/flavor/api.py#56)(self, vars_api, app_name):**
Return a flavor utils object specific to the given builder.
&mdash; **def [install](/infra/bots/recipe_modules/flavor/api.py#108)(self, skps=False, images=False, lotties=False, svgs=False, resources=False, mskps=False, texttraces=False):**
&mdash; **def [install](/infra/bots/recipe_modules/flavor/api.py#102)(self, skps=False, images=False, lotties=False, svgs=False, resources=False, mskps=False, texttraces=False):**
&mdash; **def [read\_file\_on\_device](/infra/bots/recipe_modules/flavor/api.py#102)(self, path, \*\*kwargs):**
&mdash; **def [read\_file\_on\_device](/infra/bots/recipe_modules/flavor/api.py#96)(self, path, \*\*kwargs):**
&mdash; **def [remove\_file\_on\_device](/infra/bots/recipe_modules/flavor/api.py#105)(self, path):**
&mdash; **def [remove\_file\_on\_device](/infra/bots/recipe_modules/flavor/api.py#99)(self, path):**
&mdash; **def [setup](/infra/bots/recipe_modules/flavor/api.py#75)(self, app_name):**
&mdash; **def [setup](/infra/bots/recipe_modules/flavor/api.py#69)(self, app_name):**
&mdash; **def [step](/infra/bots/recipe_modules/flavor/api.py#81)(self, name, cmd, \*\*kwargs):**
&mdash; **def [step](/infra/bots/recipe_modules/flavor/api.py#75)(self, name, cmd, \*\*kwargs):**
### *recipe_modules* / [git](/infra/bots/recipe_modules/git)
[DEPS](/infra/bots/recipe_modules/git/__init__.py#7): [recipe\_engine/path][recipe_engine/recipe_modules/path], [env](#recipe_modules-env)
@ -536,7 +536,7 @@ PYTHON_VERSION_COMPATIBILITY: PY3
PYTHON_VERSION_COMPATIBILITY: PY3
&mdash; **def [RunSteps](/infra/bots/recipes/test.py#162)(api):**
&mdash; **def [RunSteps](/infra/bots/recipes/test.py#165)(api):**
&mdash; **def [test\_steps](/infra/bots/recipes/test.py#31)(api):**

17
infra/bots/gen_tasks_logic/task_builder.go
View File

@ -318,11 +318,9 @@ func (b *taskBuilder) cipdPlatform() string {
// usesPython adds attributes to tasks which use python.
func (b *taskBuilder) usesPython() {
pythonPkgs := cipd.PkgsPython[b.cipdPlatform()]
pythonPkgs := removePython2(cipd.PkgsPython[b.cipdPlatform()])
b.cipd(pythonPkgs...)
b.addToPATH(
"cipd_bin_packages/cpython",
"cipd_bin_packages/cpython/bin",
"cipd_bin_packages/cpython3",
"cipd_bin_packages/cpython3/bin",
)
@ -334,6 +332,19 @@ func (b *taskBuilder) usesPython() {
b.env("VPYTHON_LOG_TRACE", "1")
}
// removePython2 removes all python2 packages from a list of CIPD packages. This can be used to
// enforce the lack of Python2 dependencies in our tests.
func removePython2(pyPackages []*cipd.Package) []*cipd.Package {
var python3Pkgs []*cipd.Package
for _, p := range pyPackages {
if strings.HasPrefix(p.Version, "version:2@2.7") {
continue
}
python3Pkgs = append(python3Pkgs, p)
}
return python3Pkgs
}
func (b *taskBuilder) usesNode() {
// It is very important when including node via CIPD to also add it to the PATH of the
// taskdriver or mysterious things can happen when subprocesses try to resolve node/npm.

4
infra/bots/jobs.json
View File

@ -475,10 +475,6 @@
{"name": "Test-ChromeOS-Clang-Spin513-GPU-Adreno618-arm-Release-All"},
{"name": "Test-ChromeOS-Clang-Spin514-GPU-RadeonVega3-x86_64-Debug-All"},
{"name": "Test-ChromeOS-Clang-Spin514-GPU-RadeonVega3-x86_64-Release-All"},
{"name": "Test-Debian10-GCC-GCE-CPU-AVX2-x86-Debug-All-Docker"},
{"name": "Test-Debian10-GCC-GCE-CPU-AVX2-x86-Release-All-Docker"},
{"name": "Test-Debian10-GCC-GCE-CPU-AVX2-x86_64-Debug-All-Docker"},
{"name": "Test-Debian10-GCC-GCE-CPU-AVX2-x86_64-Release-All-Docker"},
{"name": "Test-Debian10-Clang-GCE-CPU-AVX2-x86-Debug-All"},
{"name": "Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All",
"cq_config": {}

10
infra/bots/recipe_modules/checkout/examples/full.py
View File

@ -71,10 +71,7 @@ def GenTests(api):
patch_set=12,
patch_ref='refs/changes/89/456789/12',
patch_repo='https://skia.googlesource.com/skia.git',
patch_storage='gerrit') +
api.path.exists(
api.path['start_dir'].join('tmp', 'uninteresting_hashes.txt')
)
patch_storage='gerrit')
)
builder = 'Build-Debian10-Clang-x86_64-Release-NoDEPS'
@ -106,10 +103,7 @@ def GenTests(api):
patch_set=12,
patch_ref='refs/changes/89/456789/12',
patch_repo='https://skia.googlesource.com/skia.git',
patch_storage='gerrit') +
api.path.exists(
api.path['start_dir'].join('tmp', 'uninteresting_hashes.txt')
)
patch_storage='gerrit')
)
yield (
api.test('trybot') +

1
infra/bots/recipe_modules/flavor/__init__.py
View File

@ -5,7 +5,6 @@
PYTHON_VERSION_COMPATIBILITY = "PY3"
DEPS = [
'docker',
'env',
'recipe_engine/context',
'recipe_engine/file',

6
infra/bots/recipe_modules/flavor/api.py
View File

@ -11,7 +11,6 @@ from recipe_engine import recipe_api
from . import android
from . import chromebook
from . import default
from . import docker
from . import ios
from . import valgrind
@ -45,9 +44,6 @@ def is_chromebook(vars_api):
return ('Chromebook' in vars_api.extra_tokens or
'ChromeOS' in vars_api.builder_cfg.get('os', ''))
def is_docker(vars_api):
return 'Docker' in vars_api.extra_tokens
def is_ios(vars_api):
return ('iOS' in vars_api.extra_tokens or
'iOS' == vars_api.builder_cfg.get('os', ''))
@ -63,8 +59,6 @@ class SkiaFlavorApi(recipe_api.RecipeApi):
return chromebook.ChromebookFlavor(self, app_name)
if is_android(vars_api):
return android.AndroidFlavor(self, app_name)
elif is_docker(vars_api):
return docker.DockerFlavor(self, app_name)
elif is_ios(vars_api):
return ios.iOSFlavor(self, app_name)
elif is_valgrind(vars_api):

16
infra/bots/recipe_modules/flavor/default.py
View File

@ -116,10 +116,6 @@ class DefaultFlavor(object):
return self.m.run(self.m.step, title, cmd=cmd,
infra_step=infra_step, **kwargs)
def _py(self, title, script, infra_step=True, args=()):
return self.m.run(self.m.python, title, script=script, args=args,
infra_step=infra_step)
def step(self, name, cmd, **unused_kwargs):
app = self.device_dirs.bin_dir.join(cmd[0])
cmd = [app] + cmd[1:]
@ -209,17 +205,7 @@ class DefaultFlavor(object):
env['LD_LIBRARY_PATH'] = self.m.path.pathsep.join(
'%s' % p for p in ld_library_path)
to_symbolize = ['dm', 'nanobench']
if name in to_symbolize and self.m.vars.is_linux:
# Convert path objects or placeholders into strings such that they can
# be passed to symbolize_stack_trace.py
args = [workdir] + [str(x) for x in cmd]
with self.m.context(cwd=self.m.path['start_dir'].join('skia'), env=env):
self._py('symbolized %s' % name,
self.module.resource('symbolize_stack_trace.py'),
args=args,
infra_step=False)
elif 'Win' in self.m.vars.builder_cfg.get('os', ''):
if 'Win' in self.m.vars.builder_cfg.get('os', ''):
with self.m.context(env=env):
wrapped_cmd = ['powershell', '-ExecutionPolicy', 'Unrestricted',
'-File',

68
infra/bots/recipe_modules/flavor/docker.py
View File

@ -1,68 +0,0 @@
# Copyright 2014 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from . import default
"""Docker flavor, used for running inside a Docker container."""
# TODO(dogben): Move this mapping to a machine-editable file.
# TODO(dogben): Use images without extra packages installed.
IMAGES = {
'gcc-debian10': (
'gcr.io/skia-public/gcc-debian10@sha256:'
'89a72df1e2fdea6f774a3fa4199bb9aaa4a0526a3ac1f233e604d689b694f95c'),
'gcc-debian10-x86': (
'gcr.io/skia-public/gcc-debian10-x86@sha256:'
'b1ec55403ac66d9500d033d6ffd7663894d32335711fbbb0fb4c67dfce812203'),
}
class DockerFlavor(default.DefaultFlavor):
def __init__(self, m, app_name):
super(DockerFlavor, self).__init__(m, app_name)
def _map_host_path_to_docker(self, path):
"""Returns the path in the Docker container mapped to the given path.
Returns None if the path is not mapped into the Docker container.
"""
path = str(path)
for (docker_dir, host_dir) in [
(self.m.docker.mount_out(), str(self.m.vars.swarming_out_dir)),
(self.m.docker.mount_src(), str(self.m.path['start_dir'])),
]:
if path.startswith(host_dir):
return docker_dir + path[len(host_dir):]
return None
def step(self, name, cmd, **unused_kwargs):
extra_tokens = [t for t in self.m.vars.extra_tokens if t != 'Docker']
os = self.m.vars.builder_cfg.get('os', '')
model = self.m.vars.builder_cfg.get('model', '')
cpu_or_gpu = self.m.vars.builder_cfg.get('cpu_or_gpu', '')
arch = self.m.vars.builder_cfg.get('arch', '')
image_name = None
if (os == 'Debian10' and model == 'GCE' and cpu_or_gpu == 'CPU' and
not extra_tokens):
if arch == 'x86_64':
image_name = 'gcc-debian10'
elif arch == 'x86':
image_name = 'gcc-debian10-x86'
if not image_name: # pragma: nocover
raise Exception('Not implemented: ' + self.m.vars.builder_name)
image_hash = IMAGES[image_name]
# TODO(dogben): Currently Linux-specific.
app = self._map_host_path_to_docker(self.device_dirs.bin_dir.join(cmd[0]))
args = [self.m.docker.mount_src(), 'catchsegv', app] + [
self._map_host_path_to_docker(x) or x for x in cmd[1:]]
self.m.docker.run('symbolized %s in Docker' % name, image_hash,
self.m.path['start_dir'], self.m.vars.swarming_out_dir,
self.module.resource('symbolize_stack_trace.py'),
args=args)

7
infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All-MSAN.json
View File

@ -195,20 +195,15 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"[START_DIR]/build/nanobench",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LD_LIBRARY_PATH": "[START_DIR]/clang_linux/msan:[START_DIR]/clang_linux/lib",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/clang_linux/bin"
},
"name": "symbolized nanobench"
"name": "nanobench"
},
{
"name": "$result"

7
infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-ASAN.json
View File

@ -195,14 +195,9 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"[START_DIR]/build/nanobench",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"ASAN_OPTIONS": "symbolize=1 detect_leaks=1",
"CHROME_HEADLESS": "1",
@ -211,7 +206,7 @@
"PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/clang_linux/bin",
"UBSAN_OPTIONS": "symbolize=1 print_stacktrace=1"
},
"name": "symbolized nanobench"
"name": "nanobench"
},
{
"name": "$result"

7
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All-Coverage.json
View File

@ -195,21 +195,16 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LLVM_PROFILE_FILE": "[START_DIR]/[SWARM_OUT_DIR]/All.profraw",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm"
"name": "dm"
},
{
"name": "$result"

7
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-Lottie.json
View File

@ -123,20 +123,15 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm"
"name": "dm"
},
{
"name": "$result"

7
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-TSAN.json
View File

@ -195,21 +195,16 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"[START_DIR]/build/dm",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LD_LIBRARY_PATH": "[START_DIR]/clang_linux/tsan:[START_DIR]/clang_linux/lib",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/clang_linux/bin",
"TSAN_OPTIONS": "report_signal_unsafe=0"
},
"name": "symbolized dm"
"name": "dm"
},
{
"name": "$result"

7
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-GPU-SwiftShader-x86_64-Debug-All-SwiftShader.json
View File

@ -195,21 +195,16 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LD_LIBRARY_PATH": "[START_DIR]/build/swiftshader_out",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm"
"name": "dm"
},
{
"name": "$result"

7
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-NUC7i5BNK-GPU-IntelIris640-x86_64-Debug-All-ASAN_Vulkan.json
View File

@ -195,14 +195,9 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"[START_DIR]/build/dm",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"ASAN_OPTIONS": "symbolize=1 detect_leaks=1 fast_unwind_on_malloc=0",
"CHROME_HEADLESS": "1",
@ -214,7 +209,7 @@
"VK_ICD_FILENAMES": "[START_DIR]/mesa_intel_driver_linux/intel_icd.x86_64.json",
"VULKAN_SDK": "[START_DIR]/linux_vulkan_sdk"
},
"name": "symbolized dm"
"name": "dm"
},
{
"name": "$result"

7
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-NUC7i5BNK-GPU-IntelIris640-x86_64-Debug-All-Vulkan.json
View File

@ -195,15 +195,10 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--some-flag"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LD_LIBRARY_PATH": "[START_DIR]/mesa_intel_driver_linux:[START_DIR]/linux_vulkan_sdk/lib",
@ -213,7 +208,7 @@
"VK_LAYER_PATH": "[START_DIR]/linux_vulkan_sdk/etc/vulkan/explicit_layer.d",
"VULKAN_SDK": "[START_DIR]/linux_vulkan_sdk"
},
"name": "symbolized dm"
"name": "dm"
},
{
"name": "$result"

297
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-GCC-GCE-CPU-AVX2-x86-Debug-All-Docker.json
View File

@ -1,297 +0,0 @@
[
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"file.txt",
"/path/to/tmp/"
],
"infra_step": true,
"name": "read file.txt",
"~followup_annotations": [
"@@@STEP_LOG_END@file.txt@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"remove",
"file.txt"
],
"infra_step": true,
"name": "remove file.txt"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"rmtree",
"results_dir"
],
"infra_step": true,
"name": "rmtree results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"ensure-directory",
"--mode",
"0777",
"results_dir"
],
"infra_step": true,
"name": "makedirs results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"rmtree",
"device_results_dir"
],
"infra_step": true,
"name": "rmtree device_results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"ensure-directory",
"--mode",
"0777",
"device_results_dir"
],
"infra_step": true,
"name": "makedirs device_results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"[START_DIR]/skia/infra/bots/assets/skp/VERSION",
"/path/to/tmp/"
],
"infra_step": true,
"name": "Get skp VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@VERSION@42@@@",
"@@@STEP_LOG_END@VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"42",
"[START_DIR]/tmp/SKP_VERSION"
],
"infra_step": true,
"name": "write SKP_VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@SKP_VERSION@42@@@",
"@@@STEP_LOG_END@SKP_VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"[START_DIR]/skia/infra/bots/assets/skimage/VERSION",
"/path/to/tmp/"
],
"infra_step": true,
"name": "Get skimage VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@VERSION@42@@@",
"@@@STEP_LOG_END@VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"42",
"[START_DIR]/tmp/SK_IMAGE_VERSION"
],
"infra_step": true,
"name": "write SK_IMAGE_VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@SK_IMAGE_VERSION@42@@@",
"@@@STEP_LOG_END@SK_IMAGE_VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"[START_DIR]/skia/infra/bots/assets/svg/VERSION",
"/path/to/tmp/"
],
"infra_step": true,
"name": "Get svg VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@VERSION@42@@@",
"@@@STEP_LOG_END@VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"42",
"[START_DIR]/tmp/SVG_VERSION"
],
"infra_step": true,
"name": "write SVG_VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@SVG_VERSION@42@@@",
"@@@STEP_LOG_END@SVG_VERSION@@@"
]
},
{
"cmd": [],
"name": "Docker setup"
},
{
"cmd": [
"python",
"-u",
"import os\nprint('%d:%d' % (os.getuid(), os.getgid()))\n"
],
"name": "Docker setup.Get uid and gid",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@",
"@@@STEP_LOG_LINE@python.inline@import os@@@",
"@@@STEP_LOG_LINE@python.inline@print('%d:%d' % (os.getuid(), os.getgid()))@@@",
"@@@STEP_LOG_END@python.inline@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"ensure-directory",
"--mode",
"0777",
"[START_DIR]/[SWARM_OUT_DIR]"
],
"infra_step": true,
"name": "Docker setup.mkdirs out_dir",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"chmod",
"777",
"[START_DIR]/[SWARM_OUT_DIR]"
],
"infra_step": true,
"name": "Docker setup.chmod 777 [START_DIR]/[SWARM_OUT_DIR]",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"chmod",
"755",
"[START_DIR]"
],
"infra_step": true,
"name": "Docker setup.chmod 755 [START_DIR]",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"chmod",
"0755",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py"
],
"infra_step": true,
"name": "Docker setup.chmod 0755 RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"docker",
"run",
"--shm-size=2gb",
"--rm",
"--user",
"13:17",
"--mount",
"type=bind,source=[START_DIR],target=/SRC",
"--mount",
"type=bind,source=[START_DIR]/[SWARM_OUT_DIR],target=/OUT",
"gcr.io/skia-public/gcc-debian10-x86@sha256:b1ec55403ac66d9500d033d6ffd7663894d32335711fbbb0fb4c67dfce812203",
"/SRC/../RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"/SRC",
"catchsegv",
"/SRC/build/dm",
"--some-flag"
],
"env": {
"CHROME_HEADLESS": "1",
"DOCKER_CONFIG": "/home/chrome-bot/.docker",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm in Docker"
},
{
"name": "$result"
}
]

297
infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-GCC-GCE-CPU-AVX2-x86_64-Debug-All-Docker.json
View File

@ -1,297 +0,0 @@
[
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"file.txt",
"/path/to/tmp/"
],
"infra_step": true,
"name": "read file.txt",
"~followup_annotations": [
"@@@STEP_LOG_END@file.txt@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"remove",
"file.txt"
],
"infra_step": true,
"name": "remove file.txt"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"rmtree",
"results_dir"
],
"infra_step": true,
"name": "rmtree results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"ensure-directory",
"--mode",
"0777",
"results_dir"
],
"infra_step": true,
"name": "makedirs results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"rmtree",
"device_results_dir"
],
"infra_step": true,
"name": "rmtree device_results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"ensure-directory",
"--mode",
"0777",
"device_results_dir"
],
"infra_step": true,
"name": "makedirs device_results_dir"
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"[START_DIR]/skia/infra/bots/assets/skp/VERSION",
"/path/to/tmp/"
],
"infra_step": true,
"name": "Get skp VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@VERSION@42@@@",
"@@@STEP_LOG_END@VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"42",
"[START_DIR]/tmp/SKP_VERSION"
],
"infra_step": true,
"name": "write SKP_VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@SKP_VERSION@42@@@",
"@@@STEP_LOG_END@SKP_VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"[START_DIR]/skia/infra/bots/assets/skimage/VERSION",
"/path/to/tmp/"
],
"infra_step": true,
"name": "Get skimage VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@VERSION@42@@@",
"@@@STEP_LOG_END@VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"42",
"[START_DIR]/tmp/SK_IMAGE_VERSION"
],
"infra_step": true,
"name": "write SK_IMAGE_VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@SK_IMAGE_VERSION@42@@@",
"@@@STEP_LOG_END@SK_IMAGE_VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"[START_DIR]/skia/infra/bots/assets/svg/VERSION",
"/path/to/tmp/"
],
"infra_step": true,
"name": "Get svg VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@VERSION@42@@@",
"@@@STEP_LOG_END@VERSION@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"copy",
"42",
"[START_DIR]/tmp/SVG_VERSION"
],
"infra_step": true,
"name": "write SVG_VERSION",
"~followup_annotations": [
"@@@STEP_LOG_LINE@SVG_VERSION@42@@@",
"@@@STEP_LOG_END@SVG_VERSION@@@"
]
},
{
"cmd": [],
"name": "Docker setup"
},
{
"cmd": [
"python",
"-u",
"import os\nprint('%d:%d' % (os.getuid(), os.getgid()))\n"
],
"name": "Docker setup.Get uid and gid",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@",
"@@@STEP_LOG_LINE@python.inline@import os@@@",
"@@@STEP_LOG_LINE@python.inline@print('%d:%d' % (os.getuid(), os.getgid()))@@@",
"@@@STEP_LOG_END@python.inline@@@"
]
},
{
"cmd": [
"vpython",
"-u",
"RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py",
"--json-output",
"/path/to/tmp/json",
"ensure-directory",
"--mode",
"0777",
"[START_DIR]/[SWARM_OUT_DIR]"
],
"infra_step": true,
"name": "Docker setup.mkdirs out_dir",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"chmod",
"777",
"[START_DIR]/[SWARM_OUT_DIR]"
],
"infra_step": true,
"name": "Docker setup.chmod 777 [START_DIR]/[SWARM_OUT_DIR]",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"chmod",
"755",
"[START_DIR]"
],
"infra_step": true,
"name": "Docker setup.chmod 755 [START_DIR]",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"chmod",
"0755",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py"
],
"infra_step": true,
"name": "Docker setup.chmod 0755 RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"~followup_annotations": [
"@@@STEP_NEST_LEVEL@1@@@"
]
},
{
"cmd": [
"docker",
"run",
"--shm-size=2gb",
"--rm",
"--user",
"13:17",
"--mount",
"type=bind,source=[START_DIR],target=/SRC",
"--mount",
"type=bind,source=[START_DIR]/[SWARM_OUT_DIR],target=/OUT",
"gcr.io/skia-public/gcc-debian10@sha256:89a72df1e2fdea6f774a3fa4199bb9aaa4a0526a3ac1f233e604d689b694f95c",
"/SRC/../RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"/SRC",
"catchsegv",
"/SRC/build/dm",
"--some-flag"
],
"env": {
"CHROME_HEADLESS": "1",
"DOCKER_CONFIG": "/home/chrome-bot/.docker",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm in Docker"
},
{
"name": "$result"
}
]

2
infra/bots/recipe_modules/flavor/examples/full.py
View File

@ -97,8 +97,6 @@ TEST_BUILDERS = [
'Test-Android-Clang-Nexus5x-GPU-Adreno418-arm64-Release-All-Android_ASAN',
'Test-Android-Clang-Pixel3a-GPU-Adreno615-arm64-Debug-All-Android_Vulkan',
'Test-ChromeOS-Clang-SamsungChromebookPlus-GPU-MaliT860-arm-Release-All',
'Test-Debian10-GCC-GCE-CPU-AVX2-x86-Debug-All-Docker',
'Test-Debian10-GCC-GCE-CPU-AVX2-x86_64-Debug-All-Docker',
'Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All-Coverage',
'Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-Lottie',
'Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-TSAN',

102
infra/bots/recipe_modules/flavor/resources/symbolize_stack_trace.py
View File

@ -1,102 +0,0 @@
#!/usr/bin/env python
# Copyright 2017 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# pylint: disable=line-too-long
from __future__ import print_function
import collections
import os
import re
import subprocess
import sys
# Run a command and symbolize anything that looks like a stacktrace in the
# stdout/stderr. This will return with the same error code as the command.
# First parameter is the current working directory, which will be stripped
# out of stacktraces. The rest of the parameters will be fed to
# subprocess.check_output() and should be the command and arguments that
# will be fed in. If any environment variables are set when running this
# script, they will be automatically used by the call to
# subprocess.check_output().
# This wrapper function is needed to make sure stdout and stderr stay properly
# interleaved, to assist in debugging. There are no clean ways to achieve
# this with recipes. For example, running the dm step with parameters like
# stdout=api.raw_io.output(), stderr=api.raw_io.output() ended up with
# stderr and stdout being separate files, which eliminated the interwoven logs.
# Aside from specifying stdout/stderr, there are no ways to capture or reason
# about the logs of previous steps without using a wrapper like this.
def main(basedir, cmd):
logs = collections.deque(maxlen=500)
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
for line in iter(proc.stdout.readline, ''):
line = line.decode('utf-8')
sys.stdout.write(line)
logs.append(line)
proc.wait()
print('Command exited with code %s' % proc.returncode)
# Stacktraces generally look like:
# /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7fa90e8d0c62]
# /b/s/w/irISUIyA/linux_vulkan_intel_driver_debug/./libvulkan_intel.so(+0x1f4d0a) [0x7fa909eead0a]
# /b/s/w/irISUIyA/out/Debug/dm() [0x17c3c5f]
# The stack_line regex splits those into three parts. Experimentation has
# shown that the address in () works best for external libraries, but our code
# doesn't have that. So, we capture both addresses and prefer using the first
# over the second, unless the first is blank or invalid. Relative offsets
# like abort+0x16a are ignored.
stack_line = r'^(?P<path>.+)\(\+?(?P<addr>.*)\) ?\[(?P<addr2>.+)\]'
# After performing addr2line, the result can be something obnoxious like:
# foo(bar) at /b/s/w/a39kd/Skia/out/Clang/../../src/gpu/Frobulator.cpp:13
# The extra_path strips off the not-useful prefix and leaves just the
# important src/gpu/Frobulator.cpp:13 bit.
extra_path = r'/.*\.\./'
is_first = True
for line in logs:
line = line.strip()
m = re.search(stack_line, line)
if m:
if is_first:
print('#######################################')
print('symbolized stacktrace follows')
print('#######################################')
is_first = False
path = m.group('path')
addr = m.group('addr')
addr2 = m.group('addr2')
if os.path.exists(path):
if not addr or not addr.startswith('0x'):
addr = addr2
try:
sym = subprocess.check_output([
'addr2line', '-Cfpe', path, addr]).decode('utf-8')
except subprocess.CalledProcessError:
sym = ''
sym = sym.strip()
# If addr2line doesn't return anything useful, we don't replace the
# original address, so the human can see it.
if sym and not sym.startswith('?'):
if path.startswith(basedir):
path = path[len(basedir)+1:]
sym = re.sub(extra_path, '', sym)
line = path + ' ' + sym
print(line)
sys.exit(proc.returncode)
if __name__ == '__main__':
if len(sys.argv) < 3:
print('USAGE: %s working_dir cmd_and_args...' % sys.argv[0],
file=sys.stderr)
sys.exit(1)
main(sys.argv[1], sys.argv[2:])

5
infra/bots/recipes/compile.py
View File

@ -75,10 +75,7 @@ def GenTests(api):
repository='https://skia.googlesource.com/skia.git',
revision='abc123',
path_config='kitchen',
swarm_out_dir='[SWARM_OUT_DIR]') +
api.path.exists(
api.path['start_dir'].join('tmp', 'uninteresting_hashes.txt')
)
swarm_out_dir='[SWARM_OUT_DIR]')
)
if 'Win' in builder:
test += api.platform('win', 64)

1
infra/bots/recipes/perf.py
View File

@ -153,7 +153,6 @@ def GenTests(api):
'skimage', 'VERSION'),
api.path['start_dir'].join('skia', 'infra', 'bots', 'assets',
'skp', 'VERSION'),
api.path['start_dir'].join('tmp', 'uninteresting_hashes.txt')
) +
api.step_data('get swarming bot id',
stdout=api.raw_io.output('skia-bot-123')) +

21
infra/bots/recipes/perf_skottietrace.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-SkottieTracing.json
View File

@ -100,10 +100,6 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--resourcePath",
@ -124,12 +120,11 @@
"8888",
"--nogpu"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm"
"name": "dm"
},
{
"cmd": [
@ -235,10 +230,6 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--resourcePath",
@ -259,12 +250,11 @@
"8888",
"--nogpu"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm (2)"
"name": "dm (2)"
},
{
"cmd": [
@ -370,10 +360,6 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--resourcePath",
@ -394,12 +380,11 @@
"8888",
"--nogpu"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm (3)"
"name": "dm (3)"
},
{
"cmd": [

21
infra/bots/recipes/perf_skottietrace.expected/Perf-Debian10-Clang-NUC7i5BNK-GPU-IntelIris640-x86_64-Release-All-SkottieTracing.json
View File

@ -100,10 +100,6 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--resourcePath",
@ -124,7 +120,6 @@
"gles",
"--nocpu"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LD_LIBRARY_PATH": "[START_DIR]/mesa_intel_driver_linux",
@ -132,7 +127,7 @@
"PATH": "<PATH>:RECIPE_REPO[depot_tools]",
"VK_ICD_FILENAMES": "[START_DIR]/mesa_intel_driver_linux/intel_icd.x86_64.json"
},
"name": "symbolized dm"
"name": "dm"
},
{
"cmd": [
@ -238,10 +233,6 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--resourcePath",
@ -262,7 +253,6 @@
"gles",
"--nocpu"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LD_LIBRARY_PATH": "[START_DIR]/mesa_intel_driver_linux",
@ -270,7 +260,7 @@
"PATH": "<PATH>:RECIPE_REPO[depot_tools]",
"VK_ICD_FILENAMES": "[START_DIR]/mesa_intel_driver_linux/intel_icd.x86_64.json"
},
"name": "symbolized dm (2)"
"name": "dm (2)"
},
{
"cmd": [
@ -376,10 +366,6 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--resourcePath",
@ -400,7 +386,6 @@
"gles",
"--nocpu"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"LD_LIBRARY_PATH": "[START_DIR]/mesa_intel_driver_linux",
@ -408,7 +393,7 @@
"PATH": "<PATH>:RECIPE_REPO[depot_tools]",
"VK_ICD_FILENAMES": "[START_DIR]/mesa_intel_driver_linux/intel_icd.x86_64.json"
},
"name": "symbolized dm (3)"
"name": "dm (3)"
},
{
"cmd": [

5
infra/bots/recipes/sync_and_compile.py
View File

@ -102,10 +102,7 @@ def GenTests(api):
repository='https://skia.googlesource.com/skia.git',
revision='abc123',
path_config='kitchen',
swarm_out_dir='[SWARM_OUT_DIR]') +
api.path.exists(
api.path['start_dir'].join('tmp', 'uninteresting_hashes.txt')
)
swarm_out_dir='[SWARM_OUT_DIR]')
)
if 'Win' in builder:
test += api.platform('win', 64)

9
infra/bots/recipes/test.expected/Test-Android-Clang-Pixel2XL-GPU-Adreno540-arm64-Debug-All-Android.json
View File

@ -1113,7 +1113,7 @@
"cmd": [
"python",
"-u",
"\nimport contextlib\nimport math\nimport socket\nimport sys\nimport time\nimport urllib2\n\nHASHES_URL = sys.argv[1]\nRETRIES = 5\nTIMEOUT = 60\nWAIT_BASE = 15\n\nsocket.setdefaulttimeout(TIMEOUT)\nfor retry in range(RETRIES):\n try:\n with contextlib.closing(\n urllib2.urlopen(HASHES_URL, timeout=TIMEOUT)) as w:\n hashes = w.read()\n with open(sys.argv[2], 'w') as f:\n f.write(hashes)\n break\n except Exception as e:\n print('Failed to get uninteresting hashes from %s:' % HASHES_URL)\n print(e)\n if retry == RETRIES:\n raise\n waittime = WAIT_BASE * math.pow(2, retry)\n print('Retry in %d seconds.' % waittime)\n time.sleep(waittime)\n",
"\nimport contextlib\nimport math\nimport socket\nimport sys\nimport time\n\nfrom urllib.request import urlopen\n\nHASHES_URL = sys.argv[1]\nRETRIES = 5\nTIMEOUT = 60\nWAIT_BASE = 15\n\nsocket.setdefaulttimeout(TIMEOUT)\nfor retry in range(RETRIES):\n try:\n with contextlib.closing(\n urlopen(HASHES_URL, timeout=TIMEOUT)) as w:\n hashes = w.read().decode('utf-8')\n with open(sys.argv[2], 'w') as f:\n f.write(hashes)\n break\n except Exception as e:\n print('Failed to get uninteresting hashes from %s:' % HASHES_URL)\n print(e)\n if retry == RETRIES:\n raise\n waittime = WAIT_BASE * math.pow(2, retry)\n print('Retry in %d seconds.' % waittime)\n time.sleep(waittime)\n",
"https://example.com/hashes.txt",
"[START_DIR]/tmp/uninteresting_hashes.txt"
],
@ -1130,7 +1130,8 @@
"@@@STEP_LOG_LINE@python.inline@import socket@@@",
"@@@STEP_LOG_LINE@python.inline@import sys@@@",
"@@@STEP_LOG_LINE@python.inline@import time@@@",
"@@@STEP_LOG_LINE@python.inline@import urllib2@@@",
"@@@STEP_LOG_LINE@python.inline@@@@",
"@@@STEP_LOG_LINE@python.inline@from urllib.request import urlopen@@@",
"@@@STEP_LOG_LINE@python.inline@@@@",
"@@@STEP_LOG_LINE@python.inline@HASHES_URL = sys.argv[1]@@@",
"@@@STEP_LOG_LINE@python.inline@RETRIES = 5@@@",
@ -1141,8 +1142,8 @@
"@@@STEP_LOG_LINE@python.inline@for retry in range(RETRIES):@@@",
"@@@STEP_LOG_LINE@python.inline@ try:@@@",
"@@@STEP_LOG_LINE@python.inline@ with contextlib.closing(@@@",
"@@@STEP_LOG_LINE@python.inline@ urllib2.urlopen(HASHES_URL, timeout=TIMEOUT)) as w:@@@",
"@@@STEP_LOG_LINE@python.inline@ hashes = w.read()@@@",
"@@@STEP_LOG_LINE@python.inline@ urlopen(HASHES_URL, timeout=TIMEOUT)) as w:@@@",
"@@@STEP_LOG_LINE@python.inline@ hashes = w.read().decode('utf-8')@@@",
"@@@STEP_LOG_LINE@python.inline@ with open(sys.argv[2], 'w') as f:@@@",
"@@@STEP_LOG_LINE@python.inline@ f.write(hashes)@@@",
"@@@STEP_LOG_LINE@python.inline@ break@@@",

16
infra/bots/recipes/test.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-Lottie.json
View File

@ -82,7 +82,7 @@
"cmd": [
"python",
"-u",
"\nimport contextlib\nimport math\nimport socket\nimport sys\nimport time\nimport urllib2\n\nHASHES_URL = sys.argv[1]\nRETRIES = 5\nTIMEOUT = 60\nWAIT_BASE = 15\n\nsocket.setdefaulttimeout(TIMEOUT)\nfor retry in range(RETRIES):\n try:\n with contextlib.closing(\n urllib2.urlopen(HASHES_URL, timeout=TIMEOUT)) as w:\n hashes = w.read()\n with open(sys.argv[2], 'w') as f:\n f.write(hashes)\n break\n except Exception as e:\n print('Failed to get uninteresting hashes from %s:' % HASHES_URL)\n print(e)\n if retry == RETRIES:\n raise\n waittime = WAIT_BASE * math.pow(2, retry)\n print('Retry in %d seconds.' % waittime)\n time.sleep(waittime)\n",
"\nimport contextlib\nimport math\nimport socket\nimport sys\nimport time\n\nfrom urllib.request import urlopen\n\nHASHES_URL = sys.argv[1]\nRETRIES = 5\nTIMEOUT = 60\nWAIT_BASE = 15\n\nsocket.setdefaulttimeout(TIMEOUT)\nfor retry in range(RETRIES):\n try:\n with contextlib.closing(\n urlopen(HASHES_URL, timeout=TIMEOUT)) as w:\n hashes = w.read().decode('utf-8')\n with open(sys.argv[2], 'w') as f:\n f.write(hashes)\n break\n except Exception as e:\n print('Failed to get uninteresting hashes from %s:' % HASHES_URL)\n print(e)\n if retry == RETRIES:\n raise\n waittime = WAIT_BASE * math.pow(2, retry)\n print('Retry in %d seconds.' % waittime)\n time.sleep(waittime)\n",
"https://example.com/hashes.txt",
"[START_DIR]/tmp/uninteresting_hashes.txt"
],
@ -99,7 +99,8 @@
"@@@STEP_LOG_LINE@python.inline@import socket@@@",
"@@@STEP_LOG_LINE@python.inline@import sys@@@",
"@@@STEP_LOG_LINE@python.inline@import time@@@",
"@@@STEP_LOG_LINE@python.inline@import urllib2@@@",
"@@@STEP_LOG_LINE@python.inline@@@@",
"@@@STEP_LOG_LINE@python.inline@from urllib.request import urlopen@@@",
"@@@STEP_LOG_LINE@python.inline@@@@",
"@@@STEP_LOG_LINE@python.inline@HASHES_URL = sys.argv[1]@@@",
"@@@STEP_LOG_LINE@python.inline@RETRIES = 5@@@",
@ -110,8 +111,8 @@
"@@@STEP_LOG_LINE@python.inline@for retry in range(RETRIES):@@@",
"@@@STEP_LOG_LINE@python.inline@ try:@@@",
"@@@STEP_LOG_LINE@python.inline@ with contextlib.closing(@@@",
"@@@STEP_LOG_LINE@python.inline@ urllib2.urlopen(HASHES_URL, timeout=TIMEOUT)) as w:@@@",
"@@@STEP_LOG_LINE@python.inline@ hashes = w.read()@@@",
"@@@STEP_LOG_LINE@python.inline@ urlopen(HASHES_URL, timeout=TIMEOUT)) as w:@@@",
"@@@STEP_LOG_LINE@python.inline@ hashes = w.read().decode('utf-8')@@@",
"@@@STEP_LOG_LINE@python.inline@ with open(sys.argv[2], 'w') as f:@@@",
"@@@STEP_LOG_LINE@python.inline@ f.write(hashes)@@@",
"@@@STEP_LOG_LINE@python.inline@ break@@@",
@ -154,10 +155,6 @@
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[skia::flavor]/resources/symbolize_stack_trace.py",
"[START_DIR]",
"catchsegv",
"[START_DIR]/build/dm",
"--example",
@ -179,12 +176,11 @@
"--writePath",
"[START_DIR]/[SWARM_OUT_DIR]"
],
"cwd": "[START_DIR]/skia",
"env": {
"CHROME_HEADLESS": "1",
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"name": "symbolized dm"
"name": "dm"
},
{
"cmd": [

9
infra/bots/recipes/test.expected/Test-Win10-Clang-ShuttleC-GPU-GTX960-x86_64-Debug-All-ANGLE.json
View File

@ -154,7 +154,7 @@
"cmd": [
"python",
"-u",
"\nimport contextlib\nimport math\nimport socket\nimport sys\nimport time\nimport urllib2\n\nHASHES_URL = sys.argv[1]\nRETRIES = 5\nTIMEOUT = 60\nWAIT_BASE = 15\n\nsocket.setdefaulttimeout(TIMEOUT)\nfor retry in range(RETRIES):\n try:\n with contextlib.closing(\n urllib2.urlopen(HASHES_URL, timeout=TIMEOUT)) as w:\n hashes = w.read()\n with open(sys.argv[2], 'w') as f:\n f.write(hashes)\n break\n except Exception as e:\n print('Failed to get uninteresting hashes from %s:' % HASHES_URL)\n print(e)\n if retry == RETRIES:\n raise\n waittime = WAIT_BASE * math.pow(2, retry)\n print('Retry in %d seconds.' % waittime)\n time.sleep(waittime)\n",
"\nimport contextlib\nimport math\nimport socket\nimport sys\nimport time\n\nfrom urllib.request import urlopen\n\nHASHES_URL = sys.argv[1]\nRETRIES = 5\nTIMEOUT = 60\nWAIT_BASE = 15\n\nsocket.setdefaulttimeout(TIMEOUT)\nfor retry in range(RETRIES):\n try:\n with contextlib.closing(\n urlopen(HASHES_URL, timeout=TIMEOUT)) as w:\n hashes = w.read().decode('utf-8')\n with open(sys.argv[2], 'w') as f:\n f.write(hashes)\n break\n except Exception as e:\n print('Failed to get uninteresting hashes from %s:' % HASHES_URL)\n print(e)\n if retry == RETRIES:\n raise\n waittime = WAIT_BASE * math.pow(2, retry)\n print('Retry in %d seconds.' % waittime)\n time.sleep(waittime)\n",
"https://example.com/hashes.txt",
"[START_DIR]\\tmp\\uninteresting_hashes.txt"
],
@ -171,7 +171,8 @@
"@@@STEP_LOG_LINE@python.inline@import socket@@@",
"@@@STEP_LOG_LINE@python.inline@import sys@@@",
"@@@STEP_LOG_LINE@python.inline@import time@@@",
"@@@STEP_LOG_LINE@python.inline@import urllib2@@@",
"@@@STEP_LOG_LINE@python.inline@@@@",
"@@@STEP_LOG_LINE@python.inline@from urllib.request import urlopen@@@",
"@@@STEP_LOG_LINE@python.inline@@@@",
"@@@STEP_LOG_LINE@python.inline@HASHES_URL = sys.argv[1]@@@",
"@@@STEP_LOG_LINE@python.inline@RETRIES = 5@@@",
@ -182,8 +183,8 @@
"@@@STEP_LOG_LINE@python.inline@for retry in range(RETRIES):@@@",
"@@@STEP_LOG_LINE@python.inline@ try:@@@",
"@@@STEP_LOG_LINE@python.inline@ with contextlib.closing(@@@",
"@@@STEP_LOG_LINE@python.inline@ urllib2.urlopen(HASHES_URL, timeout=TIMEOUT)) as w:@@@",
"@@@STEP_LOG_LINE@python.inline@ hashes = w.read()@@@",
"@@@STEP_LOG_LINE@python.inline@ urlopen(HASHES_URL, timeout=TIMEOUT)) as w:@@@",
"@@@STEP_LOG_LINE@python.inline@ hashes = w.read().decode('utf-8')@@@",
"@@@STEP_LOG_LINE@python.inline@ with open(sys.argv[2], 'w') as f:@@@",
"@@@STEP_LOG_LINE@python.inline@ f.write(hashes)@@@",
"@@@STEP_LOG_LINE@python.inline@ break@@@",

61
infra/bots/recipes/test.py
View File

@ -63,39 +63,42 @@ def test_steps(api):
api.python.inline,
'get uninteresting hashes',
program="""
import contextlib
import math
import socket
import sys
import time
import urllib2
import contextlib
import math
import socket
import sys
import time
HASHES_URL = sys.argv[1]
RETRIES = 5
TIMEOUT = 60
WAIT_BASE = 15
from urllib.request import urlopen
socket.setdefaulttimeout(TIMEOUT)
for retry in range(RETRIES):
try:
with contextlib.closing(
urllib2.urlopen(HASHES_URL, timeout=TIMEOUT)) as w:
hashes = w.read()
with open(sys.argv[2], 'w') as f:
f.write(hashes)
break
except Exception as e:
print('Failed to get uninteresting hashes from %s:' % HASHES_URL)
print(e)
if retry == RETRIES:
raise
waittime = WAIT_BASE * math.pow(2, retry)
print('Retry in %d seconds.' % waittime)
time.sleep(waittime)
HASHES_URL = sys.argv[1]
RETRIES = 5
TIMEOUT = 60
WAIT_BASE = 15
socket.setdefaulttimeout(TIMEOUT)
for retry in range(RETRIES):
try:
with contextlib.closing(
urlopen(HASHES_URL, timeout=TIMEOUT)) as w:
hashes = w.read().decode('utf-8')
with open(sys.argv[2], 'w') as f:
f.write(hashes)
break
except Exception as e:
print('Failed to get uninteresting hashes from %s:' % HASHES_URL)
print(e)
if retry == RETRIES:
raise
waittime = WAIT_BASE * math.pow(2, retry)
print('Retry in %d seconds.' % waittime)
time.sleep(waittime)
""",
args=[api.properties['gold_hashes_url'], host_hashes_file],
abort_on_failure=False,
fail_build_on_failure=False,
# If this fails, we want to know about it because it means Gold is down
# and proceeding onwards would take a very long time, but be hard to notice.
abort_on_failure=True,
fail_build_on_failure=True,
infra_step=True)
if api.path.exists(host_hashes_file):

6156
infra/bots/tasks.json
View File

File diff suppressed because it is too large Load Diff