Kevin Lubick 52b040968a [infra] Remove Python2 when running CI tasks

This is similar to https://skia-review.googlesource.com/c/buildbot/+/538218

BIG CHANGE: If we fail to download uninteresting hashes
(because Gold is down or the script otherwise fails), we
now crash/fail our Test-* tasks. In an early version of
this removal, that lack of failure masked an issue with
the script (urllib2 is not in Python3) and nearly would
have landed. Failing loudly is better, IMO.

This removes the symbolize_stack_trace script, which
stopped working with Python3 for reasons unknown.
If we need the behavior, we can rewrite it.
Relatedly, we removed 4 jobs, the Docker ones, because
we will not get much value out of them as we migrate
towards Bazel and removing the symbolization script
was tricky to get right.

There are a few cleanups around copypasta that I noticed
when combing through the recipes.

Change-Id: I8dfab416e964fd494267800b4ebe216061895f19
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/538636
Reviewed-by: Ravi Mistry <rmistry@google.com>
Commit-Queue: Kevin Lubick <kjlubick@google.com>
Reviewed-by: Eric Boren <borenet@google.com>

2022-05-17 20:26:48 +00:00

Repo documentation for skia

Recipe Modules

build (Python3 ✅) — Build Skia for various platforms.
builder_name_schema (Python3 ✅)
checkout (Python3 ✅)
docker (Python3 ✅)
doxygen (Python3 ✅)
env (Python3 ✅)
flavor (Python3 ✅)
git (Python3 ✅)
gold_upload (Python3 ✅)
gsutil (Python3 ✅)
infra (Python3 ✅)
run (Python3 ✅)
vars (Python3 ✅)

Recipes

build:examples/full (Python3 ✅)
builder_name_schema:examples/full (Python3 ✅)
check_generated_files (Python3 ✅)
checkout:examples/full (Python3 ✅)
compile (Python3 ✅)
compute_buildstats (Python3 ✅)
docker:examples/full (Python3 ✅)
doxygen:examples/full (Python3 ✅)
env:examples/full (Python3 ✅)
flavor:examples/full (Python3 ✅)
git:examples/full (Python3 ✅)
gold_upload:examples/full (Python3 ✅)
gsutil:examples/full (Python3 ✅)
housekeeper (Python3 ✅)
infra (Python3 ✅)
infra:examples/full (Python3 ✅)
perf (Python3 ✅)
perf_pathkit (Python3 ✅)
perf_skottietrace (Python3 ✅)
perf_skottiewasm_lottieweb (Python3 ✅)
run:examples/full (Python3 ✅)
skpbench (Python3 ✅)
sync_and_compile (Python3 ✅)
test (Python3 ✅)
test_canvaskit (Python3 ✅)
test_lottie_web (Python3 ✅)
test_pathkit (Python3 ✅)
upload_buildstats_results (Python3 ✅)
upload_dm_results (Python3 ✅)
upload_nano_results (Python3 ✅)
vars:examples/full (Python3 ✅)

Recipe Modules

recipe_modules / build

DEPS: depot_tools/gclient, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/python, recipe_engine/step, docker, env, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

Build Skia for various platforms.

class BuildApi(RecipeApi):

— def __call__(self, checkout_root, out_dir):

Compile the code.

— def copy_build_products(self, out_dir, dst):

Copy selected build products to dst.

recipe_modules / builder_name_schema

PYTHON_VERSION_COMPATIBILITY: PY3

class BuilderNameSchemaApi(RecipeApi):

— def DictForBuilderName(self, *args, **kwargs):

— def MakeBuilderName(self, **kwargs):

recipe_modules / checkout

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class CheckoutApi(RecipeApi):

— def assert_git_is_from_cipd(self):

Fail if git is not obtained from CIPD.

— def bot_update(self, checkout_root, gclient_cache=None, checkout_flutter=False, flutter_android=False):

Run the steps to obtain a checkout using bot_update.

Args: checkout_root: Root directory where the code will be synced. gclient_cache: Optional, directory of the gclient cache. checkout_flutter: If True, will checkout flutter in addition to the primary repo. flutter_android: Indicates that we're checking out flutter for Android.

@property
— def default_checkout_root(self):

The default location for cached persistent checkouts.

— def git(self, checkout_root):

Run the steps to perform a pure-git checkout without DEPS.

recipe_modules / docker

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, env, run

PYTHON_VERSION_COMPATIBILITY: PY3

class DockerApi(RecipeApi):

— def mount_out(self):

— def mount_src(self):

— def run(self, name, docker_image, src_dir, out_dir, script, args=None, docker_args=None, copies=None, recursive_read=None, attempts=1, match_directory_structure=False):

recipe_modules / doxygen

DEPS: recipe_engine/context, recipe_engine/step, run

PYTHON_VERSION_COMPATIBILITY: PY3

class DoxygenApi(RecipeApi):

— def generate_and_upload(self, skia_dir):

recipe_modules / env

DEPS: recipe_engine/context

PYTHON_VERSION_COMPATIBILITY: PY3

class EnvApi(RecipeApi):

recipe_modules / flavor

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, env, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class SkiaFlavorApi(RecipeApi):

— def cleanup_steps(self):

— def copy_directory_contents_to_device(self, host_dir, device_dir):

— def copy_directory_contents_to_host(self, device_dir, host_dir):

— def copy_file_to_device(self, host_path, device_path):

— def create_clean_device_dir(self, path):

— def create_clean_host_dir(self, path):

— def device_path_join(self, *args):

— def get_flavor(self, vars_api, app_name):

Return a flavor utils object specific to the given builder.

— def install(self, skps=False, images=False, lotties=False, svgs=False, resources=False, mskps=False, texttraces=False):

— def read_file_on_device(self, path, **kwargs):

— def remove_file_on_device(self, path):

— def setup(self, app_name):

— def step(self, name, cmd, **kwargs):

recipe_modules / git

DEPS: recipe_engine/path, env

PYTHON_VERSION_COMPATIBILITY: PY3

class GitApi(RecipeApi):

— def env(self):

Add Git to PATH

Requires the infra/git and infra/tools/git CIPD packages to be installed in the 'git' relative path.

recipe_modules / gold_upload

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/properties, recipe_engine/python, recipe_engine/step, recipe_engine/time, flavor, gsutil, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class GoldUploadApi(RecipeApi):

— def upload(self):

Attempt to upload files to Gold. This module assumes setup has occurred for the vars and flavor modules.

recipe_modules / gsutil

DEPS: recipe_engine/context, recipe_engine/python, recipe_engine/step, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class GSUtilApi(RecipeApi):

— def __call__(self, step_name, *args):

Run gsutil with the given args.

This assumes there exists an executable called gsutil on the PATH. This probably only works for Linux/Mac, but those are the only hosts that we try to upload to GCS from anyway.

— def cp(self, name, src, dst, extra_args=None, multithread=False):

Attempt to upload or download files to/from Google Cloud Storage (GCS).

Args: name: string. Will be used to fill out the step name. src: string. Absolute path for a local file or gcs file (e.g. gs://...) dst: string. Same as src. extra_args: optional list of args to be passed to gsutil. e.g. [-Z] asks all files be compressed with gzip after upload and before download. multi_thread: if the -m argument should be used to copy multiple items at once (e.g. gsutil -m cp foo* gs://bar/dir)

If the operation fails, it will be retried multiple times.

recipe_modules / infra

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/raw_io, recipe_engine/step, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class InfraApi(RecipeApi):

@property
— def go_bin(self):

@property
— def go_env(self):

@property
— def gopath(self):

@property
— def goroot(self):

recipe_modules / run

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, env, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class SkiaStepApi(RecipeApi):

— def __call__(self, steptype, name, abort_on_failure=True, fail_build_on_failure=True, **kwargs):

Run a step. If it fails, keep going but mark the build status failed.

— def __init__(self, *args, **kwargs):

Initialize the recipe module.

— def asset_version(self, asset_name, skia_dir, test_data=None):

Return the contents of VERSION for the given asset as a string.

If test_data is not specified, reads the property 'test_<asset_name>_version' or if not present, uses TEST_DEFAULT_ASSET_VERSION.

— def check_failure(self):

Raise an exception if any step failed.

@property
— def failed_steps(self):

— def readfile(self, filename, *args, **kwargs):

Convenience function for reading files.

— def rmtree(self, path):

Wrapper around api.file.rmtree.

— def run_once(self, fn, *args, **kwargs):

— def with_retry(self, steptype, name, attempts, between_attempts_fn=None, abort_on_failure=True, fail_build_on_failure=True, **kwargs):

— def writefile(self, filename, contents):

Convenience function for writing files.

recipe_modules / vars

DEPS: depot_tools/bot_update, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, builder_name_schema

PYTHON_VERSION_COMPATIBILITY: PY3

class SkiaVarsApi(RecipeApi):

@property
— def is_linux(self):

— def setup(self):

Prepare the variables.

@property
— def swarming_bot_id(self):

@property
— def swarming_task_id(self):

Recipes

recipes / build:examples/full

DEPS: recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, build, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / builder_name_schema:examples/full

DEPS: builder_name_schema

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / check_generated_files

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, build, checkout, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / checkout:examples/full

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, checkout, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / compile

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step, build, checkout, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / compute_buildstats

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, checkout, env, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def add_binary_size_output_property(result, source, binary_size):

— def analyze_cpp_lib(api, checkout_root, out_dir, files):

— def analyze_flutter_lib(api, checkout_root, out_dir, files):

— def analyze_wasm_file(api, checkout_root, out_dir, files):

— def analyze_web_file(api, checkout_root, out_dir, files):

— def keys_and_props(api):

— def make_treemap(api, checkout_root, out_dir, files):

recipes / docker:examples/full

DEPS: recipe_engine/context, recipe_engine/properties, recipe_engine/step, docker, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / doxygen:examples/full

DEPS: recipe_engine/path, recipe_engine/properties, doxygen, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / env:examples/full

DEPS: recipe_engine/context, recipe_engine/step, env

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / flavor:examples/full

DEPS: recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, flavor, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def test_exceptions(api):

recipes / git:examples/full

DEPS: recipe_engine/platform, recipe_engine/step, git

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / gold_upload:examples/full

DEPS: recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, flavor, gold_upload, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / gsutil:examples/full

DEPS: recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, gsutil, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / housekeeper

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/properties, checkout, doxygen, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / infra

DEPS: recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/step, infra, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def git_init(api, repo_root, env):

recipes / infra:examples/full

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / perf

DEPS: recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time, env, flavor, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def perf_steps(api):

Run Skia benchmarks.

recipes / perf_pathkit

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, checkout, docker, env, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / perf_skottietrace

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time, flavor, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def get_trace_match(lottie_filename, is_android):

Returns the DM regex to match the specified lottie file name.

— def parse_trace(trace_json, lottie_filename, api):

parse_trace parses the specified trace JSON.

Parses the trace JSON and calculates the time of a single frame. Frame time is considered the same as seek time + render time. Note: The first seek is ignored because it is a constructor call.

A dictionary is returned that has the following structure: { 'frame_max_us': 100, 'frame_min_us': 90, 'frame_avg_us': 95, }

— def perf_steps(api):

Run DM on lottie files with tracing turned on and then parse the output.

recipes / perf_skottiewasm_lottieweb

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, recipe_engine/time, checkout, env, flavor, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def parse_trace(trace_json, lottie_filename, api, renderer):

parse_trace parses the specified trace JSON.

Parses the trace JSON and calculates the time of a single frame. A dictionary is returned that has the following structure: { 'frame_max_us': 100, 'frame_min_us': 90, 'frame_avg_us': 95, }