Eric Boren 7bc4c9a8d1 [python3] Reland recipes -> python3 changes

This was enabled by moving the iPhones off of the old RPI2 hosts.

This reverts commit 04cd6fba97.
This reverts commit a726978ae7.

Bug: chromium:1256037
Change-Id: I35069089aa39baf62a18235c8d0514923f327c53
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/477987
Reviewed-by: Ravi Mistry <rmistry@google.com>
Commit-Queue: Eric Boren <borenet@google.com>

2021-12-01 13:04:03 +00:00

Repo documentation for skia

Recipe Modules

build (Python3 ✅) — Build Skia for various platforms.
builder_name_schema (Python3 ✅)
checkout (Python3 ✅)
docker (Python3 ✅)
doxygen (Python3 ✅)
env (Python3 ✅)
flavor (Python3 ✅)
git (Python3 ✅)
gold_upload (Python3 ✅)
gsutil (Python3 ✅)
infra (Python3 ✅)
run (Python3 ✅)
vars (Python3 ✅)

Recipes

build:examples/full (Python3 ✅)
builder_name_schema:examples/full (Python3 ✅)
check_generated_files (Python3 ✅)
checkout:examples/full (Python3 ✅)
compile (Python3 ✅)
compute_buildstats (Python3 ✅)
docker:examples/full (Python3 ✅)
doxygen:examples/full (Python3 ✅)
env:examples/full (Python3 ✅)
flavor:examples/full (Python3 ✅)
git:examples/full (Python3 ✅)
gold_upload:examples/full (Python3 ✅)
gsutil:examples/full (Python3 ✅)
housekeeper (Python3 ✅)
infra (Python3 ✅)
infra:examples/full (Python3 ✅)
perf (Python3 ✅)
perf_pathkit (Python3 ✅)
perf_skottietrace (Python3 ✅)
perf_skottiewasm_lottieweb (Python3 ✅)
run:examples/full (Python3 ✅)
skpbench (Python3 ✅)
sync_and_compile (Python3 ✅)
test (Python3 ✅)
test_canvaskit (Python3 ✅)
test_lottie_web (Python3 ✅)
test_pathkit (Python3 ✅)
upload_buildstats_results (Python3 ✅)
upload_dm_results (Python3 ✅)
upload_nano_results (Python3 ✅)
vars:examples/full (Python3 ✅)

Recipe Modules

recipe_modules / build

DEPS: depot_tools/gclient, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/python, recipe_engine/step, docker, env, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

Build Skia for various platforms.

class BuildApi(RecipeApi):

— def __call__(self, checkout_root, out_dir):

Compile the code.

— def copy_build_products(self, out_dir, dst):

Copy selected build products to dst.

recipe_modules / builder_name_schema

PYTHON_VERSION_COMPATIBILITY: PY3

class BuilderNameSchemaApi(RecipeApi):

— def DictForBuilderName(self, *args, **kwargs):

— def MakeBuilderName(self, **kwargs):

recipe_modules / checkout

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class CheckoutApi(RecipeApi):

— def assert_git_is_from_cipd(self):

Fail if git is not obtained from CIPD.

— def bot_update(self, checkout_root, gclient_cache=None, checkout_chromium=False, checkout_flutter=False, extra_gclient_env=None, flutter_android=False):

Run the steps to obtain a checkout using bot_update.

Args: checkout_root: Root directory where the code will be synced. gclient_cache: Optional, directory of the gclient cache. checkout_chromium: If True, will check out chromium/src.git in addition to the primary repo. checkout_flutter: If True, will checkout flutter in addition to the primary repo. extra_gclient_env: Map of extra environment variable names to their values to supply while running gclient. flutter_android: Indicates that we're checking out flutter for Android.

@property
— def default_checkout_root(self):

The default location for cached persistent checkouts.

— def git(self, checkout_root):

Run the steps to perform a pure-git checkout without DEPS.

recipe_modules / docker

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, env, run

PYTHON_VERSION_COMPATIBILITY: PY3

class DockerApi(RecipeApi):

— def mount_out(self):

— def mount_src(self):

— def run(self, name, docker_image, src_dir, out_dir, script, args=None, docker_args=None, copies=None, recursive_read=None, attempts=1, match_directory_structure=False):

recipe_modules / doxygen

DEPS: recipe_engine/context, recipe_engine/step, run

PYTHON_VERSION_COMPATIBILITY: PY3

class DoxygenApi(RecipeApi):

— def generate_and_upload(self, skia_dir):

recipe_modules / env

DEPS: recipe_engine/context

PYTHON_VERSION_COMPATIBILITY: PY3

class EnvApi(RecipeApi):

recipe_modules / flavor

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, docker, env, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class SkiaFlavorApi(RecipeApi):

— def cleanup_steps(self):

— def copy_directory_contents_to_device(self, host_dir, device_dir):

— def copy_directory_contents_to_host(self, device_dir, host_dir):

— def copy_file_to_device(self, host_path, device_path):

— def create_clean_device_dir(self, path):

— def create_clean_host_dir(self, path):

— def device_path_join(self, *args):

— def get_flavor(self, vars_api, app_name):

Return a flavor utils object specific to the given builder.

— def install(self, skps=False, images=False, lotties=False, svgs=False, resources=False, mskps=False, texttraces=False):

— def read_file_on_device(self, path, **kwargs):

— def remove_file_on_device(self, path):

— def setup(self, app_name):

— def step(self, name, cmd, **kwargs):

recipe_modules / git

DEPS: recipe_engine/path, env

PYTHON_VERSION_COMPATIBILITY: PY3

class GitApi(RecipeApi):

— def env(self):

Add Git to PATH

Requires the infra/git and infra/tools/git CIPD packages to be installed in the 'git' relative path.

recipe_modules / gold_upload

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/properties, recipe_engine/python, recipe_engine/step, recipe_engine/time, flavor, gsutil, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class GoldUploadApi(RecipeApi):

— def upload(self):

Attempt to upload files to Gold. This module assumes setup has occurred for the vars and flavor modules.

recipe_modules / gsutil

DEPS: recipe_engine/context, recipe_engine/python, recipe_engine/step, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class GSUtilApi(RecipeApi):

— def __call__(self, step_name, *args):

Run gsutil with the given args.

— def cp(self, name, src, dst, extra_args=None, multithread=False):

Attempt to upload or download files to/from Google Cloud Storage (GCS).

Args: name: string. Will be used to fill out the step name. src: string. Absolute path for a local file or gcs file (e.g. gs://...) dst: string. Same as src. extra_args: optional list of args to be passed to gsutil. e.g. [-Z] asks all files be compressed with gzip after upload and before download. multi_thread: if the -m argument should be used to copy multiple items at once (e.g. gsutil -m cp foo* gs://bar/dir)

If the operation fails, it will be retried multiple times.

recipe_modules / infra

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/step, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class InfraApi(RecipeApi):

@property
— def go_bin(self):

@property
— def go_env(self):

@property
— def gopath(self):

@property
— def goroot(self):

recipe_modules / run

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, env, vars

PYTHON_VERSION_COMPATIBILITY: PY3

class SkiaStepApi(RecipeApi):

— def __call__(self, steptype, name, abort_on_failure=True, fail_build_on_failure=True, **kwargs):

Run a step. If it fails, keep going but mark the build status failed.

— def __init__(self, *args, **kwargs):

Initialize the recipe module.

— def asset_version(self, asset_name, skia_dir, test_data=None):

Return the contents of VERSION for the given asset as a string.

If test_data is not specified, reads the property 'test_<asset_name>_version' or if not present, uses TEST_DEFAULT_ASSET_VERSION.

— def check_failure(self):

Raise an exception if any step failed.

@property
— def failed_steps(self):

— def readfile(self, filename, *args, **kwargs):

Convenience function for reading files.

— def rmtree(self, path):

Wrapper around api.file.rmtree.

— def run_once(self, fn, *args, **kwargs):

— def with_retry(self, steptype, name, attempts, between_attempts_fn=None, abort_on_failure=True, fail_build_on_failure=True, **kwargs):

— def writefile(self, filename, contents):

Convenience function for writing files.

recipe_modules / vars

DEPS: depot_tools/bot_update, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, builder_name_schema

PYTHON_VERSION_COMPATIBILITY: PY3

class SkiaVarsApi(RecipeApi):

@property
— def is_linux(self):

— def setup(self):

Prepare the variables.

@property
— def swarming_bot_id(self):

@property
— def swarming_task_id(self):

Recipes

recipes / build:examples/full

DEPS: recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, build, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / builder_name_schema:examples/full

DEPS: builder_name_schema

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / check_generated_files

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, build, checkout, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / checkout:examples/full

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, checkout, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / compile

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step, build, checkout, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / compute_buildstats

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, checkout, env, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def add_binary_size_output_property(result, source, binary_size):

— def analyze_cpp_lib(api, checkout_root, out_dir, files):

— def analyze_flutter_lib(api, checkout_root, out_dir, files):

— def analyze_wasm_file(api, checkout_root, out_dir, files):

— def analyze_web_file(api, checkout_root, out_dir, files):

— def keys_and_props(api):

— def make_treemap(api, checkout_root, out_dir, files):

recipes / docker:examples/full

DEPS: recipe_engine/context, recipe_engine/properties, recipe_engine/step, docker, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / doxygen:examples/full

DEPS: recipe_engine/path, recipe_engine/properties, doxygen, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / env:examples/full

DEPS: recipe_engine/context, recipe_engine/step, env

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / flavor:examples/full

DEPS: recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, flavor, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def test_exceptions(api):

recipes / git:examples/full

DEPS: recipe_engine/platform, recipe_engine/step, git

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / gold_upload:examples/full

DEPS: recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, flavor, gold_upload, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / gsutil:examples/full

DEPS: recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, gsutil, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / housekeeper

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/properties, checkout, doxygen, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / infra

DEPS: recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/step, infra, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def git_init(api, repo_root, env):

recipes / infra:examples/full

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / perf

DEPS: recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time, env, flavor, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def perf_steps(api):

Run Skia benchmarks.

recipes / perf_pathkit

DEPS: recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, checkout, docker, env, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

recipes / perf_skottietrace

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time, flavor, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def get_trace_match(lottie_filename, is_android):

Returns the DM regex to match the specified lottie file name.

— def parse_trace(trace_json, lottie_filename, api):

parse_trace parses the specified trace JSON.

Parses the trace JSON and calculates the time of a single frame. Frame time is considered the same as seek time + render time. Note: The first seek is ignored because it is a constructor call.

A dictionary is returned that has the following structure: { 'frame_max_us': 100, 'frame_min_us': 90, 'frame_avg_us': 95, }

— def perf_steps(api):

Run DM on lottie files with tracing turned on and then parse the output.

recipes / perf_skottiewasm_lottieweb

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step, recipe_engine/time, checkout, env, flavor, infra, run, vars

PYTHON_VERSION_COMPATIBILITY: PY3

— def RunSteps(api):

— def parse_trace(trace_json, lottie_filename, api, renderer):

parse_trace parses the specified trace JSON.

Parses the trace JSON and calculates the time of a single frame. A dictionary is returned that has the following structure: { 'frame_max_us': 100, 'frame_min_us': 90, 'frame_avg_us': 95, }