From c4872ce6442c1d20d8f3352d1f1ddbbe2c3534d5 Mon Sep 17 00:00:00 2001 From: Kevin Lubick Date: Wed, 22 Jun 2022 15:28:10 -0400 Subject: [PATCH] [bazel] Add support for Macs to make Linux RBE builds The big change here is having the C++ toolchain use Bazel platforms instead of the C++ specific flags/setup. In Bazel, platforms are a general purpose way to define things like os, cpu architecture, etc. We were not using platforms previously, because the best documentation at the time focused on the old ways. However, the old ways were clumsy/difficult when trying to manage cross-compilation, specifically when trying to have a Mac host trigger a build on our Linux RBE system targeting a Linux x64 system. Thus, rather than keep investing in the legacy system, this CL migrates us to using platforms where possible. Suggested background reading to better understand this CL: - https://bazel.build/concepts/platforms-intro - https://bazel.build/docs/platforms - https://bazel.build/docs/toolchains#registering-building-toolchains The hermetic toolchain itself is not changing in this CL (and likely does not need to), only how we tell Bazel about it (i.e. registering it) and how Bazel decides to use it (i.e. resolving toolchains). Here is my understanding of how platforms and toolchains interact (supported by some evidence from [1][2]) - Bazel needs to resolve platforms for the Host, Execution, and Target. - If not specified via flags, these are the machine from which Bazel is invoked, aka "@local_config_platform//:host". - With this CL, the Host could be a Mac laptop, the Execution platform is our Linux RBE pool, and the Target is "a Linux system with a x64 CPU" - To specify the Host, that is, describe to Bazel the capabilities of the system it is running on, one can set --host_platform [3] with a label pointing to a platform() containing the appropriate settings. Tip: have this platform inherit from @local_config_platform//:host so it can add to any of the constraint_settings and constraint_values that Bazel deduces automatically. - To specify the Target platform(s), that is, the system on which a final output resides and can execute, one can set the --platforms flag with a label referencing a platform(). - Bazel will then choose an execution platform to fulfill that request. Bazel will look through a list of available platforms, which can be augmented* with the --extra_execution_platforms. Platforms specified by this flag will be considered higher than the default platforms! - Having selected the appropriate platforms, Bazel now needs to select a toolchain to actually run the actions of the appropriate type. - Bazel looks through the list of available toolchains and finds one that "matches" the Execution and the Target platform. This means, the toolchain's exec_compatible_with is a strict subset of the Execution platform and the toolchain's target_compatible_with is a strict subset of the Target platform. To register toolchains* (i.e. add them to the resolution list), we use --extra_toolchains. Once Bazel finds a match, it stops looking. Using --toolchain_resolution_debug=".*" makes Bazel log how it is resolving these toolchains and what execution platform it picked. * We can also register execution platforms and toolchains in WORKSPACE.bazel [4], but the flags come with higher priority and that made resolution a bit tricky. Also, when we want to conditionally add them (e.g. --config=linux_rbe), we cannot remove them conditionally in the WORKSPACE.bazel file. The above resolution flow directly necessitated the changes in this CL. Example usage of the new configs and platforms: # Can be run on a x64 Linux host and uses the hermetic toolchain. bazel build //:skia_public # Can be run on Mac or Linux and uses the Linux RBE system along # with the hermetic toolchain to compile a binary for Linux x64. bazel build //:skia_public --config=linux_rbe --config=for_linux_x64 # Shorthand for above bazel build //:skia_public --config=for_linux_x64_with_rbe Notice we don't have to type out --config=clang_linux anymore! That was due to me reading the Bazel docs more carefully and realizing we can set options for *all* Bazel build commands. Current Limitations: - Targets which require a py_binary (e.g. Dawn's genrules) will not work on RBE when cross compiling because the python runtime we download is for the host machine, not the executor. This means //example:hello_world_dawn does not work on Mac when cross-compiling via linux_rbe. - Mac M1 linking not quite working with SkOpts settings. Probably need to set -target [5] Suggested Review order: - toolchain/BUILD.bazel Notice how we do away with cc_toolchain_suite for toolchain. These have the same role: giving Bazel the information about where a toolchain can run. The platforms one is more expressive (IMO), allowing us to say both where to run the toolchain and what it can make. In order to more easily force the use of our hermetic toolchain, but also allow the hermetic toolchain to be used on RBE, we specify "use_hermetic_toolchain" only on the target, because the RBE image does not have the hermetic toolchain on it by default (but can certainly run it). - bazel/platform/BUILD.bazel to see the custom constraint_setting and corresponding constraint_value. The names for both of these are completely arbitrary - they do not need to have any deeper meaning or relation to any file or Docker image or system or any other constraints. Think of the constraint_setting as an Enum and the constraint_value being the one and only member. We need to pass around a constant value, not a type, so we need to provide the constraint_value (e.g. in toolchain/BUILD.bazel) but not a constraint_setting. However we need a constraint_setting declared so we can make a constraint_value of that "type". Notice the platform declared here - it allows us to force Bazel to use the hermetic toolchain because of the extra constraint_value. - .bazelrc I set a few flags that will be on for all bazel build commands. Importantly, this causes the C++ build logic to use platforms and not the old, bespoke way. I also found a way to avoid using the local toolchain on the host, which will hopefully lead to clearer errors if platforms are mis-specified instead of odd compile errors because the host toolchain is too old or something. There are also a few RBE settings tweaked to be a bit more modern, as well the new shorthands for specifying target platforms (e.g. for_linux_x64). - bazel/buildrc where we have to turn off the platforms logic for emscripten https://github.com/emscripten-core/emsdk/issues/984 - bazel/rbe/BUILD.bazel for a fix in the platform description that makes it work on Mac. - Notice that _m1 has been removed from the mac-related toolchain files because the same toolchain should work on both architectures. - All other changes in any order. [1] https://bazel.build/docs/toolchains#debugging-toolchains [2] https://bazel.build/docs/toolchains#toolchain-resolution [3] https://bazel.build/reference/command-line-reference [4] https://bazel.build/docs/toolchains#registering-building-toolchains [5] https://github.com/google/skia/blob/17dc3f16fc78477503ba1ef484c3b47bc3aab893/gn/skia/BUILD.gn#L258-L271 Change-Id: I515c114099d659639a808f74e47d489a68b7af62 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549737 Reviewed-by: Erik Rose Reviewed-by: Jorge Betancourt --- .bazelrc | 73 ++++++++----- PRESUBMIT.py | 2 +- WORKSPACE.bazel | 2 +- bazel/Makefile | 65 ++++++----- bazel/buildrc | 12 +-- bazel/platform/BUILD.bazel | 48 +++++++++ bazel/rbe/BUILD.bazel | 12 ++- infra/bots/gen_tasks_logic/gen_tasks_logic.go | 2 +- infra/bots/jobs.json | 2 +- infra/bots/tasks.json | 28 ++--- modules/canvaskit/BUILD.bazel | 7 +- toolchain/BUILD.bazel | 101 ++++++++++++------ toolchain/download_linux_amd64_toolchain.bzl | 27 ++--- ...olchain.bzl => download_mac_toolchain.bzl} | 40 ++++--- toolchain/download_toolchains.bzl | 6 +- ...in_config.bzl => mac_toolchain_config.bzl} | 16 +-- .../mac_trampolines/ar_trampoline_mac.sh | 3 +- .../mac_trampolines/clang_trampoline_mac.sh | 2 +- .../mac_trampolines/lld_trampoline_mac.sh | 3 +- toolchain/utils.bzl | 40 +++++++ 20 files changed, 313 insertions(+), 178 deletions(-) create mode 100644 bazel/platform/BUILD.bazel rename toolchain/{download_mac_m1_toolchain.bzl => download_mac_toolchain.bzl} (65%) rename toolchain/{mac_m1_toolchain_config.bzl => mac_toolchain_config.bzl} (97%) create mode 100644 toolchain/utils.bzl diff --git a/.bazelrc b/.bazelrc index 098c5a1eef..67ac68a05b 100644 --- a/.bazelrc +++ b/.bazelrc @@ -1,12 +1,39 @@ -# We don't have a great way to test the host system in order to use the right -# version of the toolchain. So we need to have the user specify what their host is. -build:clang_linux --crosstool_top=//toolchain:clang_suite_linux -build:clang_linux --compiler=host_is_linux_amd64 +# https://bazel.build/concepts/platforms-intro#cxx +# This forces Bazel's C++ rules use platforms to select toolchains instead of the default +# --crosstool_top, --compiler, etc. +build --incompatible_enable_cc_toolchain_resolution -build:clang_mac --crosstool_top=//toolchain:clang_suite_mac -build:clang_mac --apple_crosstool_top=//toolchain:clang_suite_mac -build:clang_mac --compiler=host_is_mac_m1 -build:clang_mac --apple_platform_type=macos +# We do not want Bazel to detect any C++ toolchains on the local machine +# https://github.com/bazelbuild/bazel/blob/4ccc21f2f089971e5f4032397764a4be3549c40a/tools/cpp/cc_configure.bzl#L47 +build --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 + +# We would like developers to not upload to the remote build cache, only the RBE workers themselves. +build --remote_upload_local_results=false + +# This tells Bazel that the host platform can use the hermetic toolchain. This is a small lie until +# Windows support is added. +build --host_platform=//bazel/platform:host_with_hermetic_toolchain + +# Register our toolchains. We do this here, and not in WORKSPACE.bazel because +# --extra_toolchains has priority over register_toolchains and we conditionally add some toolchains +# for RBE. +build --extra_toolchains=//toolchain:clang_linux_x64_toolchain +build --extra_toolchains=//toolchain:clang_mac_x64_toolchain +build --extra_toolchains=//toolchain:clang_mac_arm64_toolchain + +# --platforms refers to the target for which we are compiling. By setting the target to be a +# platform which has our own custom constraint_setting and constraint_value +# (skia_hermetic_toolchain=use_hermetic_toolchain), this causes Bazel to resolve the toolchain +# to be our hermetic one because our hermetic toolchains have that same constraint set in their +# target_compatible_with list. +build:for_linux_x64 --platforms=//bazel/platform:linux_x64_hermetic +build:for_linux_x64_with_rbe --config=linux_rbe --platforms=//bazel/platform:linux_x64_hermetic + +build:for_mac_arm64 --platforms=//bazel/platform:mac_arm64_hermetic +build:for_mac_x64 --platforms=//bazel/platform:mac_x64_hermetic +# some aliases using more common lingo +build:for_mac_m1 --config=for_mac_arm64 +build:for_mac_intel --config=for_mac_x64 # ============================================================================= # Alias to build configurations below. This makes configuring things from @@ -57,15 +84,8 @@ build:remote --jobs=50 build:remote --java_runtime_version=rbe_jdk build:remote --tool_java_runtime_version=rbe_jdk -build:remote --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 -# Platform flags: -# The toolchain container used for execution is defined in the target indicated -# by "extra_execution_platforms", "host_platform" and "platforms". -# More about platforms: https://docs.bazel.build/versions/master/platforms.html - -# Starting with Bazel 0.27.0 strategies do not need to be explicitly -# defined. See https://github.com/bazelbuild/bazel/issues/7480 -build:remote --define=EXECUTOR=remote +# When a remote configuration is chosen, add "remote" to the list of spawn_strategies. +build:remote --spawn_strategy=remote,sandboxed # Enable remote execution so actions are performed on the remote systems. build:remote --remote_executor=grpcs://remotebuildexecution.googleapis.com @@ -83,6 +103,7 @@ build:remote --remote_timeout=180 # default. You can use --google_credentials=some_file.json to use a service # account credential instead. # See https://developers.google.com/remote-build-execution/docs/authentication +# To authenticate, you should run gcloud auth application-default login build:remote --google_default_credentials=true # End of the copied RBE settings @@ -91,17 +112,17 @@ build:remote --google_default_credentials=true # Use the RBE instance on the skia-rbe GCP project. build:remote --remote_instance_name projects/skia-rbe/instances/default_instance -# These settings are specific to compiling on our Linux RBE workers. For example, -# Use the worker pool as specified by the gce_linux_platform platform in -# //bazel/rbe/BUILD.bazel. +# The linxu_rbe config will build on RBE and default to compiling for linux_x64 using +# the hermetic toolchain. build:linux_rbe --config=remote -# Use our hermetic toolchain instead of the clang in the toolchain. -build:linux_rbe --crosstool_top=//toolchain:clang_suite_linux -# We want to run on this RBE platform -build:linux_rbe --extra_execution_platforms=//bazel/rbe:gce_linux_platform -# On the RBE instances, this Java and C++ toolchain are available +# On the RBE instances, we also have the Java and C++ toolchain from the Docker image available. +# These need to come after the --extra_toolchains flags (above) that register our hermetic +# toolchains, because we only want these backup toolchains to be used by Bazel internals, +# if at all. build:linux_rbe --extra_toolchains=//bazel/rbe/gce_linux/java:all build:linux_rbe --extra_toolchains=//bazel/rbe/gce_linux/config:cc-toolchain +# Make the RBE platform available for selection as an Execution platform. +build:linux_rbe --extra_execution_platforms=//bazel/rbe:gce_linux_platform # Import our specified build configurations # https://docs.bazel.build/versions/main/best-practices.html#using-the-bazelrc-file @@ -109,4 +130,4 @@ build:linux_rbe --extra_toolchains=//bazel/rbe/gce_linux/config:cc-toolchain # because we anticipate that file growing large. import %workspace%/bazel/buildrc # Import User's custom builds if they have defined any -try-import %workspace%/bazel/user/buildrc \ No newline at end of file +try-import %workspace%/bazel/user/buildrc diff --git a/PRESUBMIT.py b/PRESUBMIT.py index 68450505f4..915685abba 100644 --- a/PRESUBMIT.py +++ b/PRESUBMIT.py @@ -266,7 +266,7 @@ def _CheckBazelBUILDFiles(input_api, output_api): is_bazel = affected_file_path.endswith('BUILD.bazel') # This list lines up with the one in autoroller_lib.py (see G3). excluded_paths = ["infra/", "bazel/rbe/", "bazel/external/", "bazel/common_config_settings/", - "modules/canvaskit/go/", "experimental/"] + "modules/canvaskit/go/", "experimental/", "bazel/platform"] is_excluded = any(affected_file_path.startswith(n) for n in excluded_paths) if is_bazel and not is_excluded: with open(affected_file_path, 'r') as file: diff --git a/WORKSPACE.bazel b/WORKSPACE.bazel index 18eb52c399..fb9357465c 100644 --- a/WORKSPACE.bazel +++ b/WORKSPACE.bazel @@ -2,7 +2,7 @@ workspace(name = "skia") load("//toolchain:download_toolchains.bzl", "download_toolchains_for_skia") -download_toolchains_for_skia("clang_linux_amd64", "clang_mac_m1") +download_toolchains_for_skia("clang_linux_amd64", "clang_mac") load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") diff --git a/bazel/Makefile b/bazel/Makefile index 61ed3cb9a4..d6cb4493ae 100644 --- a/bazel/Makefile +++ b/bazel/Makefile @@ -10,56 +10,55 @@ gazelle_update_repo: # Run this target to test all known working Bazel builds known_good_builds: - bazelisk build //:skia_public --config=clang_linux - bazelisk build //:skia_internal --config=clang_linux - bazelisk build //experimental/bazel_test/... --config=clang_linux - bazelisk run //experimental/bazel_test:bazel_test_exe --config=clang_linux - bazelisk build //example:hello_world_gl --config=clang_linux - bazelisk build //example:hello_world_vulkan --config=clang_linux - bazelisk build //example:hello_world_dawn --config=clang_linux - bazelisk build //example:vulkan_basic --config=clang_linux - bazelisk build //src/sksl/lex:sksllex --config=clang_linux - bazelisk build //tools/skdiff --config=clang_linux - bazelisk build //tools/skslc --config=clang_linux + bazelisk build //:skia_public + bazelisk build //:skia_internal + bazelisk build //experimental/bazel_test/... + bazelisk run //experimental/bazel_test:bazel_test_exe + bazelisk build //example:hello_world_gl + bazelisk build //example:hello_world_vulkan + bazelisk build //example:hello_world_dawn + bazelisk build //example:vulkan_basic + bazelisk build //src/sksl/lex:sksllex + bazelisk build //tools/skdiff + bazelisk build //tools/skslc bazelisk build //modules/canvaskit:canvaskit_wasm --config=ck_release rbe_known_good_builds: - bazelisk build //:skia_public --config=linux_rbe --remote_download_minimal - bazelisk build //:skia_internal --config=linux_rbe --remote_download_minimal - bazelisk build //experimental/bazel_test/... --config=linux_rbe --remote_download_minimal - bazelisk run //experimental/bazel_test:bazel_test_exe --config=linux_rbe --remote_download_toplevel - bazelisk build //example:hello_world_gl --config=linux_rbe --remote_download_minimal - bazelisk build //example:hello_world_vulkan --config=linux_rbe --remote_download_minimal - bazelisk build //example:hello_world_dawn --config=linux_rbe --remote_download_minimal - bazelisk build //example:vulkan_basic --config=linux_rbe --remote_download_minimal - bazelisk build //src/sksl/lex:sksllex --config=linux_rbe --remote_download_minimal - bazelisk build //tools/skdiff --config=linux_rbe --remote_download_minimal - bazelisk build //tools/skslc --config=linux_rbe --remote_download_minimal + bazelisk build //:skia_public --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //:skia_internal --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //experimental/bazel_test:bazel_test_exe --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //example:hello_world_gl --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //example:hello_world_vulkan --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //example:hello_world_dawn --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //example:vulkan_basic --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //src/sksl/lex:sksllex --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //tools/skdiff --config=for_linux_x64_with_rbe --remote_download_minimal + bazelisk build //tools/skslc --config=for_linux_x64_with_rbe --remote_download_minimal ## TODO(kjlubick) CanvasKit in release mode (i.e. with Closure) requires ## https://github.com/emscripten-core/emscripten/pull/16640 to land - bazelisk build //modules/canvaskit:canvaskit_wasm --compilation_mode dbg --config=linux_rbe \ + bazelisk build //modules/canvaskit:canvaskit_wasm --config=linux_rbe --config=ck_debug \ --remote_download_minimal iwyu_rbe: - bazelisk build //:skia_public --config=linux_rbe --config=enforce_iwyu \ + bazelisk build //:skia_public --config=for_linux_x64_with_rbe --config=enforce_iwyu \ --keep_going --remote_download_minimal - bazelisk build //example:hello_world_gl --config=linux_rbe --config=enforce_iwyu \ + bazelisk build //example:hello_world_gl --config=for_linux_x64_with_rbe --config=enforce_iwyu \ --keep_going --remote_download_minimal - bazelisk build //example:hello_world_vulkan --config=linux_rbe --config=enforce_iwyu \ + bazelisk build //example:hello_world_vulkan --config=for_linux_x64_with_rbe --config=enforce_iwyu \ --keep_going --remote_download_minimal - bazelisk build //example:hello_world_dawn --config=linux_rbe --config=enforce_iwyu \ + bazelisk build //example:hello_world_dawn --config=for_linux_x64_with_rbe --config=enforce_iwyu \ --keep_going --remote_download_minimal - bazelisk build //example:vulkan_basic --config=linux_rbe --config=enforce_iwyu \ + bazelisk build //example:vulkan_basic --config=for_linux_x64_with_rbe --config=enforce_iwyu \ --keep_going --remote_download_minimal iwyu: - bazelisk build //:skia_public --config=clang_linux --config=enforce_iwyu \ + bazelisk build //:skia_public --config=enforce_iwyu \ --keep_going - bazelisk build //example:hello_world_gl --config=clang_linux --config=enforce_iwyu \ + bazelisk build //example:hello_world_gl --config=enforce_iwyu \ --keep_going - bazelisk build //example:hello_world_vulkan --config=clang_linux --config=enforce_iwyu \ + bazelisk build //example:hello_world_vulkan --config=enforce_iwyu \ --keep_going - bazelisk build //example:hello_world_dawn --config=clang_linux --config=enforce_iwyu \ + bazelisk build //example:hello_world_dawn --config=enforce_iwyu \ --keep_going - bazelisk build //example:vulkan_basic --config=clang_linux --config=enforce_iwyu \ + bazelisk build //example:vulkan_basic --config=enforce_iwyu \ --keep_going diff --git a/bazel/buildrc b/bazel/buildrc index 967c3ce24a..4ae2677592 100644 --- a/bazel/buildrc +++ b/bazel/buildrc @@ -27,15 +27,9 @@ build:release --compilation_mode=opt build:debug --compilation_mode=dbg -# We use spawn_strategy=local for CanvasKit builds because emscripten assumes there -# is a cache in the home directory that it needs to fill with compiled versions of libc etc. -# https://emscripten.org/docs/tools_reference/emcc.html -# By setting spawn_strategy=local, we can avoid recompiling all of this for every compilation -# unit, by letting the cache be used (and not dropped from the sandbox), which gets expensive. -# Local testing showed using the local strategy sped up a clean build from 9.5 minutes -# to 1 minute. https://bazel.build/docs/user-manual#execution-strategy -build:ck_release --config=release --spawn_strategy=local -build:ck_debug --config=debug --spawn_strategy=local +# See also https://github.com/emscripten-core/emsdk/issues/984 for disabling cc_toolchain_resolution +build:ck_release --config=release --noincompatible_enable_cc_toolchain_resolution +build:ck_debug --config=debug --noincompatible_enable_cc_toolchain_resolution # We only want to enforce IWYU on debug builds because we have some things that are only # necessary to include in debug mode (e.g. SkDEBUGCODE), but very rarely something that is diff --git a/bazel/platform/BUILD.bazel b/bazel/platform/BUILD.bazel new file mode 100644 index 0000000000..9c296f5ac8 --- /dev/null +++ b/bazel/platform/BUILD.bazel @@ -0,0 +1,48 @@ +# https://bazel.build/concepts/platforms-intro +# https://bazel.build/docs/platforms +platform( + name = "linux_x64_hermetic", + constraint_values = [ + "@platforms//os:linux", # https://github.com/bazelbuild/platforms/blob/main/os/BUILD + "@platforms//cpu:x86_64", # https://github.com/bazelbuild/platforms/blob/main/cpu/BUILD + ":use_hermetic_toolchain", + ], +) + +platform( + name = "mac_x64_hermetic", + constraint_values = [ + "@platforms//os:macos", + "@platforms//cpu:x86_64", + ":use_hermetic_toolchain", + ], +) + +platform( + name = "mac_arm64_hermetic", + constraint_values = [ + "@platforms//os:macos", + "@platforms//cpu:arm64", + ":use_hermetic_toolchain", + ], +) + +platform( + name = "host_with_hermetic_toolchain", + constraint_values = [ + ":use_hermetic_toolchain", + ], + parents = ["@local_config_platform//:host"], +) + +# This constraint allows us to force Bazel to resolve our hermetic toolchain to build +# the target and not a default one (e.g. on the Linux RBE instance). We do this by +# adding the constraint to our platforms that describe the target we want Bazel to build for. +# https://bazel.build/reference/be/platform#constraint_setting +constraint_setting(name = "skia_hermetic_toolchain") + +constraint_value( + name = "use_hermetic_toolchain", + constraint_setting = ":skia_hermetic_toolchain", + visibility = ["//visibility:public"], +) diff --git a/bazel/rbe/BUILD.bazel b/bazel/rbe/BUILD.bazel index 79c2312fb0..dfa5b4de23 100644 --- a/bazel/rbe/BUILD.bazel +++ b/bazel/rbe/BUILD.bazel @@ -3,12 +3,19 @@ load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "create_rbe licenses(["notice"]) +# This platform describes our Linux RBE environment. The machines (aka workers) have x64 cpus +# and will use the specified Docker container, which is (Debian) Linux and has Clang in it. +# +# exec_properties specify some information that is passed along the Remote Execution API (REAPI) +# to the dispatcher which will use it to direct the request to the appropriate machine/worker, +# using the Remote Worker API (RWAPI). # https://bazel.build/docs/platforms +# http://go/skolo-rbe platform( name = "gce_linux_platform", constraint_values = [ - "@bazel_tools//platforms:linux", - "@bazel_tools//platforms:x86_64", + "@platforms//os:linux", + "@platforms//cpu:x86_64", "@bazel_tools//tools/cpp:clang", ], exec_properties = create_rbe_exec_properties_dict( @@ -16,5 +23,4 @@ platform( os_family = "Linux", pool = "gce_linux", ), - parents = ["@local_config_platform//:host"], ) diff --git a/infra/bots/gen_tasks_logic/gen_tasks_logic.go b/infra/bots/gen_tasks_logic/gen_tasks_logic.go index 53b35d7753..76fa6a259f 100644 --- a/infra/bots/gen_tasks_logic/gen_tasks_logic.go +++ b/infra/bots/gen_tasks_logic/gen_tasks_logic.go @@ -2103,7 +2103,7 @@ func (b *jobBuilder) bazelBuild() { b.cipd(b.MustGetCipdPackageFromAsset("bazel_build_task_driver")) // We want all Linux Bazel Builds to use RBE - cmd = append(cmd, "--bazel_arg=--config=linux_rbe") + cmd = append(cmd, "--bazel_arg=--config=for_linux_x64_with_rbe") cmd = append(cmd, "--bazel_arg=--jobs=100") cmd = append(cmd, "--bazel_arg=--remote_download_minimal") } else { diff --git a/infra/bots/jobs.json b/infra/bots/jobs.json index 29a6b98456..323f5f15de 100644 --- a/infra/bots/jobs.json +++ b/infra/bots/jobs.json @@ -1,5 +1,5 @@ [ - {"name": "BazelBuild-modules_canvaskit_canvaskit_wasm-debug-linux_x64"}, + {"name": "BazelBuild-modules_canvaskit_canvaskit_wasm-ck_debug-linux_x64"}, {"name": "BazelBuild-example_hello_world_dawn-enforce_iwyu-linux_x64"}, {"name": "BazelBuild-example_hello_world_dawn-release-linux_x64", "cq_config": {} diff --git a/infra/bots/tasks.json b/infra/bots/tasks.json index 744a199156..82fa8f47ad 100755 --- a/infra/bots/tasks.json +++ b/infra/bots/tasks.json @@ -30,9 +30,9 @@ "BazelBuild-example_hello_world_vulkan-release-linux_x64" ] }, - "BazelBuild-modules_canvaskit_canvaskit_wasm-debug-linux_x64": { + "BazelBuild-modules_canvaskit_canvaskit_wasm-ck_debug-linux_x64": { "tasks": [ - "BazelBuild-modules_canvaskit_canvaskit_wasm-debug-linux_x64" + "BazelBuild-modules_canvaskit_canvaskit_wasm-ck_debug-linux_x64" ] }, "BazelBuild-skia_public-enforce_iwyu-linux_x64": { @@ -3257,7 +3257,7 @@ "--label=//example:hello_world_dawn", "--config=enforce_iwyu", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3299,7 +3299,7 @@ "--label=//example:hello_world_dawn", "--config=release", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3341,7 +3341,7 @@ "--label=//example:hello_world_gl", "--config=enforce_iwyu", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3383,7 +3383,7 @@ "--label=//example:hello_world_gl", "--config=release", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3425,7 +3425,7 @@ "--label=//example:hello_world_vulkan", "--config=enforce_iwyu", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3467,7 +3467,7 @@ "--label=//example:hello_world_vulkan", "--config=release", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3487,7 +3487,7 @@ "max_attempts": 1, "service_account": "skia-external-compile-tasks@skia-swarming-bots.iam.gserviceaccount.com" }, - "BazelBuild-modules_canvaskit_canvaskit_wasm-debug-linux_x64": { + "BazelBuild-modules_canvaskit_canvaskit_wasm-ck_debug-linux_x64": { "casSpec": "compile", "cipd_packages": [ { @@ -3505,12 +3505,12 @@ "bazel_build_task_driver/bazel_build", "--project_id=skia-swarming-bots", "--task_id=<(TASK_ID)", - "--task_name=BazelBuild-modules_canvaskit_canvaskit_wasm-debug-linux_x64", + "--task_name=BazelBuild-modules_canvaskit_canvaskit_wasm-ck_debug-linux_x64", "--label=//modules/canvaskit:canvaskit_wasm", - "--config=debug", + "--config=ck_debug", "--workdir=.", "--expunge_cache", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3552,7 +3552,7 @@ "--label=//:skia_public", "--config=enforce_iwyu", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], @@ -3594,7 +3594,7 @@ "--label=//:skia_public", "--config=release", "--workdir=.", - "--bazel_arg=--config=linux_rbe", + "--bazel_arg=--config=for_linux_x64_with_rbe", "--bazel_arg=--jobs=100", "--bazel_arg=--remote_download_minimal" ], diff --git a/modules/canvaskit/BUILD.bazel b/modules/canvaskit/BUILD.bazel index 0407104605..1e74b962f1 100644 --- a/modules/canvaskit/BUILD.bazel +++ b/modules/canvaskit/BUILD.bazel @@ -74,6 +74,9 @@ CK_DEFINES = [ }) + select({ ":enable_runtime_effect_true": ["CK_INCLUDE_RUNTIME_EFFECT=1"], ":enable_runtime_effect_false": [], +}) + select({ + "//bazel/common_config_settings:gl_backend": ["CK_ENABLE_WEBGL"], + "//conditions:default": [], }) CK_RELEASE_OPTS = [ @@ -200,8 +203,6 @@ JS_INTERFACE_FILES = [ "cpu.js", "debug.js", "font.js", - "webgl.js", - "webgpu.js", "interface.js", "matrix.js", "memory.js", @@ -215,6 +216,8 @@ JS_INTERFACE_FILES = [ "skottie.js", "skp.js", "util.js", + "webgl.js", + "webgpu.js", ] + [ "htmlcanvas/canvas2dcontext.js", "htmlcanvas/color.js", diff --git a/toolchain/BUILD.bazel b/toolchain/BUILD.bazel index ef34ce29da..252f03e885 100644 --- a/toolchain/BUILD.bazel +++ b/toolchain/BUILD.bazel @@ -1,38 +1,73 @@ load("//bazel:macros.bzl", "exports_files_legacy") load("@rules_cc//cc:defs.bzl", "cc_toolchain", "cc_toolchain_suite") load(":linux_amd64_toolchain_config.bzl", "provide_linux_amd64_toolchain_config") -load(":mac_m1_toolchain_config.bzl", "provide_mac_m1_toolchain_config") +load(":mac_toolchain_config.bzl", "provide_mac_toolchain_config") licenses(["notice"]) exports_files_legacy() -# https://bazel.build/reference/be/c-cpp#cc_toolchain_suite -# cc_toolchain_suite will fetch deps for toolchains it will not use, which -# is why we split up the suites by OS. When attempting to fetch "@clang_linux_amd64//:all_files", -# a script compiled for linux to extract ar archives will break on other platforms. -cc_toolchain_suite( - name = "clang_suite_linux", - toolchains = { - # The key is target_cpu|compiler - # compiler appears to be a string we can choose arbitrarily - # https://bazel.build/reference/command-line-reference?hl=en#flag--compiler - "k8|host_is_linux_amd64": ":linux_amd64_host", - "x86_64|host_is_linux_amd64": ":linux_amd64_host", - "k8": ":linux_amd64_host", - }, +# https://bazel.build/docs/toolchains +# https://bazel.build/reference/be/platform#toolchain +toolchain( + name = "clang_linux_x64_toolchain", + # Where should we run this toolchain? + exec_compatible_with = [ + "@platforms//os:linux", + "@platforms//cpu:x86_64", + # We do not want an extra constraint here related to the hermetic toolchain because + # we want the toolchain to run on any Linux x64 machine (and it certainly can). + ], + # What can this toolchain build? + target_compatible_with = [ + "@platforms//os:linux", + "@platforms//cpu:x86_64", + # We want to be able to explicitly tell Bazel to use this toolchain, and not the + # default one on a user's machine or on the RBE worker. Thus we need an extra constraint + # that we can use to differentiate the "stock" C++ toolchain from our hermetic one and + # force that use by specifying the target platform. + "//bazel/platform:use_hermetic_toolchain", + ], + toolchain = ":linux_amd64_host", + # https://github.com/bazelbuild/rules_cc/blob/8bb0eb5c5ccd96b91753bb112096bb6993d16d13/cc/BUILD#L32-L36 + toolchain_type = "@rules_cc//cc:toolchain_type", ) -cc_toolchain_suite( - name = "clang_suite_mac", - toolchains = { - "arm64|host_is_mac_m1": ":mac_m1_host", - "darwin|host_is_mac_m1": ":mac_m1_host", - "darwin_arm64|host_is_mac_m1": ":mac_m1_host", - "darwin_x86_64": ":mac_m1_host", - }, +# Our one mac toolchain can run on either Intel Macs or M1 Macs, however Bazel does not allow you to specify +# more than one cpu type in exec_compatible_with. Thus, we list the same toolchain twice. +toolchain( + name = "clang_mac_x64_toolchain", + exec_compatible_with = [ + "@platforms//os:macos", + "@platforms//cpu:x86_64", + ], + target_compatible_with = [ + "@platforms//os:macos", + "@platforms//cpu:x86_64", + "//bazel/platform:use_hermetic_toolchain", + ], + toolchain = ":mac_host", + # https://github.com/bazelbuild/rules_cc/blob/8bb0eb5c5ccd96b91753bb112096bb6993d16d13/cc/BUILD#L32-L36 + toolchain_type = "@rules_cc//cc:toolchain_type", ) +toolchain( + name = "clang_mac_arm64_toolchain", + exec_compatible_with = [ + "@platforms//os:macos", + "@platforms//cpu:arm64", + ], + target_compatible_with = [ + "@platforms//os:macos", + "@platforms//cpu:arm64", + "//bazel/platform:use_hermetic_toolchain", + ], + toolchain = ":mac_host", + # https://github.com/bazelbuild/rules_cc/blob/8bb0eb5c5ccd96b91753bb112096bb6993d16d13/cc/BUILD#L32-L36 + toolchain_type = "@rules_cc//cc:toolchain_type", +) + + filegroup(name = "not_implemented") filegroup( @@ -62,12 +97,12 @@ filegroup( ) filegroup( - name = "all_mac_m1_files", + name = "all_mac_files", srcs = [ "mac_trampolines/ar_trampoline_mac.sh", "mac_trampolines/clang_trampoline_mac.sh", "mac_trampolines/lld_trampoline_mac.sh", - "@clang_mac_m1//:all_files", + "@clang_mac//:all_files", ], ) @@ -75,8 +110,8 @@ provide_linux_amd64_toolchain_config( name = "linux_amd64_toolchain_config", ) -provide_mac_m1_toolchain_config( - name = "mac_m1_toolchain_config", +provide_mac_toolchain_config( + name = "mac_toolchain_config", ) # https://bazel.build/reference/be/c-cpp#cc_toolchain @@ -94,14 +129,14 @@ cc_toolchain( ) cc_toolchain( - name = "mac_m1_host", - all_files = ":all_mac_m1_files", - ar_files = ":all_mac_m1_files", - compiler_files = ":all_mac_m1_files", + name = "mac_host", + all_files = ":all_mac_files", + ar_files = ":all_mac_files", + compiler_files = ":all_mac_files", dwp_files = ":not_implemented", - linker_files = ":all_mac_m1_files", + linker_files = ":all_mac_files", objcopy_files = ":not_implemented", strip_files = ":not_implemented", supports_param_files = False, - toolchain_config = ":mac_m1_toolchain_config", + toolchain_config = ":mac_toolchain_config", ) diff --git a/toolchain/download_linux_amd64_toolchain.bzl b/toolchain/download_linux_amd64_toolchain.bzl index 6be9a57493..edda3bea1d 100644 --- a/toolchain/download_linux_amd64_toolchain.bzl +++ b/toolchain/download_linux_amd64_toolchain.bzl @@ -13,23 +13,13 @@ resolver) and extracted to which will act as our sysroot. """ +load("//toolchain:utils.bzl", "gcs_mirror_url") + # From https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz.sha256 clang_prefix = "clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/" clang_sha256 = "2c2fb857af97f41a5032e9ecadf7f78d3eff389a5cd3c9ec620d24f134ceb3c8" clang_url = "https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz" -# Files are expected to be in the mirror location named after their sha256 hash. The files should -# still have their file extension, as some of the Starlark functions sniff the file extension -# (e.g. download_and_extract). See //bazel/gcs_mirror for an automated way to update this mirror. -mirror_prefix = "https://storage.googleapis.com/skia-world-readable/bazel/" - -# Set this to True to only use the files from the mirror host. This can be used to test the data -# in the mirrors is not corrupt and publicly accessible. -# If testing this, you need to delete the download cache, which defaults to -# ~/.cache/bazel/_bazel_$USER/cache/repos/v1/ -# https://bazel.build/docs/build#repository-cache -force_test_of_mirrors = False - debs_to_install = [ # These three comprise glibc. libc6 has the shared libraries, like libc itself, the math library # (libm), etc. linux-libc-dev has the header files specific to linux. libc6-dev has the libc @@ -154,7 +144,7 @@ def _download_and_extract_deb(ctx, deb, sha256, prefix, output = ""): # A .deb file has a data.tar.xz and a control.tar.xz, but the important contents # (i.e. the headers or libs) are in the data.tar.xz ctx.download_and_extract( - url = _mirror([deb, mirror_prefix + sha256 + ".deb"]), + url = gcs_mirror_url(deb, sha256), output = "tmp", sha256 = sha256, ) @@ -173,7 +163,7 @@ def _download_linux_amd64_toolchain_impl(ctx): # Download the clang toolchain (the extraction can take a while) # https://bazel.build/rules/lib/repository_ctx#download_and_extract ctx.download_and_extract( - url = _mirror([clang_url, mirror_prefix + clang_sha256 + ".tar.xz"]), + url = gcs_mirror_url(clang_url, clang_sha256), output = "", stripPrefix = clang_prefix, sha256 = clang_sha256, @@ -198,6 +188,8 @@ def _download_linux_amd64_toolchain_impl(ctx): ctx.file( "BUILD.bazel", content = """ +# DO NOT EDIT THIS BAZEL FILE DIRECTLY +# Generated from ctx.file action in download_linux_amd64_toolchain.bzl filegroup( name = "archive_files", srcs = [ @@ -247,13 +239,6 @@ filegroup( executable = False, ) -# If force_test_of_mirrors is set, return a list containing only the second item. This assumes -# that the given list will have a primary source and a mirror source (precisely two items). -def _mirror(arr): - if force_test_of_mirrors: - return [arr[1]] - return arr - # https://bazel.build/rules/repository_rules download_linux_amd64_toolchain = repository_rule( implementation = _download_linux_amd64_toolchain_impl, diff --git a/toolchain/download_mac_m1_toolchain.bzl b/toolchain/download_mac_toolchain.bzl similarity index 65% rename from toolchain/download_mac_m1_toolchain.bzl rename to toolchain/download_mac_toolchain.bzl index 2b61b22180..a47684fa31 100644 --- a/toolchain/download_mac_m1_toolchain.bzl +++ b/toolchain/download_mac_toolchain.bzl @@ -1,28 +1,31 @@ """ -This file assembles a toolchain for a Mac M1 host using the Clang Compiler and glibc. +This file assembles a toolchain for a Mac host (either M1 or Intel) using the Clang Compiler +and a locally-installed XCode. -It downloads the necessary headers, executables, and pre-compiled static/shared libraries to -the external subfolder of the Bazel cache (the same place third party deps are downloaded with -http_archive or similar functions in WORKSPACE.bazel). These will be able to be used via our -custom c++ toolchain configuration (see //toolchain/mac_m1_toolchain_config.bzl) +It downloads the necessary executables and creates symlinks in the external subfolder of the Bazel +cache (the same place third party deps are downloaded with http_archive or similar functions in +WORKSPACE.bazel). These will be able to be used via our +custom c++ toolchain configuration (see //toolchain/mac_toolchain_config.bzl) -Clang files are downloaded in tars while the rest of the required files are found in the user's local -Xcode directory (with us acting as the dependency resolver) and extracted (or symlinked) to - [outputRoot (aka Bazel cache)]/[outputUserRoot]/[outputBase]/external/clang_mac_m1 +The destination folder for these files and symlinks are: + [outputRoot (aka Bazel cache)]/[outputUserRoot]/[outputBase]/external/clang_mac (See https://bazel.build/docs/output_directories#layout-diagram) -which will act as our sysroot. """ +load("//toolchain:utils.bzl", "gcs_mirror_url") + # From https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang+llvm-13.0.0-x86_64-apple-darwin.tar.xz +# When updating this, don't forget to use //bazel/gcs_mirror to upload a new version. +# go run bazel/gcs_mirror/gcs_mirror.go --url [clang_url] --sha256 [clang_sha256] clang_prefix = "clang+llvm-13.0.0-x86_64-apple-darwin" clang_sha256 = "d051234eca1db1f5e4bc08c64937c879c7098900f7a0370f3ceb7544816a8b09" clang_url = "https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang+llvm-13.0.0-x86_64-apple-darwin.tar.xz" -def _download_mac_m1_toolchain_impl(ctx): +def _download_mac_toolchain_impl(ctx): # Download the clang toolchain (the extraction can take a while) # https://bazel.build/rules/lib/repository_ctx#download_and_extract ctx.download_and_extract( - url = clang_url, + url = gcs_mirror_url(clang_url, clang_sha256), output = "", stripPrefix = clang_prefix, sha256 = clang_sha256, @@ -30,13 +33,14 @@ def _download_mac_m1_toolchain_impl(ctx): # Some std library headers use #include_next to include system specific headers, and # some skia source files require Xcode headers when compiling, (see SkTypes.h and look - # for TargetedConditionals.h)) All of these are located in Xcode, stopping the Mac + # for TargetedConditionals.h)) All of these are located in Xcode, stopping the Mac # builds from being purely hermetic. # For now, we can grab the user's Xcode path by calling xcode-select and create a symlink in # our toolchain directory to refer to during compilation. # https://developer.apple.com/library/archive/technotes/tn2339/_index.html res = ctx.execute(["xcode-select", "-p"]) + # https://bazel.build/rules/lib/actions#symlink ctx.symlink( # from = @@ -50,11 +54,14 @@ def _download_mac_m1_toolchain_impl(ctx): # and create a symlink looping error when resolving the filegroup. # available for use in rules, i.e. in the toolchain declaration. # https://bazel.build/rules/lib/repository_ctx#file + # TODO(kjlubick, jmbetancourt): Let's have compile-specific and link-specific rules, similar + # to https://skia-review.googlesource.com/c/skia/+/547822 and + # https://github.com/emscripten-core/emsdk/commit/93f21c9ef30bab52de24f9d4ea3f2f377cf6326a ctx.file( "BUILD.bazel", content = """ # DO NOT EDIT THIS BAZEL FILE DIRECTLY -# Generated from ctx.file action in download_mac_m1_toolchain.bzl +# Generated from ctx.file action in download_mac_toolchain.bzl filegroup( name = "all_files", srcs = glob( @@ -69,11 +76,10 @@ filegroup( ) # https://bazel.build/rules/repository_rules -download_mac_m1_toolchain = repository_rule( - implementation = _download_mac_m1_toolchain_impl, +download_mac_toolchain = repository_rule( + implementation = _download_mac_toolchain_impl, attrs = {}, - doc = "Downloads clang, and all supporting headers, executables, " + - "and shared libraries required to build Skia on a Mac M1 host." + + doc = "Downloads clang to build Skia with." + "Assumes you have xcode located on your device and have" + "xcode-select in your $PATH.", ) diff --git a/toolchain/download_toolchains.bzl b/toolchain/download_toolchains.bzl index 436ef89f23..bde81ed87b 100644 --- a/toolchain/download_toolchains.bzl +++ b/toolchain/download_toolchains.bzl @@ -3,7 +3,7 @@ This file exports the various toolchains for the hosts that we support building Supported: - Linux amd64 - - Mac M1 + - Mac (one toolchain for both M1 and Intel CPUs) Planned: - Windows amd64 @@ -11,14 +11,14 @@ Planned: """ load(":download_linux_amd64_toolchain.bzl", "download_linux_amd64_toolchain") -load(":download_mac_m1_toolchain.bzl", "download_mac_m1_toolchain") +load(":download_mac_toolchain.bzl", "download_mac_toolchain") # This key in this dictionary (and thus the name passed into the rule) controls what the subfolder # will be called in the external directory. It must match what we use in the appropriate # toolchain_config.bzl file or it will not be able to locate the sysroot to build with. name_toolchain = { "clang_linux_amd64": download_linux_amd64_toolchain, - "clang_mac_m1": download_mac_m1_toolchain, + "clang_mac": download_mac_toolchain, } def download_toolchains_for_skia(*args): diff --git a/toolchain/mac_m1_toolchain_config.bzl b/toolchain/mac_toolchain_config.bzl similarity index 97% rename from toolchain/mac_m1_toolchain_config.bzl rename to toolchain/mac_toolchain_config.bzl index 53a7b2574a..a2f2491d91 100644 --- a/toolchain/mac_m1_toolchain_config.bzl +++ b/toolchain/mac_toolchain_config.bzl @@ -1,14 +1,14 @@ """ -This file specifies a clang toolchain that can run on a Mac host. +This file specifies a clang toolchain that can run on a Mac host (with either M1 or Intel CPU). Hermetic toolchains still need access to Xcode for sys headers included in Skia's codebase. -See download_mac_m1_toolchain.bzl for more details on the creation of the toolchain. +See download_mac_toolchain.bzl for more details on the creation of the toolchain. It uses the usr subfolder of the built toolchain as a sysroot It follows the example of: - - lunix_amd64_toolchain_config.bzl + - linux_amd64_toolchain_config.bzl """ load( @@ -25,13 +25,13 @@ load( load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") # The location of the created clang toolchain. -EXTERNAL_TOOLCHAIN = "external/clang_mac_m1" +EXTERNAL_TOOLCHAIN = "external/clang_mac" # Symlink location. -# Must be the same as where the symlink points to in download_mac_m1_toolchain.bzl +# Must be the same as where the symlink points to in download_mac_toolchain.bzl XCODE_SYMLINK = EXTERNAL_TOOLCHAIN + "/symlinks/xcode/MacSDK/usr" -def _mac_m1_toolchain_info(ctx): +def _mac_toolchain_info(ctx): action_configs = _make_action_configs() features = [] features += _make_default_flags() @@ -56,10 +56,10 @@ def _mac_m1_toolchain_info(ctx): toolchain_identifier = "clang-toolchain", ) -provide_mac_m1_toolchain_config = rule( +provide_mac_toolchain_config = rule( attrs = {}, provides = [CcToolchainConfigInfo], - implementation = _mac_m1_toolchain_info, + implementation = _mac_toolchain_info, ) def _make_action_configs(): diff --git a/toolchain/mac_trampolines/ar_trampoline_mac.sh b/toolchain/mac_trampolines/ar_trampoline_mac.sh index 3ac75c5de8..57053c9e3b 100755 --- a/toolchain/mac_trampolines/ar_trampoline_mac.sh +++ b/toolchain/mac_trampolines/ar_trampoline_mac.sh @@ -3,5 +3,4 @@ # # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - -external/clang_mac_m1/bin/llvm-ar $@ +external/clang_mac/bin/llvm-ar $@ diff --git a/toolchain/mac_trampolines/clang_trampoline_mac.sh b/toolchain/mac_trampolines/clang_trampoline_mac.sh index c0b97f79b0..f77828d4b4 100755 --- a/toolchain/mac_trampolines/clang_trampoline_mac.sh +++ b/toolchain/mac_trampolines/clang_trampoline_mac.sh @@ -3,4 +3,4 @@ # # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. -external/clang_mac_m1/bin/clang $@ +external/clang_mac/bin/clang $@ diff --git a/toolchain/mac_trampolines/lld_trampoline_mac.sh b/toolchain/mac_trampolines/lld_trampoline_mac.sh index f9f07b7390..e84d607aaa 100755 --- a/toolchain/mac_trampolines/lld_trampoline_mac.sh +++ b/toolchain/mac_trampolines/lld_trampoline_mac.sh @@ -3,5 +3,4 @@ # # Use of this source code is governed by a BSD-style license that can be # found in the LICENSE file. - -external/clang_mac_m1/bin/ld.lld $@ +external/clang_mac/bin/ld.lld $@ diff --git a/toolchain/utils.bzl b/toolchain/utils.bzl new file mode 100644 index 0000000000..6d71150853 --- /dev/null +++ b/toolchain/utils.bzl @@ -0,0 +1,40 @@ +"""This module provides the gcs_mirror_url macro.""" + +# Set to True to force the macro to only return the mirror URL. +_TEST_GCS_MIRROR = False + +# Must be kept in sync with the suffixes supported by gcs_mirror (e.g. +# https://skia.googlesource.com/skia/+/8ad66c2340713234df6b249e793415233337a103/bazel/gcs_mirror/gcs_mirror.go#140). +_SUPPORTED_SUFFIXES = [".tar.gz", ".tgz", ".tar.xz", ".deb", ".zip"] + +_GCS_MIRROR_PREFIX = "https://storage.googleapis.com/skia-world-readable/bazel" + +def gcs_mirror_url(url, sha256): + """Takes the URL of an external resource and computes its GCS mirror URL. + + We store backup copies of external resources in the skia-world-readable GCS bucket. This macro + returns a list with two elements: the original URL, and the mirrored URL. + + Files are expected to be in the mirror location named after their sha256 hash. The files should + still have their file extension, as some of the Starlark functions sniff the file extension + (e.g. download_and_extract). See //bazel/gcs_mirror for an automated way to update this mirror. + + To mirror a new URL, please use the `gcs_mirror` utility found at + https://skia.googlesource.com/skia/+/8ad66c2340713234df6b249e793415233337a103/bazel/gcs_mirror/gcs_mirror.go. + + Args: + url: URL of the mirrored resource. + sha256: SHA256 hash of the mirrored resource. + Returns: + A list of the form [original URL, mirror URL]. + """ + extension = "" + for suffix in _SUPPORTED_SUFFIXES: + if url.endswith(suffix): + extension = suffix + break + if extension == "": + fail("URL %s has an unsupported suffix." % url) + + mirror_url = "%s/%s.%s" % (_GCS_MIRROR_PREFIX, sha256, extension) + return [mirror_url] if _TEST_GCS_MIRROR else [url, mirror_url]