AuroraMiddleware/skia2
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
c53d283c33
skia2/toolchain/download_linux_amd64_toolchain.bzl

249 lines
11 KiB
Python
Raw Normal View History

[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
"""
[bazel] Sketch out changes for Mac toolchain Change-Id: Idae84d8d9538012e5cfb75a1a477dbc72a4da5bc Bug: skia:13125 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/526264 Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-04-01 12:29:11 +00:00
This file assembles a toolchain for an amd64 Linux host using the Clang Compiler and glibc.
It downloads the necessary headers, executables, and pre-compiled static/shared libraries to
the external subfolder of the Bazel cache (the same place third party deps are downloaded with
http_archive or similar functions in WORKSPACE.bazel). These will be able to be used via our
set up tools for building Skia on Mac semi hermetically Reviewer notes: PS1 and PS2 handle everything up to the linking stage of the build PS6 and PS7 are trivial renamings and rebases, diff between Base->PS5 for a cleaner review No-Try: true Change-Id: Ib21ce2e8839ecd4b4dd57280e82f56a98194e476 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/532765 Reviewed-by: Kevin Lubick <kjlubick@google.com> Commit-Queue: Jorge Betancourt <jmbetancourt@google.com>
2022-05-04 15:06:39 +00:00
custom c++ toolchain configuration (see //toolchain/linux_amd64_toolchain_config.bzl)
[bazel] Sketch out changes for Mac toolchain Change-Id: Idae84d8d9538012e5cfb75a1a477dbc72a4da5bc Bug: skia:13125 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/526264 Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-04-01 12:29:11 +00:00
Most files are downloaded as .deb files from packages.debian.org (with us acting as the dependency
resolver) and extracted to
[outputRoot (aka Bazel cache)]/[outputUserRoot]/[outputBase]/external/clang_linux_amd64
(See https://bazel.build/docs/output_directories#layout-diagram)
which will act as our sysroot.
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
"""
[bazel] Add support for Macs to make Linux RBE builds The big change here is having the C++ toolchain use Bazel platforms instead of the C++ specific flags/setup. In Bazel, platforms are a general purpose way to define things like os, cpu architecture, etc. We were not using platforms previously, because the best documentation at the time focused on the old ways. However, the old ways were clumsy/difficult when trying to manage cross-compilation, specifically when trying to have a Mac host trigger a build on our Linux RBE system targeting a Linux x64 system. Thus, rather than keep investing in the legacy system, this CL migrates us to using platforms where possible. Suggested background reading to better understand this CL: - https://bazel.build/concepts/platforms-intro - https://bazel.build/docs/platforms - https://bazel.build/docs/toolchains#registering-building-toolchains The hermetic toolchain itself is not changing in this CL (and likely does not need to), only how we tell Bazel about it (i.e. registering it) and how Bazel decides to use it (i.e. resolving toolchains). Here is my understanding of how platforms and toolchains interact (supported by some evidence from [1][2]) - Bazel needs to resolve platforms for the Host, Execution, and Target. - If not specified via flags, these are the machine from which Bazel is invoked, aka "@local_config_platform//:host". - With this CL, the Host could be a Mac laptop, the Execution platform is our Linux RBE pool, and the Target is "a Linux system with a x64 CPU" - To specify the Host, that is, describe to Bazel the capabilities of the system it is running on, one can set --host_platform [3] with a label pointing to a platform() containing the appropriate settings. Tip: have this platform inherit from @local_config_platform//:host so it can add to any of the constraint_settings and constraint_values that Bazel deduces automatically. - To specify the Target platform(s), that is, the system on which a final output resides and can execute, one can set the --platforms flag with a label referencing a platform(). - Bazel will then choose an execution platform to fulfill that request. Bazel will look through a list of available platforms, which can be augmented* with the --extra_execution_platforms. Platforms specified by this flag will be considered higher than the default platforms! - Having selected the appropriate platforms, Bazel now needs to select a toolchain to actually run the actions of the appropriate type. - Bazel looks through the list of available toolchains and finds one that "matches" the Execution and the Target platform. This means, the toolchain's exec_compatible_with is a strict subset of the Execution platform and the toolchain's target_compatible_with is a strict subset of the Target platform. To register toolchains* (i.e. add them to the resolution list), we use --extra_toolchains. Once Bazel finds a match, it stops looking. Using --toolchain_resolution_debug=".*" makes Bazel log how it is resolving these toolchains and what execution platform it picked. * We can also register execution platforms and toolchains in WORKSPACE.bazel [4], but the flags come with higher priority and that made resolution a bit tricky. Also, when we want to conditionally add them (e.g. --config=linux_rbe), we cannot remove them conditionally in the WORKSPACE.bazel file. The above resolution flow directly necessitated the changes in this CL. Example usage of the new configs and platforms: # Can be run on a x64 Linux host and uses the hermetic toolchain. bazel build //:skia_public # Can be run on Mac or Linux and uses the Linux RBE system along # with the hermetic toolchain to compile a binary for Linux x64. bazel build //:skia_public --config=linux_rbe --config=for_linux_x64 # Shorthand for above bazel build //:skia_public --config=for_linux_x64_with_rbe Notice we don't have to type out --config=clang_linux anymore! That was due to me reading the Bazel docs more carefully and realizing we can set options for *all* Bazel build commands. Current Limitations: - Targets which require a py_binary (e.g. Dawn's genrules) will not work on RBE when cross compiling because the python runtime we download is for the host machine, not the executor. This means //example:hello_world_dawn does not work on Mac when cross-compiling via linux_rbe. - Mac M1 linking not quite working with SkOpts settings. Probably need to set -target [5] Suggested Review order: - toolchain/BUILD.bazel Notice how we do away with cc_toolchain_suite for toolchain. These have the same role: giving Bazel the information about where a toolchain can run. The platforms one is more expressive (IMO), allowing us to say both where to run the toolchain and what it can make. In order to more easily force the use of our hermetic toolchain, but also allow the hermetic toolchain to be used on RBE, we specify "use_hermetic_toolchain" only on the target, because the RBE image does not have the hermetic toolchain on it by default (but can certainly run it). - bazel/platform/BUILD.bazel to see the custom constraint_setting and corresponding constraint_value. The names for both of these are completely arbitrary - they do not need to have any deeper meaning or relation to any file or Docker image or system or any other constraints. Think of the constraint_setting as an Enum and the constraint_value being the one and only member. We need to pass around a constant value, not a type, so we need to provide the constraint_value (e.g. in toolchain/BUILD.bazel) but not a constraint_setting. However we need a constraint_setting declared so we can make a constraint_value of that "type". Notice the platform declared here - it allows us to force Bazel to use the hermetic toolchain because of the extra constraint_value. - .bazelrc I set a few flags that will be on for all bazel build commands. Importantly, this causes the C++ build logic to use platforms and not the old, bespoke way. I also found a way to avoid using the local toolchain on the host, which will hopefully lead to clearer errors if platforms are mis-specified instead of odd compile errors because the host toolchain is too old or something. There are also a few RBE settings tweaked to be a bit more modern, as well the new shorthands for specifying target platforms (e.g. for_linux_x64). - bazel/buildrc where we have to turn off the platforms logic for emscripten https://github.com/emscripten-core/emsdk/issues/984 - bazel/rbe/BUILD.bazel for a fix in the platform description that makes it work on Mac. - Notice that _m1 has been removed from the mac-related toolchain files because the same toolchain should work on both architectures. - All other changes in any order. [1] https://bazel.build/docs/toolchains#debugging-toolchains [2] https://bazel.build/docs/toolchains#toolchain-resolution [3] https://bazel.build/reference/command-line-reference [4] https://bazel.build/docs/toolchains#registering-building-toolchains [5] https://github.com/google/skia/blob/17dc3f16fc78477503ba1ef484c3b47bc3aab893/gn/skia/BUILD.gn#L258-L271 Change-Id: I515c114099d659639a808f74e47d489a68b7af62 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549737 Reviewed-by: Erik Rose <erikrose@google.com> Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-06-22 19:28:10 +00:00
load("//toolchain:utils.bzl", "gcs_mirror_url")
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
# From https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz.sha256
clang_prefix = "clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/"
clang_sha256 = "2c2fb857af97f41a5032e9ecadf7f78d3eff389a5cd3c9ec620d24f134ceb3c8"
clang_url = "https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz"
debs_to_install = [
# These three comprise glibc. libc6 has the shared libraries, like libc itself, the math library
# (libm), etc. linux-libc-dev has the header files specific to linux. libc6-dev has the libc
# system headers (e.g. malloc.h, math.h).
{
# From https://packages.debian.org/bullseye/amd64/libc6/download
"sha256": "3d9421c3fc0ef0d8ce57c0a149e1f8dbad78aba067f120be9e652af28902e346",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/g/glibc/libc6_2.31-13+deb11u2_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
{
# From https://packages.debian.org/bullseye/amd64/linux-libc-dev/download
"sha256": "1bb053863873916cb8d5fa877cc4972a6279931783c1fd9e4339d0369a617af4",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/l/linux/linux-libc-dev_5.10.84-1_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
{
# From https://packages.debian.org/bullseye/amd64/libc6-dev/download
"sha256": "1911bac1137f8f51359047d2fc94053f831abcfb50f1d7584e3ae95ea0831569",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/g/glibc/libc6-dev_2.31-13+deb11u2_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
# These two put the X11 include files in ${PWD}/usr/include/X11
# libx11-dev puts libX11.a in ${PWD}/usr/lib/x86_64-linux-gnu
{
# From https://packages.debian.org/bullseye/amd64/libx11-dev/download
"sha256": "11e5f9dcded1a1226b3ee02847b86edce525240367b3989274a891a43dc49f5f",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libx/libx11/libx11-dev_1.7.2-1_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
{
# From https://packages.debian.org/bullseye/all/x11proto-dev/download
"sha256": "d5568d587d9ad2664c34c14b0ac538ccb3c567e126ee5291085a8de704a565f5",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/x/xorgproto/x11proto-dev_2020.1-1_all.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
# xcb is a dep of X11
{
# From https://packages.debian.org/bullseye/amd64/libxcb1-dev/download
"sha256": "b75544f334c8963b8b7b0e8a88f8a7cde95a714dddbcda076d4beb669a961b58",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libx/libxcb/libxcb1-dev_1.14-3_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
# Xau is a dep of xcb
{
# From https://packages.debian.org/bullseye/amd64/libxau-dev/download
"sha256": "d1a7f5d484e0879b3b2e8d512894744505e53d078712ce65903fef2ecfd824bb",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libx/libxau/libxau-dev_1.0.9-1_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
# Xdmcp is a dep of xcb. libxdmcp-dev provides the the libXdmcp.so symlink (and the
# .a if we want to statically include it). libxdmcp6 actually provides the .so file
{
# https://packages.debian.org/bullseye/amd64/libxdmcp-dev/download
"sha256": "c6733e5f6463afd261998e408be6eb37f24ce0a64b63bed50a87ddb18ebc1699",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libx/libxdmcp/libxdmcp-dev_1.1.2-3_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
{
# https://packages.debian.org/bullseye/amd64/libxdmcp6/download
"sha256": "ecb8536f5fb34543b55bb9dc5f5b14c9dbb4150a7bddb3f2287b7cab6e9d25ef",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libx/libxdmcp/libxdmcp6_1.1.2-3_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
# These two put GL include files in ${PWD}/usr/include/GL
{
# From https://packages.debian.org/bullseye/amd64/libgl-dev/download
"sha256": "a6487873f2706bbabf9346cdb190f47f23a1464f31cecf92c363bac37c342f2f",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libg/libglvnd/libgl-dev_1.3.2-1_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
{
# From https://packages.debian.org/bullseye/amd64/libglx-dev/download
"sha256": "5a50549948bc4363eab32b1083dad2165402c3628f2ee85e9a32563228cc61c1",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libg/libglvnd/libglx-dev_1.3.2-1_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
# This provides libGL.so for us to link against.
{
# From https://packages.debian.org/bullseye/amd64/libgl1/download
"sha256": "f300f9610b5f05f1ce566c4095f1bf2170e512ac5d201c40d895b8fce29dec98",
[bazel] Add mirrors to every external dependency. - We always download from https - All external assets have a primary and a mirror URL. - We prioritize the sources as follows: bazel mirror, github/original source, our mirror - There is a way (see build_toolchain) to test the sources from the mirrors (done before CL submission). This adds a utility to upload files to the mirror in a consistent, scripted way. It includes a way to copy in parts of our bazel files (e.g. debs_to_install from toolchain/build_toolchain.bzl) to update many things at once. Our Bazel mirror (gs://skia-world-readable/bazel) is a Content Addressable Storage system, where the file name is based on the sha256sum of the contents (the same hash that Bazel uses). All files in it should be publicly accessible. Change-Id: Ida8b8e07d27a0a557bc49467ebbc86c806cabbd3 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/494478 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-01-13 15:19:38 +00:00
"url": "https://ftp.debian.org/debian/pool/main/libg/libglvnd/libgl1_1.3.2-1_amd64.deb",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
},
[bazel] Support Vulkan PS1 regenerates BUILD.bazel files I suggest reviewing the deltas between PS1 and the latest PS to focus on the interesting bits. The changes here allow for a Vulkan-only build of HelloWorld based on sk_app. The toughest change was properly fetching the VisualID after removing the gl calls that used to fill that in. There are a few changes that fix resolution of Dawn header files, but those won't actually be built until a follow-on CL. Change-Id: I54fb58b5dd7ecd4313562aed401759b3eaed53c0 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/516999 Reviewed-by: Greg Daniel <egdaniel@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-03-08 20:27:12 +00:00
# This is used by sk_app for Vulkan and Dawn on Unix.
{
# From https://packages.debian.org/bullseye/amd64/libx11-xcb-dev/download
"sha256": "80a2413ace2a0a073f2472059b9e589737cbf8a336fb6862684a5811bf640aa3",
"url": "https://ftp.debian.org/debian/pool/main/libx/libx11/libx11-xcb-dev_1.7.2-1_amd64.deb",
},
[bazel] Use toolchain features to opt files into being IWYU compliant. PS1 regenerates the Bazel files. It is recommended to review this CL with a diff from PS1. Example output when a file does not pass the test: tools/sk_app/CommandSet.h should add these lines: #include "include/core/SkTypes.h" #include "include/private/SkTArray.h" #include "tools/skui/InputState.h" #include "tools/skui/Key.h" #include "tools/skui/ModifierKey.h" namespace sk_app { class Window; } tools/sk_app/CommandSet.h should remove these lines: - #include "tools/sk_app/Window.h" The full include-list for tools/sk_app/CommandSet.h: #include "include/core/SkString.h" #include "include/core/SkTypes.h" #include "include/private/SkTArray.h" #include "tools/skui/InputState.h" #include "tools/skui/Key.h" #include "tools/skui/ModifierKey.h" #include <functional> #include <vector> class SkCanvas; namespace sk_app { class Window; } --- This makes use of Bazel's toolchain features https://bazel.build/docs/cc-toolchain-config-reference#features to allow us to configure compiler flags when compiling individual files. This analysis is off by default, and can be turned on with --features skia_enforce_iwyu. When enabled, it will only be run for files that have opted in. Example: bazelisk build //example:hello_world_gl --config=clang \ --sandbox_base=/dev/shm --features skia_enforce_iwyu There are two ways to opt files in: - Add enforce_iwyu = True to a generated_cc_atom rule - Add enforce_iwyu_on_package() to a BUILD.bazel file (which enforces IWYU for all rules in that file) Note that Bazel does not propagate features to dependencies or dependents, so trying to enable the feature on cc_library or cc_executable targets will only impact any files listed in srcs or hdrs, not deps. This may be counter-intuitive when compared to things like defines. IWYU supports a mapping file, which we supply to help properly handle things system headers (//toolchain/IWYU_mapping.imp) Suggested Review Order: - toolchain/build_toolchain.bzl to see how we get the IWYU binaries into the toolchain - toolchain/BUILD.bazel and toolchain/IWYU_mapping.imp to see how the mapping file is made available for all compile steps - toolchain/clang_toolchain_config.bzl, where we define the skia_enforce_iwyu feature to turn on any verification at all and skia_opt_file_into_iwyu to enable the check for specific files using a define. - toolchain/clang_trampoline.sh, which is the toolchain is configured to call instead of clang directly (see line 83 of clang_toolchain_config.bzl). This bash script used to just forward all arguments directly onto clang. Now it inspects them and either calls clang directly (if it does not find the define in the arguments or we are linking [bazel sometimes links with clang instead of ld]) or calls clang and then include-what-you-use. In all cases, the trampoline sends the arguments to clang and IWYU unchanged). - //tools/sk_app/... to see enforcement enabled (and fixed) for select files, as an example of that method. - //experimental/bazel_test/... to see enforcement enabled for all rules in a BUILD.bazel file. - all other files. Change-Id: I60a2ea9d5dc9955b6a8f166bd449de9e2b81a233 Bug: skia:13052 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/519776 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-03-16 12:33:44 +00:00
# This is used to make sure we include only the headers we need. This corresponds to
# IWYU version 0.17, which uses Clang 13, like we compile with.
{
# From https://packages.debian.org/sid/amd64/iwyu/download
"sha256": "9fd6932a7609e89364f7edc5f9613892c98c21c88a3931e51cf1a0f8744759bd",
"url": "https://ftp.debian.org/debian/pool/main/i/iwyu/iwyu_8.17-1_amd64.deb",
},
[bazel] Add RBE support using hermetic Linux Clang toolchain A new RBE worker-pool called gce_linux was created in conjunction with this CL. See https://docs.google.com/document/d/14xMZCKews69SSTfULhE8HDUzT5XvPwZ4CvRufEvcZ74/edit# for some details on that. Note: everything under bazel/rbe/gce_linux was autogenerated and can be ignored from manual review. It basically specifies what files are on the RBE image that are necessary for running Bazel. Testing it out can be done by authenticating for RBE gcloud auth application-default login --no-browser Then, run make -C bazel rbe_known_good_builds to test it out. On my 4 core laptop with an empty local cache, but a warm remote cache, the build took <2 min instead of the 10+ minutes it would have [1]. The folder structure in //bazel/rbe is meant to let us have multiple remote configurations there, e.g. //bazel/rbe/gce_windows. Suggested Review Order: - bazel/rbe/README.md - bazel/rbe/gce_linux_container/Dockerfile to see the bare-bones RBE image. - bazel/rbe/BUILD.bazel to see a custom platform defined. It is nearly identical to the autogenerated one in bazel/rbe/gce_linux/config/BUILD, with one extra field to force the gce_linux pool to be used. - .bazelrc to see the settings needed to make --config=linux-rbe work. The naming convention was inspired by SkCMS's setup [2], and allows us to have some common RBE settings (i.e. config:remote) and some specialized ones for the given host machine (e.g. config:linux-rbe) A very important, but subtle configuration, is on line 86 of .bazelrc where we say to use our hermetic toolchain and not whatever C++ compiler and headers are on the host machine (aka the RBE container). - toolchain/build_toolchain.bzl to see some additional dependencies needed in the toolchain (to run IWYU) which I had installed locally but didn't realize were important. - third_party/BUILD.bazel to see an example of how failing to specify all files can result in something that works locally, but fails remotely. --execution_log_json_file=/tmp/execlog.json helped debug these issues. - All other files. [1] http://go/scrcast/NjM1ODE4MDI0NzM3MTc3Nnw3ODViZmFkMi1iOA [2] https://skia.googlesource.com/skcms/+/30c8e303800c256febb03a09fdcda7f75d119b1b/.bazelrc#20 Change-Id: Ia0a9e6a06c1a13071949ab402dc5d897df6b12e1 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524359 Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2022-03-25 18:59:33 +00:00
{
# This is a requirement of iwyu
# https://packages.debian.org/sid/amd64/libclang-cpp13/download
"sha256": "c6e2471de8f3ec06e40c8e006e06bbd251dd0c8000dee820a4b6dca3d3290c0d",
"url": "https://ftp.debian.org/debian/pool/main/l/llvm-toolchain-13/libclang-cpp13_13.0.1-3+b1_amd64.deb",
},
{
# This is a requirement of libclang-cpp13
# https://packages.debian.org/sid/amd64/libstdc++6/download
"sha256": "f37e5954423955938c5309a8d0e475f7e84e92b56b8301487fb885192dee8085",
"url": "https://ftp.debian.org/debian/pool/main/g/gcc-12/libstdc++6_12-20220319-1_amd64.deb",
},
{
# This is a requirement of iwyu
# https://packages.debian.org/sid/amd64/libllvm13/download
"sha256": "49f29a6c9fbc3097077931529e7fe1c032b1d04a984d971aa1e6990a5133556e",
"url": "https://ftp.debian.org/debian/pool/main/l/llvm-toolchain-13/libllvm13_13.0.1-3+b1_amd64.deb",
},
{
# This is a requirement of libllvm13
# https://packages.debian.org/sid/amd64/libffi8/download
"sha256": "87c55b36951aed18ef2c357683e15c365713bda6090f15386998b57df433b387",
"url": "https://ftp.debian.org/debian/pool/main/libf/libffi/libffi8_3.4.2-4_amd64.deb",
},
{
# This is a requirement of libllvm13
# https://packages.debian.org/sid/libz3-4
"sha256": "b415b863678625dee3f3c75bd48b1b9e3b6e11279ebec337904d7f09630d107f",
"url": "https://ftp.debian.org/debian/pool/main/z/z3/libz3-4_4.8.12-1+b1_amd64.deb",
},
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
]
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
def _download_and_extract_deb(ctx, deb, sha256, prefix, output = ""):
"""Downloads a debian file and extracts the data into the provided output directory"""
Reland "[bazel] Update to v5.2.0" This is a reland of commit fb13c18ddf7df9b6d64315fac6bac840f3289fe7 Bazel 5.2.0 was fine, it was the fact that our toolchain was not erroring when clang failed that was an issue. We were always returning error code 0, but Bazel 5.2.0 was correctly identifying that we hadn't produced the promised build artifacts. Original change's description: > [bazel] Update to v5.2.0 > > The primary change we are looking for is > "Add support for .ar archives (and .deb files) > https://github.com/bazelbuild/bazel/pull/15218" > > https://blog.bazel.build/2022/06/08/bazel-5.2.html > > In theory, we should be able to trigger Linux RBE > compilations (e.g. IWYU) from Mac with this change. > > Change-Id: I217406d21fd55507e090c4bb5f79c796230717e6 > Bug: skia:12541 > Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549659 > Reviewed-by: Joe Gregorio <jcgregorio@google.com> Bug: skia:12541 Change-Id: I160b5802adc232d5cf7f7d05b20d5eabbb3d5102 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/551841 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-06-21 18:36:28 +00:00
# https://bazel.build/rules/lib/repository_ctx#download_and_extract
# A .deb file has a data.tar.xz and a control.tar.xz, but the important contents
# (i.e. the headers or libs) are in the data.tar.xz
ctx.download_and_extract(
[bazel] Add support for Macs to make Linux RBE builds The big change here is having the C++ toolchain use Bazel platforms instead of the C++ specific flags/setup. In Bazel, platforms are a general purpose way to define things like os, cpu architecture, etc. We were not using platforms previously, because the best documentation at the time focused on the old ways. However, the old ways were clumsy/difficult when trying to manage cross-compilation, specifically when trying to have a Mac host trigger a build on our Linux RBE system targeting a Linux x64 system. Thus, rather than keep investing in the legacy system, this CL migrates us to using platforms where possible. Suggested background reading to better understand this CL: - https://bazel.build/concepts/platforms-intro - https://bazel.build/docs/platforms - https://bazel.build/docs/toolchains#registering-building-toolchains The hermetic toolchain itself is not changing in this CL (and likely does not need to), only how we tell Bazel about it (i.e. registering it) and how Bazel decides to use it (i.e. resolving toolchains). Here is my understanding of how platforms and toolchains interact (supported by some evidence from [1][2]) - Bazel needs to resolve platforms for the Host, Execution, and Target. - If not specified via flags, these are the machine from which Bazel is invoked, aka "@local_config_platform//:host". - With this CL, the Host could be a Mac laptop, the Execution platform is our Linux RBE pool, and the Target is "a Linux system with a x64 CPU" - To specify the Host, that is, describe to Bazel the capabilities of the system it is running on, one can set --host_platform [3] with a label pointing to a platform() containing the appropriate settings. Tip: have this platform inherit from @local_config_platform//:host so it can add to any of the constraint_settings and constraint_values that Bazel deduces automatically. - To specify the Target platform(s), that is, the system on which a final output resides and can execute, one can set the --platforms flag with a label referencing a platform(). - Bazel will then choose an execution platform to fulfill that request. Bazel will look through a list of available platforms, which can be augmented* with the --extra_execution_platforms. Platforms specified by this flag will be considered higher than the default platforms! - Having selected the appropriate platforms, Bazel now needs to select a toolchain to actually run the actions of the appropriate type. - Bazel looks through the list of available toolchains and finds one that "matches" the Execution and the Target platform. This means, the toolchain's exec_compatible_with is a strict subset of the Execution platform and the toolchain's target_compatible_with is a strict subset of the Target platform. To register toolchains* (i.e. add them to the resolution list), we use --extra_toolchains. Once Bazel finds a match, it stops looking. Using --toolchain_resolution_debug=".*" makes Bazel log how it is resolving these toolchains and what execution platform it picked. * We can also register execution platforms and toolchains in WORKSPACE.bazel [4], but the flags come with higher priority and that made resolution a bit tricky. Also, when we want to conditionally add them (e.g. --config=linux_rbe), we cannot remove them conditionally in the WORKSPACE.bazel file. The above resolution flow directly necessitated the changes in this CL. Example usage of the new configs and platforms: # Can be run on a x64 Linux host and uses the hermetic toolchain. bazel build //:skia_public # Can be run on Mac or Linux and uses the Linux RBE system along # with the hermetic toolchain to compile a binary for Linux x64. bazel build //:skia_public --config=linux_rbe --config=for_linux_x64 # Shorthand for above bazel build //:skia_public --config=for_linux_x64_with_rbe Notice we don't have to type out --config=clang_linux anymore! That was due to me reading the Bazel docs more carefully and realizing we can set options for *all* Bazel build commands. Current Limitations: - Targets which require a py_binary (e.g. Dawn's genrules) will not work on RBE when cross compiling because the python runtime we download is for the host machine, not the executor. This means //example:hello_world_dawn does not work on Mac when cross-compiling via linux_rbe. - Mac M1 linking not quite working with SkOpts settings. Probably need to set -target [5] Suggested Review order: - toolchain/BUILD.bazel Notice how we do away with cc_toolchain_suite for toolchain. These have the same role: giving Bazel the information about where a toolchain can run. The platforms one is more expressive (IMO), allowing us to say both where to run the toolchain and what it can make. In order to more easily force the use of our hermetic toolchain, but also allow the hermetic toolchain to be used on RBE, we specify "use_hermetic_toolchain" only on the target, because the RBE image does not have the hermetic toolchain on it by default (but can certainly run it). - bazel/platform/BUILD.bazel to see the custom constraint_setting and corresponding constraint_value. The names for both of these are completely arbitrary - they do not need to have any deeper meaning or relation to any file or Docker image or system or any other constraints. Think of the constraint_setting as an Enum and the constraint_value being the one and only member. We need to pass around a constant value, not a type, so we need to provide the constraint_value (e.g. in toolchain/BUILD.bazel) but not a constraint_setting. However we need a constraint_setting declared so we can make a constraint_value of that "type". Notice the platform declared here - it allows us to force Bazel to use the hermetic toolchain because of the extra constraint_value. - .bazelrc I set a few flags that will be on for all bazel build commands. Importantly, this causes the C++ build logic to use platforms and not the old, bespoke way. I also found a way to avoid using the local toolchain on the host, which will hopefully lead to clearer errors if platforms are mis-specified instead of odd compile errors because the host toolchain is too old or something. There are also a few RBE settings tweaked to be a bit more modern, as well the new shorthands for specifying target platforms (e.g. for_linux_x64). - bazel/buildrc where we have to turn off the platforms logic for emscripten https://github.com/emscripten-core/emsdk/issues/984 - bazel/rbe/BUILD.bazel for a fix in the platform description that makes it work on Mac. - Notice that _m1 has been removed from the mac-related toolchain files because the same toolchain should work on both architectures. - All other changes in any order. [1] https://bazel.build/docs/toolchains#debugging-toolchains [2] https://bazel.build/docs/toolchains#toolchain-resolution [3] https://bazel.build/reference/command-line-reference [4] https://bazel.build/docs/toolchains#registering-building-toolchains [5] https://github.com/google/skia/blob/17dc3f16fc78477503ba1ef484c3b47bc3aab893/gn/skia/BUILD.gn#L258-L271 Change-Id: I515c114099d659639a808f74e47d489a68b7af62 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549737 Reviewed-by: Erik Rose <erikrose@google.com> Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-06-22 19:28:10 +00:00
url = gcs_mirror_url(deb, sha256),
Reland "[bazel] Update to v5.2.0" This is a reland of commit fb13c18ddf7df9b6d64315fac6bac840f3289fe7 Bazel 5.2.0 was fine, it was the fact that our toolchain was not erroring when clang failed that was an issue. We were always returning error code 0, but Bazel 5.2.0 was correctly identifying that we hadn't produced the promised build artifacts. Original change's description: > [bazel] Update to v5.2.0 > > The primary change we are looking for is > "Add support for .ar archives (and .deb files) > https://github.com/bazelbuild/bazel/pull/15218" > > https://blog.bazel.build/2022/06/08/bazel-5.2.html > > In theory, we should be able to trigger Linux RBE > compilations (e.g. IWYU) from Mac with this change. > > Change-Id: I217406d21fd55507e090c4bb5f79c796230717e6 > Bug: skia:12541 > Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549659 > Reviewed-by: Joe Gregorio <jcgregorio@google.com> Bug: skia:12541 Change-Id: I160b5802adc232d5cf7f7d05b20d5eabbb3d5102 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/551841 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-06-21 18:36:28 +00:00
output = "tmp",
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
sha256 = sha256,
)
[bazel] Add CI job that uses RBE to enforce includes The toolchain now uses extract_ar from https://skia-review.googlesource.com/c/buildbot/+/524764 which is a static executable to extract the .deb files. This was necessary because the llvm-ar that had previously been used requires glibc 2.31+ to run, but our Debian10 machines on Swarming have an older version (2.28). A longer-term fix is to have Bazel support .ar files, which I plan to attempt to contribute this week. The RBE task will be added as an experimental CQ job, to see how it handles the load of running often. With the remote execution cache, I hope it performs well, once the toolchains are cached on both the Swarming machines and in the RBE workers. Note: We had to add several files to the CAS spec (see compile_cas.go) which are required for Bazel to work. Change-Id: Ie70c70d5f33768c957760f9eeb7835025109b487 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524759 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-03-28 13:45:26 +00:00
# https://bazel.build/rules/lib/repository_ctx#extract
[bazel] Specify files required for the toolchain more precisely. This improves performance with sandboxing. One can have Bazel output performance data [1][2], which can be viewed via chrome://tracing or https://ui.perfetto.dev/. With Perfetto, the following SQL queries [3][4] can summarize the sandboxing metrics, as well as the actual compilation time. http://screen/5TxbeZTso4gNDfD Note that the dur column is in nanoseconds, so we convert to seconds. These numbers could further be divided by the number of processes (in my case 48) to get a scaled output. SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "sandbox.createFileSystem"; SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "sandbox.delete"; SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "subprocess.run"; I benchmarked the local compilation of //:skia_public using --config=clang_linux (our custom Linux toolchain). I was sure to clear the Bazel cache before each run and not count the time to download/update a toolchain for the first time. The control measurements (without this CL) are: Wall Time = 272.2s sandbox.createFileSystem = 5466.9s subprocess.run = 2961.0s sandbox.delete = 4112.3s With this CL: Wall Time = 53.9s (5.05x faster) sandbox.createFileSystem = 403.4s subprocess.run = 1610.3s sandbox.delete = 348.8s The control measurement is a touch misleading. Due to it being so slow, I had recommended developers use a ramdisk when building on machines with sufficient RAM via the Bazel flag --sandbox_base=/dev/shm Here is the control measurement when using a RAM disk: Wall Time = 21.2s sandbox.createFileSystem = 58.9s subprocess.run = 705.1s sandbox.delete = 46.6s With this CL and a RAM disk: Wall Time = 19.2s (10% faster) sandbox.createFileSystem = 21.8s subprocess.run = 722.9s sandbox.delete = 16.2s For devs who cannot or are not using a RAM disk, this is a huge win. With a RAM disk, it's a modest improvement. On an RBE build, this had minimal impact. I did my best to bust the action cache with a fake define and the before and after times were about the same. This was inspired by [5] and [6]. [1] Add --profile=/tmp/profile.gz to any command [2] https://bazel.build/rules/performance#performance-profiling [3] https://perfetto.dev/docs/quickstart/trace-analysis#sample-queries [4] https://perfetto.dev/docs/analysis/sql-tables#slice [5] https://github.com/emscripten-core/emsdk/commit/93f21c9ef30bab52de24f9d4ea3f2f377cf6326a [6] https://github.com/emscripten-core/emsdk/commit/311acff345fd71dcfe5f350653cec466ee7e3fbc Change-Id: Iceb2606e86111159141a286d01fc002d09fe3fe4 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/547822 Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2022-06-07 20:37:48 +00:00
ctx.extract(
[bazel] Add CI job that uses RBE to enforce includes The toolchain now uses extract_ar from https://skia-review.googlesource.com/c/buildbot/+/524764 which is a static executable to extract the .deb files. This was necessary because the llvm-ar that had previously been used requires glibc 2.31+ to run, but our Debian10 machines on Swarming have an older version (2.28). A longer-term fix is to have Bazel support .ar files, which I plan to attempt to contribute this week. The RBE task will be added as an experimental CQ job, to see how it handles the load of running often. With the remote execution cache, I hope it performs well, once the toolchains are cached on both the Swarming machines and in the RBE workers. Note: We had to add several files to the CAS spec (see compile_cas.go) which are required for Bazel to work. Change-Id: Ie70c70d5f33768c957760f9eeb7835025109b487 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524759 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-03-28 13:45:26 +00:00
archive = "tmp/data.tar.xz",
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
output = output,
stripPrefix = prefix,
)
# Clean up
[bazel] Add CI job that uses RBE to enforce includes The toolchain now uses extract_ar from https://skia-review.googlesource.com/c/buildbot/+/524764 which is a static executable to extract the .deb files. This was necessary because the llvm-ar that had previously been used requires glibc 2.31+ to run, but our Debian10 machines on Swarming have an older version (2.28). A longer-term fix is to have Bazel support .ar files, which I plan to attempt to contribute this week. The RBE task will be added as an experimental CQ job, to see how it handles the load of running often. With the remote execution cache, I hope it performs well, once the toolchains are cached on both the Swarming machines and in the RBE workers. Note: We had to add several files to the CAS spec (see compile_cas.go) which are required for Bazel to work. Change-Id: Ie70c70d5f33768c957760f9eeb7835025109b487 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524759 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-03-28 13:45:26 +00:00
ctx.delete("tmp")
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
[bazel] Sketch out changes for Mac toolchain Change-Id: Idae84d8d9538012e5cfb75a1a477dbc72a4da5bc Bug: skia:13125 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/526264 Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-04-01 12:29:11 +00:00
def _download_linux_amd64_toolchain_impl(ctx):
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
# Download the clang toolchain (the extraction can take a while)
[bazel] Add CI job that uses RBE to enforce includes The toolchain now uses extract_ar from https://skia-review.googlesource.com/c/buildbot/+/524764 which is a static executable to extract the .deb files. This was necessary because the llvm-ar that had previously been used requires glibc 2.31+ to run, but our Debian10 machines on Swarming have an older version (2.28). A longer-term fix is to have Bazel support .ar files, which I plan to attempt to contribute this week. The RBE task will be added as an experimental CQ job, to see how it handles the load of running often. With the remote execution cache, I hope it performs well, once the toolchains are cached on both the Swarming machines and in the RBE workers. Note: We had to add several files to the CAS spec (see compile_cas.go) which are required for Bazel to work. Change-Id: Ie70c70d5f33768c957760f9eeb7835025109b487 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524759 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-03-28 13:45:26 +00:00
# https://bazel.build/rules/lib/repository_ctx#download_and_extract
ctx.download_and_extract(
[bazel] Add support for Macs to make Linux RBE builds The big change here is having the C++ toolchain use Bazel platforms instead of the C++ specific flags/setup. In Bazel, platforms are a general purpose way to define things like os, cpu architecture, etc. We were not using platforms previously, because the best documentation at the time focused on the old ways. However, the old ways were clumsy/difficult when trying to manage cross-compilation, specifically when trying to have a Mac host trigger a build on our Linux RBE system targeting a Linux x64 system. Thus, rather than keep investing in the legacy system, this CL migrates us to using platforms where possible. Suggested background reading to better understand this CL: - https://bazel.build/concepts/platforms-intro - https://bazel.build/docs/platforms - https://bazel.build/docs/toolchains#registering-building-toolchains The hermetic toolchain itself is not changing in this CL (and likely does not need to), only how we tell Bazel about it (i.e. registering it) and how Bazel decides to use it (i.e. resolving toolchains). Here is my understanding of how platforms and toolchains interact (supported by some evidence from [1][2]) - Bazel needs to resolve platforms for the Host, Execution, and Target. - If not specified via flags, these are the machine from which Bazel is invoked, aka "@local_config_platform//:host". - With this CL, the Host could be a Mac laptop, the Execution platform is our Linux RBE pool, and the Target is "a Linux system with a x64 CPU" - To specify the Host, that is, describe to Bazel the capabilities of the system it is running on, one can set --host_platform [3] with a label pointing to a platform() containing the appropriate settings. Tip: have this platform inherit from @local_config_platform//:host so it can add to any of the constraint_settings and constraint_values that Bazel deduces automatically. - To specify the Target platform(s), that is, the system on which a final output resides and can execute, one can set the --platforms flag with a label referencing a platform(). - Bazel will then choose an execution platform to fulfill that request. Bazel will look through a list of available platforms, which can be augmented* with the --extra_execution_platforms. Platforms specified by this flag will be considered higher than the default platforms! - Having selected the appropriate platforms, Bazel now needs to select a toolchain to actually run the actions of the appropriate type. - Bazel looks through the list of available toolchains and finds one that "matches" the Execution and the Target platform. This means, the toolchain's exec_compatible_with is a strict subset of the Execution platform and the toolchain's target_compatible_with is a strict subset of the Target platform. To register toolchains* (i.e. add them to the resolution list), we use --extra_toolchains. Once Bazel finds a match, it stops looking. Using --toolchain_resolution_debug=".*" makes Bazel log how it is resolving these toolchains and what execution platform it picked. * We can also register execution platforms and toolchains in WORKSPACE.bazel [4], but the flags come with higher priority and that made resolution a bit tricky. Also, when we want to conditionally add them (e.g. --config=linux_rbe), we cannot remove them conditionally in the WORKSPACE.bazel file. The above resolution flow directly necessitated the changes in this CL. Example usage of the new configs and platforms: # Can be run on a x64 Linux host and uses the hermetic toolchain. bazel build //:skia_public # Can be run on Mac or Linux and uses the Linux RBE system along # with the hermetic toolchain to compile a binary for Linux x64. bazel build //:skia_public --config=linux_rbe --config=for_linux_x64 # Shorthand for above bazel build //:skia_public --config=for_linux_x64_with_rbe Notice we don't have to type out --config=clang_linux anymore! That was due to me reading the Bazel docs more carefully and realizing we can set options for *all* Bazel build commands. Current Limitations: - Targets which require a py_binary (e.g. Dawn's genrules) will not work on RBE when cross compiling because the python runtime we download is for the host machine, not the executor. This means //example:hello_world_dawn does not work on Mac when cross-compiling via linux_rbe. - Mac M1 linking not quite working with SkOpts settings. Probably need to set -target [5] Suggested Review order: - toolchain/BUILD.bazel Notice how we do away with cc_toolchain_suite for toolchain. These have the same role: giving Bazel the information about where a toolchain can run. The platforms one is more expressive (IMO), allowing us to say both where to run the toolchain and what it can make. In order to more easily force the use of our hermetic toolchain, but also allow the hermetic toolchain to be used on RBE, we specify "use_hermetic_toolchain" only on the target, because the RBE image does not have the hermetic toolchain on it by default (but can certainly run it). - bazel/platform/BUILD.bazel to see the custom constraint_setting and corresponding constraint_value. The names for both of these are completely arbitrary - they do not need to have any deeper meaning or relation to any file or Docker image or system or any other constraints. Think of the constraint_setting as an Enum and the constraint_value being the one and only member. We need to pass around a constant value, not a type, so we need to provide the constraint_value (e.g. in toolchain/BUILD.bazel) but not a constraint_setting. However we need a constraint_setting declared so we can make a constraint_value of that "type". Notice the platform declared here - it allows us to force Bazel to use the hermetic toolchain because of the extra constraint_value. - .bazelrc I set a few flags that will be on for all bazel build commands. Importantly, this causes the C++ build logic to use platforms and not the old, bespoke way. I also found a way to avoid using the local toolchain on the host, which will hopefully lead to clearer errors if platforms are mis-specified instead of odd compile errors because the host toolchain is too old or something. There are also a few RBE settings tweaked to be a bit more modern, as well the new shorthands for specifying target platforms (e.g. for_linux_x64). - bazel/buildrc where we have to turn off the platforms logic for emscripten https://github.com/emscripten-core/emsdk/issues/984 - bazel/rbe/BUILD.bazel for a fix in the platform description that makes it work on Mac. - Notice that _m1 has been removed from the mac-related toolchain files because the same toolchain should work on both architectures. - All other changes in any order. [1] https://bazel.build/docs/toolchains#debugging-toolchains [2] https://bazel.build/docs/toolchains#toolchain-resolution [3] https://bazel.build/reference/command-line-reference [4] https://bazel.build/docs/toolchains#registering-building-toolchains [5] https://github.com/google/skia/blob/17dc3f16fc78477503ba1ef484c3b47bc3aab893/gn/skia/BUILD.gn#L258-L271 Change-Id: I515c114099d659639a808f74e47d489a68b7af62 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549737 Reviewed-by: Erik Rose <erikrose@google.com> Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-06-22 19:28:10 +00:00
url = gcs_mirror_url(clang_url, clang_sha256),
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
output = "",
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
stripPrefix = clang_prefix,
sha256 = clang_sha256,
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
)
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
# Extract all the debs into our sysroot. This is very similar to installing them, except their
# dependencies are not installed automatically.
for deb in debs_to_install:
_download_and_extract_deb(
ctx,
deb["url"],
deb["sha256"],
".",
)
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
[bazel] Specify files required for the toolchain more precisely. This improves performance with sandboxing. One can have Bazel output performance data [1][2], which can be viewed via chrome://tracing or https://ui.perfetto.dev/. With Perfetto, the following SQL queries [3][4] can summarize the sandboxing metrics, as well as the actual compilation time. http://screen/5TxbeZTso4gNDfD Note that the dur column is in nanoseconds, so we convert to seconds. These numbers could further be divided by the number of processes (in my case 48) to get a scaled output. SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "sandbox.createFileSystem"; SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "sandbox.delete"; SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "subprocess.run"; I benchmarked the local compilation of //:skia_public using --config=clang_linux (our custom Linux toolchain). I was sure to clear the Bazel cache before each run and not count the time to download/update a toolchain for the first time. The control measurements (without this CL) are: Wall Time = 272.2s sandbox.createFileSystem = 5466.9s subprocess.run = 2961.0s sandbox.delete = 4112.3s With this CL: Wall Time = 53.9s (5.05x faster) sandbox.createFileSystem = 403.4s subprocess.run = 1610.3s sandbox.delete = 348.8s The control measurement is a touch misleading. Due to it being so slow, I had recommended developers use a ramdisk when building on machines with sufficient RAM via the Bazel flag --sandbox_base=/dev/shm Here is the control measurement when using a RAM disk: Wall Time = 21.2s sandbox.createFileSystem = 58.9s subprocess.run = 705.1s sandbox.delete = 46.6s With this CL and a RAM disk: Wall Time = 19.2s (10% faster) sandbox.createFileSystem = 21.8s subprocess.run = 722.9s sandbox.delete = 16.2s For devs who cannot or are not using a RAM disk, this is a huge win. With a RAM disk, it's a modest improvement. On an RBE build, this had minimal impact. I did my best to bust the action cache with a fake define and the before and after times were about the same. This was inspired by [5] and [6]. [1] Add --profile=/tmp/profile.gz to any command [2] https://bazel.build/rules/performance#performance-profiling [3] https://perfetto.dev/docs/quickstart/trace-analysis#sample-queries [4] https://perfetto.dev/docs/analysis/sql-tables#slice [5] https://github.com/emscripten-core/emsdk/commit/93f21c9ef30bab52de24f9d4ea3f2f377cf6326a [6] https://github.com/emscripten-core/emsdk/commit/311acff345fd71dcfe5f350653cec466ee7e3fbc Change-Id: Iceb2606e86111159141a286d01fc002d09fe3fe4 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/547822 Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2022-06-07 20:37:48 +00:00
# Create a BUILD.bazel file that makes the files downloaded into the toolchain visible.
# We have separate groups for each task because doing less work (sandboxing fewer files
# or uploading less data to RBE) makes compiles go faster. We try to strike a balance
# between minimal specifications and not having to edit this file often with our use
# of globs.
[bazel] Add CI job that uses RBE to enforce includes The toolchain now uses extract_ar from https://skia-review.googlesource.com/c/buildbot/+/524764 which is a static executable to extract the .deb files. This was necessary because the llvm-ar that had previously been used requires glibc 2.31+ to run, but our Debian10 machines on Swarming have an older version (2.28). A longer-term fix is to have Bazel support .ar files, which I plan to attempt to contribute this week. The RBE task will be added as an experimental CQ job, to see how it handles the load of running often. With the remote execution cache, I hope it performs well, once the toolchains are cached on both the Swarming machines and in the RBE workers. Note: We had to add several files to the CAS spec (see compile_cas.go) which are required for Bazel to work. Change-Id: Ie70c70d5f33768c957760f9eeb7835025109b487 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524759 Reviewed-by: Joe Gregorio <jcgregorio@google.com>
2022-03-28 13:45:26 +00:00
# https://bazel.build/rules/lib/repository_ctx#file
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
ctx.file(
"BUILD.bazel",
content = """
[bazel] Add support for Macs to make Linux RBE builds The big change here is having the C++ toolchain use Bazel platforms instead of the C++ specific flags/setup. In Bazel, platforms are a general purpose way to define things like os, cpu architecture, etc. We were not using platforms previously, because the best documentation at the time focused on the old ways. However, the old ways were clumsy/difficult when trying to manage cross-compilation, specifically when trying to have a Mac host trigger a build on our Linux RBE system targeting a Linux x64 system. Thus, rather than keep investing in the legacy system, this CL migrates us to using platforms where possible. Suggested background reading to better understand this CL: - https://bazel.build/concepts/platforms-intro - https://bazel.build/docs/platforms - https://bazel.build/docs/toolchains#registering-building-toolchains The hermetic toolchain itself is not changing in this CL (and likely does not need to), only how we tell Bazel about it (i.e. registering it) and how Bazel decides to use it (i.e. resolving toolchains). Here is my understanding of how platforms and toolchains interact (supported by some evidence from [1][2]) - Bazel needs to resolve platforms for the Host, Execution, and Target. - If not specified via flags, these are the machine from which Bazel is invoked, aka "@local_config_platform//:host". - With this CL, the Host could be a Mac laptop, the Execution platform is our Linux RBE pool, and the Target is "a Linux system with a x64 CPU" - To specify the Host, that is, describe to Bazel the capabilities of the system it is running on, one can set --host_platform [3] with a label pointing to a platform() containing the appropriate settings. Tip: have this platform inherit from @local_config_platform//:host so it can add to any of the constraint_settings and constraint_values that Bazel deduces automatically. - To specify the Target platform(s), that is, the system on which a final output resides and can execute, one can set the --platforms flag with a label referencing a platform(). - Bazel will then choose an execution platform to fulfill that request. Bazel will look through a list of available platforms, which can be augmented* with the --extra_execution_platforms. Platforms specified by this flag will be considered higher than the default platforms! - Having selected the appropriate platforms, Bazel now needs to select a toolchain to actually run the actions of the appropriate type. - Bazel looks through the list of available toolchains and finds one that "matches" the Execution and the Target platform. This means, the toolchain's exec_compatible_with is a strict subset of the Execution platform and the toolchain's target_compatible_with is a strict subset of the Target platform. To register toolchains* (i.e. add them to the resolution list), we use --extra_toolchains. Once Bazel finds a match, it stops looking. Using --toolchain_resolution_debug=".*" makes Bazel log how it is resolving these toolchains and what execution platform it picked. * We can also register execution platforms and toolchains in WORKSPACE.bazel [4], but the flags come with higher priority and that made resolution a bit tricky. Also, when we want to conditionally add them (e.g. --config=linux_rbe), we cannot remove them conditionally in the WORKSPACE.bazel file. The above resolution flow directly necessitated the changes in this CL. Example usage of the new configs and platforms: # Can be run on a x64 Linux host and uses the hermetic toolchain. bazel build //:skia_public # Can be run on Mac or Linux and uses the Linux RBE system along # with the hermetic toolchain to compile a binary for Linux x64. bazel build //:skia_public --config=linux_rbe --config=for_linux_x64 # Shorthand for above bazel build //:skia_public --config=for_linux_x64_with_rbe Notice we don't have to type out --config=clang_linux anymore! That was due to me reading the Bazel docs more carefully and realizing we can set options for *all* Bazel build commands. Current Limitations: - Targets which require a py_binary (e.g. Dawn's genrules) will not work on RBE when cross compiling because the python runtime we download is for the host machine, not the executor. This means //example:hello_world_dawn does not work on Mac when cross-compiling via linux_rbe. - Mac M1 linking not quite working with SkOpts settings. Probably need to set -target [5] Suggested Review order: - toolchain/BUILD.bazel Notice how we do away with cc_toolchain_suite for toolchain. These have the same role: giving Bazel the information about where a toolchain can run. The platforms one is more expressive (IMO), allowing us to say both where to run the toolchain and what it can make. In order to more easily force the use of our hermetic toolchain, but also allow the hermetic toolchain to be used on RBE, we specify "use_hermetic_toolchain" only on the target, because the RBE image does not have the hermetic toolchain on it by default (but can certainly run it). - bazel/platform/BUILD.bazel to see the custom constraint_setting and corresponding constraint_value. The names for both of these are completely arbitrary - they do not need to have any deeper meaning or relation to any file or Docker image or system or any other constraints. Think of the constraint_setting as an Enum and the constraint_value being the one and only member. We need to pass around a constant value, not a type, so we need to provide the constraint_value (e.g. in toolchain/BUILD.bazel) but not a constraint_setting. However we need a constraint_setting declared so we can make a constraint_value of that "type". Notice the platform declared here - it allows us to force Bazel to use the hermetic toolchain because of the extra constraint_value. - .bazelrc I set a few flags that will be on for all bazel build commands. Importantly, this causes the C++ build logic to use platforms and not the old, bespoke way. I also found a way to avoid using the local toolchain on the host, which will hopefully lead to clearer errors if platforms are mis-specified instead of odd compile errors because the host toolchain is too old or something. There are also a few RBE settings tweaked to be a bit more modern, as well the new shorthands for specifying target platforms (e.g. for_linux_x64). - bazel/buildrc where we have to turn off the platforms logic for emscripten https://github.com/emscripten-core/emsdk/issues/984 - bazel/rbe/BUILD.bazel for a fix in the platform description that makes it work on Mac. - Notice that _m1 has been removed from the mac-related toolchain files because the same toolchain should work on both architectures. - All other changes in any order. [1] https://bazel.build/docs/toolchains#debugging-toolchains [2] https://bazel.build/docs/toolchains#toolchain-resolution [3] https://bazel.build/reference/command-line-reference [4] https://bazel.build/docs/toolchains#registering-building-toolchains [5] https://github.com/google/skia/blob/17dc3f16fc78477503ba1ef484c3b47bc3aab893/gn/skia/BUILD.gn#L258-L271 Change-Id: I515c114099d659639a808f74e47d489a68b7af62 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/549737 Reviewed-by: Erik Rose <erikrose@google.com> Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-06-22 19:28:10 +00:00
# DO NOT EDIT THIS BAZEL FILE DIRECTLY
# Generated from ctx.file action in download_linux_amd64_toolchain.bzl
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
filegroup(
[bazel] Specify files required for the toolchain more precisely. This improves performance with sandboxing. One can have Bazel output performance data [1][2], which can be viewed via chrome://tracing or https://ui.perfetto.dev/. With Perfetto, the following SQL queries [3][4] can summarize the sandboxing metrics, as well as the actual compilation time. http://screen/5TxbeZTso4gNDfD Note that the dur column is in nanoseconds, so we convert to seconds. These numbers could further be divided by the number of processes (in my case 48) to get a scaled output. SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "sandbox.createFileSystem"; SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "sandbox.delete"; SELECT SUM(dur) / 1000000000.0 FROM slice WHERE name = "subprocess.run"; I benchmarked the local compilation of //:skia_public using --config=clang_linux (our custom Linux toolchain). I was sure to clear the Bazel cache before each run and not count the time to download/update a toolchain for the first time. The control measurements (without this CL) are: Wall Time = 272.2s sandbox.createFileSystem = 5466.9s subprocess.run = 2961.0s sandbox.delete = 4112.3s With this CL: Wall Time = 53.9s (5.05x faster) sandbox.createFileSystem = 403.4s subprocess.run = 1610.3s sandbox.delete = 348.8s The control measurement is a touch misleading. Due to it being so slow, I had recommended developers use a ramdisk when building on machines with sufficient RAM via the Bazel flag --sandbox_base=/dev/shm Here is the control measurement when using a RAM disk: Wall Time = 21.2s sandbox.createFileSystem = 58.9s subprocess.run = 705.1s sandbox.delete = 46.6s With this CL and a RAM disk: Wall Time = 19.2s (10% faster) sandbox.createFileSystem = 21.8s subprocess.run = 722.9s sandbox.delete = 16.2s For devs who cannot or are not using a RAM disk, this is a huge win. With a RAM disk, it's a modest improvement. On an RBE build, this had minimal impact. I did my best to bust the action cache with a fake define and the before and after times were about the same. This was inspired by [5] and [6]. [1] Add --profile=/tmp/profile.gz to any command [2] https://bazel.build/rules/performance#performance-profiling [3] https://perfetto.dev/docs/quickstart/trace-analysis#sample-queries [4] https://perfetto.dev/docs/analysis/sql-tables#slice [5] https://github.com/emscripten-core/emsdk/commit/93f21c9ef30bab52de24f9d4ea3f2f377cf6326a [6] https://github.com/emscripten-core/emsdk/commit/311acff345fd71dcfe5f350653cec466ee7e3fbc Change-Id: Iceb2606e86111159141a286d01fc002d09fe3fe4 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/547822 Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2022-06-07 20:37:48 +00:00
name = "archive_files",
srcs = [
"bin/llvm-ar",
],
visibility = ["//visibility:public"],
)
filegroup(
name = "compile_files",
srcs = [
"bin/clang",
"usr/bin/include-what-you-use",
] + glob(
include = [
"include/c++/v1/**",
"usr/include/**",
"lib/clang/13.0.0/**",
"usr/include/x86_64-linux-gnu/**",
],
allow_empty = False,
),
visibility = ["//visibility:public"],
)
filegroup(
name = "link_files",
srcs = [
"bin/clang",
"bin/ld.lld",
"bin/lld",
"lib/libc++.a",
"lib/libc++abi.a",
"lib/libunwind.a",
"lib64/ld-linux-x86-64.so.2",
] + glob(
include = [
"lib/clang/13.0.0/lib/**",
"lib/x86_64-linux-gnu/**",
"usr/lib/x86_64-linux-gnu/**",
],
allow_empty = False,
),
visibility = ["//visibility:public"],
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
)
""",
executable = False,
)
[bazel] Sketch out changes for Mac toolchain Change-Id: Idae84d8d9538012e5cfb75a1a477dbc72a4da5bc Bug: skia:13125 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/526264 Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-04-01 12:29:11 +00:00
# https://bazel.build/rules/repository_rules
download_linux_amd64_toolchain = repository_rule(
implementation = _download_linux_amd64_toolchain_impl,
[bazel] Sketching out HelloWorld sk_app using GL bazel run //example:hello_world --config=clang causes a window to open and draws a circle and a square. Text to follow in a future CL. To make this work, I had to get rid of musl and use glibc. All the shared libraries (.so files) that were pre-built and available for download (e.g. from https://packages.debian.org/bullseye/amd64/libgl1/download) were compiled against glibc. When I tried to run a program statically linked with musl and dynamically linked against things using glibc, I got a segmentation fault on things like calloc(). Initial attempts to use glibc had failed because it was thought that the libc.so.6 file could only be referred to by absolute path (and thus Bazel would not be happy about it). As it turns out, that was simply a misconfiguration of the builtin_sysroot parameter to cc_common.create_cc_toolchain_config_info (see //toolchain/clang_toolchain_config.bzl). By setting that to `external/clang_linux_amd64` and not `external/clang_linux_amd64/usr`, the libc binary which had been extracted to `external/clang_linux_amd64/lib/x86_64-linux-gnu` was perfectly reachable from `external/clang_linux_amd64/usr/usr/lib/x86_64-linux-gnu/libc.so` To bring in the shared libraries to link against (e.g. X11, GL) I made build_toolchain.bzl easier to modify in that we simply need to add a debian download url and sha256 hash to a list (rather than having to plumb this through via arguments). Recommended Review Order: - example/BUILD.bazel (not sure if we always want to set bare link arguments like that or if we want to use "features" to pass those along to the toolchain). - tools/sk_app/BUILD.bazel to see initial cc_library for wrapping sk_app code. - toolchain/build_toolchain.bzl to see removal of musl and new list of debs. - toolchain/clang_toolchain_config.bzl (where use of the no-canonical-prefixes was key to compilation success). Notice also that we statically linked libc++ (I did not have any shared libraries for it locally, so I guessed a typical developer might not either). - Rest of toolchain/ for trivial renames. - bazel/Makefile to see extra docs on those targets and a new target that compiles all the exes so far for a quick way to test the build. - third_party/BUILD.bazel and src/gpu/BUILD.bazel which have non-generated changes. (all other BUILD.bazel files do). - go.mod, which needed to update the infra repo version in order to pick up http://review.skia.org/491736). Change-Id: I8687bd227353040eca2dffa9465798d8bd395027 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/492117 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com> Commit-Queue: Kevin Lubick <kjlubick@google.com>
2022-01-11 12:35:26 +00:00
attrs = {},
[bazel] Sketch out changes for Mac toolchain Change-Id: Idae84d8d9538012e5cfb75a1a477dbc72a4da5bc Bug: skia:13125 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/526264 Reviewed-by: Jorge Betancourt <jmbetancourt@google.com>
2022-04-01 12:29:11 +00:00
doc = "Downloads clang, and all supporting headers, executables, " +
"and shared libraries required to build Skia on a Linux amd64 host",
[infra] Add hermetic toolchain for C/C++ using Clang+Musl This can successfully build a C library: bazel build --config=clang //third_party:libpng This can build and run a statically-linked executable: bazel test --config=clang //:bazel_test For more verbose compile and linking output, add the `--features diagnostic` flag to a Bazel command (see _make_diagnostic_flags() in toolchain/clang_toolchain_config.bzl. Similarly, a `--features print_search_dirs` can be used to show where clang is looking for libraries etc to link against. These features are made available for easier debugging. Suggested review order: - Read https://docs.bazel.build/versions/4.2.1/tutorial/cc-toolchain-config.html if unfamiliar with setting up C++ toolchains in Bazel - .bazelrc and WORKSPACE.bazel that configure use and download of the toolchain (Clang 13, musl 1.2.2) - toolchain/build_toolchain.bzl which downloads and assembles the toolchain (w/o installing anything on the host machine) - toolchain/BUILD.bazel and toolchain/*trampoline.sh to see the setup of the toolchain rules. - toolchain/clang_toolchain_config.bzl to see the configuration of the toolchain. Pay special attention to the various command line flags that are set. - See that tools/bazel_test.cc has made a new home in experimental/bazel_test/bazel_test.cpp, with a companion BUILD.bazel. Note the addition of some function calls that test use of the C++ standard library. The number being used to test the PNG library is the latest and greatest that verifies we are compiling the one brought in via DEPS (and not a local one). - third_party/* to see how png (and its dependent zlib) have been built. Pay special attention to the musl_compat hack to fix static linking (any idea what the real cause is?) - //BUILD.bazel to see definition of the bazel_test executable. Change-Id: I7b0922d0d45cb9be8df2fd5fa5a1f48492654d5f Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/461178 Reviewed-by: Ben Wagner <bungeman@google.com> Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2021-10-20 20:20:42 +00:00
)
Reference in New Issue Copy Permalink