skia2/tools/rewrite_includes.py
Kevin Lubick fed97e8f40 [bazel] Add RBE support using hermetic Linux Clang toolchain
A new RBE worker-pool called gce_linux was created in
conjunction with this CL. See
https://docs.google.com/document/d/14xMZCKews69SSTfULhE8HDUzT5XvPwZ4CvRufEvcZ74/edit#
for some details on that.

Note: everything under bazel/rbe/gce_linux was autogenerated
and can be ignored from manual review. It basically specifies
what files are on the RBE image that are necessary for running
Bazel.

Testing it out can be done by authenticating for RBE
gcloud auth application-default login --no-browser

Then, run make -C bazel rbe_known_good_builds
to test it out.

On my 4 core laptop with an empty local cache, but a
warm remote cache, the build took <2 min instead of the
10+ minutes it would have [1].

The folder structure in //bazel/rbe is meant to let us
have multiple remote configurations there, e.g.
//bazel/rbe/gce_windows.

Suggested Review Order:
 - bazel/rbe/README.md
 - bazel/rbe/gce_linux_container/Dockerfile to see the
   bare-bones RBE image.
 - bazel/rbe/BUILD.bazel to see a custom platform defined.
   It is nearly identical to the autogenerated one
   in bazel/rbe/gce_linux/config/BUILD, with one extra
   field to force the gce_linux pool to be used.
 - .bazelrc to see the settings needed to make
   --config=linux-rbe work. The naming convention was
   inspired by SkCMS's setup [2], and allows us to have
   some common RBE settings (i.e. config:remote) and
   some specialized ones for the given host machine
   (e.g. config:linux-rbe) A very important, but subtle
   configuration, is on line 86 of .bazelrc where we say
   to use our hermetic toolchain and not whatever C++
   compiler and headers are on the host machine (aka
   the RBE container).
 - toolchain/build_toolchain.bzl to see some additional
   dependencies needed in the toolchain (to run IWYU) which
   I had installed locally but didn't realize were important.
 - third_party/BUILD.bazel to see an example of how failing
   to specify all files can result in something that works
   locally, but fails remotely.
   --execution_log_json_file=/tmp/execlog.json helped debug
   these issues.
 - All other files.

[1] http://go/scrcast/NjM1ODE4MDI0NzM3MTc3Nnw3ODViZmFkMi1iOA
[2] https://skia.googlesource.com/skcms/+/30c8e303800c256febb03a09fdcda7f75d119b1b/.bazelrc#20


Change-Id: Ia0a9e6a06c1a13071949ab402dc5d897df6b12e1
Bug: skia:12541
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/524359
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
2022-03-28 13:56:16 +00:00

126 lines
3.6 KiB
Python
Executable File

#!/usr/bin/python
#
# Copyright 2019 Google Inc.
#
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
import argparse
import os
import six
import sys
from six import StringIO
parser = argparse.ArgumentParser()
parser.add_argument('-n', '--dry-run', action='store_true',
help='Just check there is nothing to rewrite.')
parser.add_argument('sources', nargs='*',
help='Source files to rewrite, or all if empty.')
args = parser.parse_args()
roots = [
'bench',
'dm',
'docs',
'example',
'experimental',
'fuzz',
'gm',
'include',
'modules',
'platform_tools/android/apps',
'samplecode',
'src',
'tests',
'third_party/etc1',
'third_party/gif',
'tools'
]
ignorelist = [
# Don't count our local Vulkan headers as Skia headers;
# we don't want #include <vulkan/vulkan_foo.h> rewritten to point to them.
'include/third_party/vulkan',
# Some node_modules/ files (used by CanvasKit et al) have c++ code which we should ignore.
'node_modules',
]
assert '/' in [os.sep, os.altsep]
def fix_path(p):
return p.replace(os.sep, os.altsep) if os.altsep else p
# Map short name -> absolute path for all Skia headers.
headers = {}
for root in roots:
for path, _, files in os.walk(root):
if not any(snippet in fix_path(path) for snippet in ignorelist):
for file_name in files:
if file_name.endswith('.h'):
if file_name in headers:
message = ('Header filename is used more than once!\n- ' + path + '/' + file_name +
'\n- ' + headers[file_name])
assert file_name not in headers, message
headers[file_name] = os.path.abspath(os.path.join(path, file_name))
def to_rewrite():
if args.sources:
for path in args.sources:
yield path
else:
for root in roots:
for path, _, files in os.walk(root):
for file_name in files:
yield os.path.join(path, file_name)
# Rewrite any #includes relative to Skia's top-level directory.
need_rewriting = []
for file_path in to_rewrite():
if ('/generated/' in file_path or
'tests/sksl/' in file_path or
'third_party/skcms' in file_path or
file_path.startswith('bazel/rbe')):
continue
if (file_path.endswith('.h') or
file_path.endswith('.c') or
file_path.endswith('.m') or
file_path.endswith('.mm') or
file_path.endswith('.inc') or
file_path.endswith('.cc') or
file_path.endswith('.cpp')):
# Read the whole file into memory.
lines = open(file_path).readlines()
# Write it back out again line by line with substitutions for #includes.
output = StringIO() if args.dry_run else open(file_path, 'w')
includes = []
for line in lines:
parts = line.replace('<', '"').replace('>', '"').split('"')
if (len(parts) == 3
and '#' in parts[0]
and 'include' in parts[0]
and os.path.basename(parts[1]) in headers):
header = fix_path(os.path.relpath(headers[os.path.basename(parts[1])], '.'))
includes.append(parts[0] + '"%s"' % header + parts[2])
else:
for inc in sorted(includes):
output.write(inc.strip('\n') + '\n')
includes = []
output.write(line.strip('\n') + '\n')
if args.dry_run and output.getvalue() != open(file_path).read():
need_rewriting.append(file_path)
rc = 1
output.close()
if need_rewriting:
print('Some files need rewritten #includes:')
for path in need_rewriting:
print('\t' + path)
print('To do this automatically, run')
print('python tools/rewrite_includes.py ' + ' '.join(need_rewriting))
sys.exit(1)