skia2/gn/compile_sksl_tests.py

144 lines
4.9 KiB
Python
Raw Normal View History

#!/usr/bin/env python
#
# Copyright 2020 Google LLC
#
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
import os
import shlex
import subprocess
import sys
import tempfile
batchCompile = True
skslc = sys.argv[1]
lang = sys.argv[2]
settings = sys.argv[3]
with open(sys.argv[4], 'r') as reader:
inputs = shlex.split(reader.read())
def pairwise(iterable):
# Iterate over an array pairwise (two elements at a time).
a = iter(iterable)
return zip(a, a)
def executeWorklist(input, worklist):
# Invoke skslc, passing in the worklist.
worklist.close()
try:
output = subprocess.check_output([skslc, worklist.name], stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as err:
if err.returncode != 1:
print("### " + input + " skslc error:\n")
print("\n".join(err.output.splitlines()))
sys.exit(err.returncode)
pass # Compile errors (exit code 1) are expected and normal in test code
# Delete the worklist file now that execution is complete.
os.remove(worklist.name)
def makeEmptyFile(path):
try:
open(path, 'wb').close()
except OSError:
pass
def extensionForSpirvAsm(ext):
return ext if (ext == '.frag' or ext == '.vert' or ext == '.geom') else '.frag'
if settings != "--settings" and settings != "--nosettings":
sys.exit("### Expected --settings or --nosettings, got " + settings)
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
targets = []
worklist = tempfile.NamedTemporaryFile(suffix='.worklist', delete=False)
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
# The `inputs` array pairs off input files with their matching output directory, e.g.:
# //skia/tests/sksl/shared/test.sksl
# //skia/tests/sksl/shared/golden/
# //skia/tests/sksl/intrinsics/abs.sksl
# //skia/tests/sksl/intrinsics/golden/
# ... (etc) ...
# Here we loop over these inputs and convert them into a worklist file for skslc.
for input, targetDir in pairwise(inputs):
noExt, ext = os.path.splitext(input)
head, tail = os.path.split(noExt)
if not os.path.isdir(targetDir):
os.mkdir(targetDir)
target = os.path.join(targetDir, tail)
if settings == "--nosettings":
target += "StandaloneSettings"
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
targets.append(target)
if lang == "--fp":
worklist.write(input + "\n")
worklist.write(target + ".cpp\n")
worklist.write(settings + "\n\n")
worklist.write(input + "\n")
worklist.write(target + ".h\n")
worklist.write(settings + "\n\n")
elif lang == "--dslfp":
worklist.write(input + "\n")
worklist.write(target + ".dsl.cpp\n")
worklist.write(settings + "\n\n")
worklist.write(input + "\n")
worklist.write(target + ".h\n")
worklist.write(settings + "\n\n")
elif lang == "--glsl":
worklist.write(input + "\n")
worklist.write(target + ".glsl\n")
worklist.write(settings + "\n\n")
elif lang == "--metal":
worklist.write(input + "\n")
worklist.write(target + ".metal\n")
worklist.write(settings + "\n\n")
elif lang == "--spirv":
worklist.write(input + "\n")
worklist.write(target + ".asm" + extensionForSpirvAsm(ext) + "\n")
worklist.write(settings + "\n\n")
elif lang == "--skvm":
worklist.write(input + "\n")
worklist.write(target + ".skvm\n")
worklist.write(settings + "\n\n")
elif lang == "--stage":
worklist.write(input + "\n")
worklist.write(target + ".stage\n")
worklist.write(settings + "\n\n")
else:
sys.exit("### Expected one of: --fp --dslfp --glsl --metal --spirv --skvm --stage, got " +
lang)
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
# Compile items one at a time.
if not batchCompile:
executeWorklist(input, worklist)
worklist = tempfile.NamedTemporaryFile(suffix='.worklist', delete=False)
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
# Compile everything all in one go.
if batchCompile:
executeWorklist("", worklist)
else:
worklist.close()
os.remove(worklist.name)
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
# A special case cleanup pass, just for CPP and H files: if either one of these files starts with
# `### Compilation failed`, its sibling should be replaced by an empty file. This improves clarity
# during code review; a failure on either file means that success on the sibling is irrelevant.
if lang == "--fp" or lang == "--dslfp":
cppExtension = (".dsl.cpp" if lang == "--dslfp" else ".cpp")
hExtension = ".h"
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
for target in targets:
cppFile = open(target + cppExtension, 'r')
hFile = open(target + hExtension, 'r')
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
if cppFile.readline().startswith("### Compilation failed"):
# The CPP had a compilation failure. Clear the header file.
hFile.close()
makeEmptyFile(target + hExtension)
Update skslc to compile multiple files during one invocation. skslc can now take a `.worklist` file as an input, containing multiple "command lines" to run in sequence. compile_sksl_tests.py now assembles a worklist file and runs skslc one time, rather than running skslc once per each target. This improves compile times on Windows significantly (where spawning skslc hundreds of times is much more expensive than on Linux/Mac). One subtle behavioral difference with .worklist files: if an error is encountered, it is written to the output file instead of to stdout. Previously, compile_sksl_tests was in charge of for capturing stdout and overwriting the compiler output with the error message, but this doesn't work when many files are being compiled (which errors are associated with which files?) This refactor exposed a minor latent bug--when encountering an error, skslc would previously exit() immediately without closing its FileOutputStream. This led to an assertion when exit() was replaced with normal returns. Since FileOutputStream is only used by skslc, and in every case the desired behavior is just to close the stream cleanly, FileOutputStream now closes the file in its destructor instead of asserting that we haven't done so. Change-Id: Ia55baff0c11fe466923bde2e0c944df9f2ccd092 Bug: skia:10919 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/334099 Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: John Stiles <johnstiles@google.com> Auto-Submit: John Stiles <johnstiles@google.com>
2020-11-11 22:29:28 +00:00
elif hFile.readline().startswith("### Compilation failed"):
# The header had a compilation failure. Clear the CPP file.
cppFile.close()
makeEmptyFile(target + cppExtension)