A flat array was consuming way too much memory and was far too slow to
initialize properly with a very large ID bound (8 million IDs, showed up as #1 hotspot in perf).
Meta struct does not have to be in-order as we never iterate over it in
a meaningful way, so using a hashmap here is reasonable. Very few IDs
should need decorations or meta-data, so this should also be a quite
decent memory save.
For the pathological case, a 6x uplift was observed.
This is a fairly fundamental change on how IDs are handled.
It serves many purposes:
- Improve performance. We only need to iterate over IDs which are
relevant at any one time.
- Makes sure we iterate through IDs in SPIR-V module declaration order
rather than ID space. IDs don't have to be monotonically increasing,
which was an assumption SPIRV-Cross used to have. It has apparently
never been a problem until now.
- Support LUTs of structs. We do this by interleaving declaration of
constants and struct types in SPIR-V module order.
To support this, the ParsedIR interface needed to change slightly.
Before setting any ID with variant_set<T> we let ParsedIR know
that an ID with a specific type has been added. The surface for change
should be minimal.
ParsedIR will maintain a per-type list of IDs which the cross-compiler
will need to consider for later.
Instead of looping over ir.ids[] (which can be extremely large), we loop
over types now, using:
ir.for_each_typed_id<SPIRVariable>([&](uint32_t id, SPIRVariable &var) {
handle_variable(var);
});
Now we make sure that we're never looking at irrelevant types.
This allows shaders to declare and use pointer-type variables. Pointers
may be loaded and stored, be the result of an `OpSelect`, be passed to
and returned from functions, and even be passed as inputs to the `OpPhi`
instruction. All types of pointers may be used as variable pointers.
Variable pointers to storage buffers and workgroup memory may even be
loaded from and stored to, as though they were ordinary variables. In
addition, this enables using an interior pointer to an array as though
it were an array pointer itself using the `OpPtrAccessChain`
instruction.
This is a rather large and involved change, mostly because this is
somewhat complicated with a lot of moving parts. It's a wonder
SPIRV-Cross's output is largely unchanged. Indeed, many of these changes
are to accomplish exactly that! Perhaps the largest source of changes
was the violation of the assumption that, when emitting types, the
pointer type didn't matter.
One of the test cases added by the change doesn't optimize very well;
the output of `spirv-opt` here is invalid SPIR-V. I need to file a bug
with SPIRV-Tools about this.
I wanted to test that variable pointers to images worked too, but I
couldn't figure out how to propagate the access qualifier properly--in
MSL, it's part of the type, so getting this right is important. I've
punted on that for now.
Just like OpAccessChain we need to make use of the meta information
available to use from access_chain_internal as we can extract a packed
vector or transposed vector from a composite, not just memory load.
A block name cannot alias with any name in its own scope,
and it cannot alias with any other "global" name.
To solve this, we need to complicate the name cache updates a little bit
where we have a "primary" namespace and "secondary" namespace.
This is required to avoid relying on complex sub-expression elimination
in compilers, and generates cleaner code.
The problem case is if a complex expression is used in an access chain,
like:
Composite comp = buffer[texture(...)];
vec4 a = comp.a + comp.b + comp.c;
Before, we did not have common subexpression tracking for
OpLoad/OpAccessChain, so we easily ended up with code like:
vec4 a = buffer[texture(...)].a + buffer[texture(...)].b + buffer[texture(...)].c;
A good compiler will optimize this, but we should not rely on it, and
forcing texture(...) to a temporary also looks better.
The solution is to add a vector "implied_expression_reads", which works
similarly to expression_dependencies. We also need an extra mechanism in
to_expression which lets us skip expression read checking and do it
later. E.g. for expr -> access chain -> load, we should only trigger
a read of expr when using the loaded expression.
Avoids certain cases of variance between translation units by forcing
every dependent expression of a store to be temporary.
Should avoid the major failure cases where invariance matters.
In GLSL, 8-bit types require GL_EXT_shader_8bit_storage. 16-bit types
can use either GL_AMD_gpu_shader_int16/GL_AMD_gpu_shader_half_float or
GL_EXT_shader_16bit_storage.
When trying to validate buffer sizes, we usually need to bail out when
using SpecConstantOps, but for some very specific cases where we allow
unsized arrays currently, we can safely allow "unknown" sized arrays as
well.
This is probably the best we can do, when we have even more difficult
cases than this, we throw a more sensible error message.
Previously, when generating non-Vulkan GLSL, each use of a spec constant
would be subsituted for its default value and the declaration of the constant
itself would be omitted completely.
This change slightly alters this behavior. The uses of the constant are kept,
as well as the declaration, although the latter is stripped of the layout
qualifier. The declaration is also prepended with the following code:
#ifndef <constant name>_value
#define <constant name> <default constant value>
#endif
and the constant itself now looks like
const <constant type> <constant name> = <constant name>_value;
The rationale for this change is that it gives the user a way to provide
custom values for specialization constants even when the target does not
support them.
Setting force_temporary to true produces invalid GLSL because sampler
variables are copied:
highp sampler2D _377 = DiffuseMapTexture;
This change fixes the problem by always forwarding forwardable
variables. I also took an opportunity to restructure the code to make
it easier to read and add extra conditions to in the future.
- Add new Windows support
- Use CMake/CTest instead of Make + shell scripts
- Use --parallel in CTest
- Fix CTest on Windows
- Cleanups in test_shaders.py
- Force specific commit for SPIRV-Headers
- Fix Inf/NaN odd-ball case by moving to ASM
This is a large refactor which splits out the SPIR-V parser from
Compiler and moves it into its more appropriately named Parser module.
The Parser is responsible for building a ParsedIR structure which is
then consumed by one or more compilers.
Compiler can take a ParsedIR by value or move reference. This should
allow for optimal case for both multiple compilations and single
compilation scenarios.
Even as of Metal 2.1, MSL still doesn't support arrays of buffers
directly. Therefore, we must manually expand them. In the prologue, we
define arrays holding the argument pointers; these arrays are what the
transpiled code ends up referencing. We might be able to do similar
things for textures and samplers prior to MSL 2.0.
Speaking of which, also enable texture arrays on iOS MSL 1.2.
Need some pretty hideous ladder variable system, but high level
languages do not support breaking out of a loop. break in switch blocks
and break in loops alias each other.
Implement this by flattening outputs and unflattening inputs explicitly.
This allows us to pass down a single struct instead of dealing with the
insanity that would be passing down each flattened member separately.
Remove stage_uniforms_var_id.
Seems to be dead code. Naked uniforms do not exist in SPIR-V for Vulkan,
which this seems to have been intended for. It was also unused elsewhere.
When the name of an alias global variable collides with a global
declaration, MSL would emit inconsistent names, sometimes with the
naming fix, sometimes without, because names were being tracked in two
separate meta blocks. Fix this by always redirecting parameter naming to
the original base variable as necessary.
Some projects build SPIRV-Cross as a single translation unit
and this causes a lot of warnings because the same macro is redeclared
multiple times in the different backends. This make sure that each
backend has its own namespace for internal macros.
Support flattening StorageOutput & StorageInput matrices and arrays.
No longer move matrix & array inputs to separate buffer.
Add separate SPIRFunction::fixup_statements_in & SPIRFunction::fixup_statements_out
instead of just SPIRFunction::fixup_statements.
Emit SPIRFunction::fixup_statements at beginning of functions.
CompilerMSL track vars_needing_early_declaration.
Pass global output variables as variables to functions that access them.
Sort input structs by location, same as output structs.
Emit struct declarations in order output, input, uniforms.
Regenerate reference shaders to new formats defined by above.
OSX 10.14 broke (?) how overload resolution works,
so overloading e.g. dot(float3, packed_float3) no longer works.
Fix this by unpacking expressions before various func ops.
This fix might need to be applied elsewhere, but do so later if needed.
- Do not emit set = in GLSL, even when non-zero.
- Fix warning on tautological comparison.
- Expose get_buffer_block_flags as mentioned in reflection guide.
textureGrad isn't part of core GL2 or GLES2. In desktop GL, it's
provided by GL_ARB_shader_texture_lod and is called texture2DGradARB; in
ES, it's provided by GL_EXT_shader_texture_lod and is called
texture2DGradEXT.
This change rewrites textureGrad similarly to textureProj as per above.
- The HLSL compiler now has its own list of keywords in addition to
the ones from GLSL.
- Added "buffer", "precise", and "shared" to the GLSL keywords.
Deal with various query functions which require dummy sampler.
In SPIR-V, separate images are used, but GLSL (even Vulkan GLSL)
requires combined sampler images ...
Support MSL typedefs to declare 3-row row-major matrices as 3-column matrices.
Allow those matrices to be decorated as packed.
Support transposing those matrices when used.
Modify how member alignments are calculated.
Certain patterns with OpVectorShuffle (and probably others) will cascade
to so large, that they can cause OOM. After we have observed
force_recompile, don't spend unnecessary memory emitting code which will
never be used.
Normally, temporary declaration must dominate any use of it,
so we generally did not need to analyze the CFG for these variables,
but there is an edge case where you have an inliner doing:
do {
create_temporary;
break;
} while(0);
use_temporary;
The inside of the loop dominates the outer scope, but we cannot emit
code like this in GLSL, so make sure we hoist these temporaries outside
the "loop".
HLSL UAVs are a bit annoying because they can share block types,
so reflection becomes rather awkward. Sometimes we will need to make
some nasty fallbacks, so add a reflection interface which lets you query
post-shader compile which names was actually declared in the shader.
We don't have a mechanism to move temporaries to their appropriate
scope, and Phi behavior is weird enough that it will be a heroic effort
to not do this rather ugly codegen :(
Support Workgroup (threadgroup) variables.
Mark if SPIRConstant is used as an array length, since it cannot be specialized.
Resolve specialized array length constants.
Support passing an array to MSL function.
Support emitting GLSL array assignments in MSL via an array copy function.
Support for memory and control barriers.
Struct packing enhancements, including packing nested structs.
Enhancements to replacing illegal MSL variable and function names.
Add Compiler::get_entry_point_name_map() function to retrieve entry point renamings.
Remove CompilerGLSL::clean_func_name() as obsolete.
Fixes to types in bitcast MSL functions.
Add Variant::get_id() member function.
Add CompilerMSL::Options::msl_version option.
Add numerous MSL compute tests.
Emit input struct assignment by assigning member by member from stage_in struct.
Map qualified member name from pointer type, not base type.
Add Comiler::expression_type_id() function, similar to expression_type().
They might potentially be used as part of OpStore in the SPIRV-Tools
inliner in some cases.
Implement these as declared variables but without any initializer.
Support BuiltInFragDepth.
Emit interface block for StorageClassUniformConstant.
Throw exception when output or fragment input structs contain matrix or array.
Dynamically created interface structs sorted by location number instead of alphabetically.
Add Compiler::is_array() function.
CompilerGLSL type_to_glsl() and image_type_glsl() functions support optional object ID.
Add SPIRType::Image::access member to support SPIR-V OpTypeImage access qualifier.
Remove SPIRType::Image::is_read and ::is_written members.
Use DecorationNonReadable and DecorationNonWritable to mark read/write access for image variables.
CompilerMSL emit access qualifiers per image variable, instead of per image type.
CompilerGLSL and CompilerHLSL behaviour is unchanged.
Add bool members is_read and is_written to SPIRType::Image.
Output correct texture read/write access by marking whether textures
are read from and written to by the shader.
Override bitcast_glsl_op() to use Metal as_type<type> functions.
Add implementations of SPIR-V functions inverse(), degrees() & radians().
Map inverseSqrt() to rsqrt().
Map roundEven() to rint().
GLSL functions imageSize() and textureSize() map to equivalent
expression using MSL get_width() & get_height() functions.
Map several SPIR-V integer bitfield functions to MSL equivalents.
Map SPIR-V atomic functions to MSL equivalents.
Map texture packing and unpacking functions to MSL equivalents.
Refactor existing, and add new, image query functions.
Reorganize header lines into includes and pragmas.
Simplify type_to_glsl() logic.
Add MSL test case vert/functions.vert for added function implementations.
Add MSL test case comp/atomic.comp for added function implementations.
test_shaders.py use macOS compilation for MSL shader compilation validations.
WebGL supports lod texture funcs only in fragment
shaders but SPIR-V supports only lod texture funcs
in vertex shaders. This reverts calls which were
forced (infered from using a 0 constant) to use
an lod to plain calls in vertex shaders when
using legacy es.
spirv_msl optionally add padding and packing to allow MSL
struct members to align with SPIR-V struct alignments.
spirv_cross add convenience methods for testing Decorations.
spirv_glsl replace member_decl() function with new emit_stuct_member().
Allow struct member types to be marked as packed via DecorationCPacked decoration.
In some cases, the compiler decided to emit continue block first,
which invalidated the expressions used by the condition.
Parameters to functions can be evaluated in any order which caused
"random" behavior.
To extract a column from row-major matrix, we need to do a strided load one
component at a time. In this case flattened_access_chain_offset still returns
the offset to the first element, but the stride is equal to matrix stride
instead of vector stride.
For this to work, we need to pass matrix stride (and transpose flag) through,
similar to how matrix flattening works.
Additionally slightly clean up recursive flattened_access_chain structure -
specifically, instead of deciding mid-traversal that we need matrix stride
information, we can just pass the matrix stride through - for access chains
that end in matrix/vector this gets us what we need, and for access chains
that end in structs the flattened_access_chain_struct code will recompute
correct stride/transposition data to pass through further.
We currently only support access chains that end in a matrix by propagating
"needs transpose" flag upstream which flips the matrix multiplication order.
It's possible to support indexed extraction as well, however it would have to
generate code like this:
vec4 row = vec4(UBO[0].y, UBO[1].y, UBO[2].y, UBO[3].y);
for a column equivalent of:
vec4 row = UBO[1];
It is definitely possible to do so but it requires signaling the vector output
that it needs to switch to per-component extraction which is a bit more trouble
than this is worth for now.
Instead of filling a std::string buffer passed by reference return a new
string. This may be slightly slower in certain cases but they are pretty
rare and this matches the code style better.
Also streamline error handling in different branches and extract function
to generate vector swizzle.
Legacy GLSL targets do not support uniform buffers, and as such require
some sort of emulation. There are two alternatives - one is to represent
a uniform buffer as a uniform struct, and another one is to flatten it
into an array of primitive vector types (vec4).
Uniform struct have two disadvantages that make using them prohibitive
in some applications:
- The location assignment for struct members is arbitrary which means
the application has to set each struct member one by one
- Some Android drivers fail to link shader programs if both vertex and
fragment shader use the same uniform struct
Because of this, we need to support flattening uniform buffers into an
array. This is not just important for legacy GLSL but also is sometimes
useful for ESSL 3.0 where some Android drivers do not have stable UBO
support.
The way flattening works is the entire buffer is represented as a vec4
array; each access chain is rewritten into a combination of array
accesses, swizzles and data type constructors. Specifically:
- Extracting a vector or a scalar requires indexing into the array with
an optional swizzle, for example CB0[13].yz for reading vec2
- Extracting a matrix or a struct requires extracting each individual
vector or struct member and then combining them into the resulting
object
- Extracting arrays is not supported, mostly because the resulting
construct is very inefficient and ESSL 1.0 does not support array
constructors.
Additionally, while we try to constant-fold each individual indexing
operation, there are cases where we have to use dynamic index
computation (specifically for indexing arrays with non-constants); so
the general form of the primitive array extraction expression is:
buffer[stride0*index0+...+strideN*indexN+offset]
Where stride/offset are integer literals and index represents variables.
- By default, emit uniform structs for UBOs, like push constant.
- Forward transpose information,
and optimize transpose(matrix) * vector to vector * matrix.
CompilerGLSL add to_function_name() and to_function_args() functions to organize
structure of emit_texture_op() function.
CompilerMSL add support for MSL gather(), gather_compare() and sample_compare() functions.
