glibc/sysdeps/x86_64/multiarch/memrchr-avx2-rtm.S

#ifndef MEMRCHR
# define MEMRCHR __memrchr_avx2_rtm
#endif

#define COND_VZEROUPPER	COND_VZEROUPPER_XTEST
#define ZERO_UPPER_VEC_REGISTERS_RETURN \
  ZERO_UPPER_VEC_REGISTERS_RETURN_XTEST

#define VZEROUPPER_RETURN jmp	 L(return_vzeroupper)

#define SECTION(p) p##.avx.rtm

#include "memrchr-avx2.S"
x86-64: Add AVX optimized string/memory functions for RTM Since VZEROUPPER triggers RTM abort while VZEROALL won't, select AVX optimized string/memory functions with xtest jz 1f vzeroall ret 1: vzeroupper ret at function exit on processors with usable RTM, but without 256-bit EVEX instructions to avoid VZEROUPPER inside a transactionally executing RTM region. 2021-03-05 15:26:42 +00:00			`#ifndef MEMRCHR`
			`# define MEMRCHR __memrchr_avx2_rtm`
			`#endif`

x86: Optimize memrchr-avx2.S The new code: 1. prioritizes smaller user-arg lengths more. 2. optimizes target placement more carefully 3. reuses logic more 4. fixes up various inefficiencies in the logic. The biggest case here is the `lzcnt` logic for checking returns which saves either a branch or multiple instructions. The total code size saving is: 306 bytes Geometric Mean of all benchmarks New / Old: 0.760 Regressions: There are some regressions. Particularly where the length (user arg length) is large but the position of the match char is near the beginning of the string (in first VEC). This case has roughly a 10-20% regression. This is because the new logic gives the hot path for immediate matches to shorter lengths (the more common input). This case has roughly a 15-45% speedup. Full xcheck passes on x86_64. Reviewed-by: H.J. Lu <hjl.tools@gmail.com> 2022-06-07 04:11:32 +00:00			`#define COND_VZEROUPPER COND_VZEROUPPER_XTEST`
x86-64: Add AVX optimized string/memory functions for RTM Since VZEROUPPER triggers RTM abort while VZEROALL won't, select AVX optimized string/memory functions with xtest jz 1f vzeroall ret 1: vzeroupper ret at function exit on processors with usable RTM, but without 256-bit EVEX instructions to avoid VZEROUPPER inside a transactionally executing RTM region. 2021-03-05 15:26:42 +00:00			`#define ZERO_UPPER_VEC_REGISTERS_RETURN \`
			`ZERO_UPPER_VEC_REGISTERS_RETURN_XTEST`

			`#define VZEROUPPER_RETURN jmp L(return_vzeroupper)`

			`#define SECTION(p) p##.avx.rtm`

			`#include "memrchr-avx2.S"`