AuroraMiddleware/glibc
1
0
Fork 0
mirror of https://sourceware.org/git/glibc.git synced 2024-12-13 14:50:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

x86-64: Avoid rep movsb with short distance [BZ #27130]

Browse Source 
When copying with "rep movsb", if the distance between source and
destination is N*4GB + [1..63] with N >= 0, performance may be very
slow.  This patch updates memmove-vec-unaligned-erms.S for AVX and
AVX512 versions with the distance in RCX:

	cmpl	$63, %ecx
	// Don't use "rep movsb" if ECX <= 63
	jbe	L(Don't use rep movsb")
	Use "rep movsb"

Benchtests data with bench-memcpy, bench-memcpy-large, bench-memcpy-random
and bench-memcpy-walk on Skylake, Ice Lake and Tiger Lake show that its
performance impact is within noise range as "rep movsb" is only used for
data size >= 4KB.

(cherry picked from commit 3ec5d83d2a)
This commit is contained in:
H.J. Lu 2021-01-12 05:15:49 -08:00
parent 61e8ae9b66
commit 369b47aaf7
2 changed files with 22 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
NEWS
View File

@ -57,6 +57,7 @@ The following bugs are resolved with this release:
[25933] Off by one error in __strncmp_avx2
[25966] Incorrect access of __x86_shared_non_temporal_threshold for x32
[25976] nss_compat: internal_end*ent may clobber errno, hiding ERANGE
[27130] "rep movsb" performance issue
Version 2.30

21
sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S
View File

@ -67,6 +67,13 @@
# define REP_MOVSB_THRESHOLD (2048 * (VEC_SIZE / 16))
#endif
/* Avoid short distance rep movsb only with non-SSE vector. */
#ifndef AVOID_SHORT_DISTANCE_REP_MOVSB
# define AVOID_SHORT_DISTANCE_REP_MOVSB (VEC_SIZE > 16)
#else
# define AVOID_SHORT_DISTANCE_REP_MOVSB 0
#endif
#ifndef PREFETCH
# define PREFETCH(addr) prefetcht0 addr
#endif
@ -257,7 +264,21 @@ L(movsb):
# error Unsupported REP_MOVSB_THRESHOLD and VEC_SIZE!
# endif
jb L(more_8x_vec_backward)
# if AVOID_SHORT_DISTANCE_REP_MOVSB
movq %rdi, %rcx
subq %rsi, %rcx
jmp 2f
# endif
1:
# if AVOID_SHORT_DISTANCE_REP_MOVSB
movq %rsi, %rcx
subq %rdi, %rcx
2:
/* Avoid "rep movsb" if RCX, the distance between source and destination,
is N*4GB + [1..63] with N >= 0. */
cmpl $63, %ecx
jbe L(more_2x_vec) /* Avoid "rep movsb" if ECX <= 63. */
# endif
mov %RDX_LP, %RCX_LP
rep movsb
L(nop):