As for gettimeofday, time will be implemented based on clock_gettime
on all platforms and internal code should use clock_gettime
directly. In addition to removing a layer of indirection, this will
allow us to remove the PLT-bypass gunk for gettimeofday.
The changed code always assumes __clock_gettime (CLOCK_REALTIME)
or __clock_gettime (CLOCK_REALTIME_COARSE) (for Linux case) cannot
fail, using the same rationale for gettimeofday change. And internal
helper was added (time_now).
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc64le-linux-gnu,
powerpc64-linux-gnu, and powerpc-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Since gettimeofday will shortly be implemented in terms of
clock_gettime on all platforms, internal code should use clock_gettime
directly; in addition to removing a layer of indirection, this will
allow us to remove the PLT-bypass gunk for gettimeofday. (We can't
quite do that yet, but it'll be coming later in this patch series.)
In many cases, the changed code does fewer conversions.
The changed code always assumes __clock_gettime (CLOCK_REALTIME)
cannot fail. Most of the call sites were assuming gettimeofday could
not fail, but a few places were checking for errors. POSIX says
clock_gettime can only fail if the clock constant is invalid or
unsupported, and CLOCK_REALTIME is the one and only clock constant
that's required to be supported. For consistency I grepped the entire
source tree for any other places that checked for errors from
__clock_gettime (CLOCK_REALTIME), found one, and changed it too.
(For the record, POSIX also says gettimeofday can never fail.)
(It would be nice if we could declare that GNU systems will always
support CLOCK_MONOTONIC as well as CLOCK_REALTIME; there are several
places where we are using CLOCK_REALTIME where _MONOTONIC would be
more appropriate, and/or trying to use _MONOTONIC and then falling
back to _REALTIME. But the Hurd doesn't support CLOCK_MONOTONIC yet,
and it looks like adding it would involve substantial changes to
gnumach's internals and API. Oh well.)
A few Hurd-specific files were changed to use __host_get_time instead
of __clock_gettime, as this seemed tidier. We also assume this cannot
fail. Skimming the code in gnumach leads me to believe the only way
it could fail is if __mach_host_self also failed, and our
Hurd-specific code consistently assumes that can't happen, so I'm
going with that.
With the exception of support/support_test_main.c, test cases are not
modified, mainly because I didn't want to have to figure out which
test cases were testing gettimeofday specifically.
The definition of GETTIME in sysdeps/generic/memusage.h had a typo and
was not reading tv_sec at all. I fixed this. It appears nobody has been
generating malloc traces on a machine that doesn't have a superseding
definition.
There are a whole bunch of places where the code could be simplified
by factoring out timespec subtraction and/or comparison logic, but I
want to keep this patch as mechanical as possible.
Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc64le-linux-gnu,
powerpc64-linux-gnu, powerpc-linux-gnu, and aarch64-linux-gnu.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
This change adds a warning to nscd.conf about running multiple caching
services together and that it may lead to unexpected behaviours. Also we
add a note that enabling the 'shared' option will cause cache hit rates
to be misreported (a side effect of the implementation).
v2
- Rewrite comment to avoid implementation details.
The function uses the internal service_user type, so it is not
really usable from the outside of glibc. Rename the function
to __nss_database_lookup2 for internal use, and change
__nss_database_lookup to always indicate failure to the caller.
__nss_next already was a compatibility symbol. The new
implementation always fails and no longer calls __nss_next2.
unscd, the alternative nscd implementation, does not use
__nss_database_lookup, so it is not affected by this change.
This functionality was deprecated in glibc 2.25.
This commit only includes the core changes to remove the
functionality. It does not remove the RES_USE_INET6 handling in the
individual NSS service modules and the res_use_inet6 function.
These changes will happen in future commits.
I'm looking at the warnings from building glibc with -Wextra, to see
if we could use -Wextra by default, possibly with a few of its
warnings disabled, and so benefit from warnings in -Wextra but not in
-Wall. (The vast bulk of the extra warnings so produced are from
-Wunused-parameter -Wsign-compare -Wmissing-field-initializers
-Wtype-limits, so I expect those would be disabled at least at first.)
Various miscellaneous warnings show up with -Wextra that it clearly
seems to make sense to fix independent of whether we add -Wextra to
the normal options for building glibc. This patch fixes one:
"initialized field overwritten [-Woverride-init]" in nscd.
Tested for x86_64.
* nscd/connections.c (reqinfo): Initialize SHUTDOWN element only
once.
The IPv4 address parser in the getaddrinfo function is changed so that
it does not ignore trailing whitespace and all characters after it.
For backwards compatibility, the getaddrinfo function still recognizes
legacy name syntax, such as 192.000.002.010 interpreted as 192.0.2.8
(octal).
This commit does not change the behavior of inet_addr and inet_aton.
gethostbyname already had additional sanity checks (but is switched
over to the new __inet_aton_exact function for completeness as well).
To avoid sending the problematic query names over DNS, commit
6ca53a2453 ("resolv: Do not send queries
for non-host-names in nss_dns [BZ #24112]") is needed.
This is a major rewrite of the description of 'crypt', 'getentropy',
and 'getrandom'.
A few highlights of the content changes:
- Throughout the manual, public headers, and user-visible messages,
I replaced the term "password" with "passphrase", the term
"password database" with "user database", and the term
"encrypt(ion)" with "(one-way) hashing" whenever it was applied to
passphrases. I didn't bother making this change in internal code
or tests. The use of the term "password" in ruserpass.c survives,
because that refers to a keyword in netrc files, but it is adjusted
to make this clearer.
There is a note in crypt.texi explaining that they were
traditionally called passwords but single words are not good enough
anymore, and a note in users.texi explaining that actual passphrase
hashes are found in a "shadow" database nowadays.
- There is a new short introduction to the "Cryptographic Functions"
section, explaining how we do not intend to be a general-purpose
cryptography library, and cautioning that there _are_, or have
been, legal restrictions on the use of cryptography in many
countries, without getting into any kind of detail that we can't
promise to keep up to date.
- I added more detail about what a "one-way function" is, and why
they are used to obscure passphrases for storage. I removed the
paragraph saying that systems not connected to a network need no
user authentication, because that's a pretty rare situation
nowadays. (It still says "sometimes it is necessary" to
authenticate the user, though.)
- I added documentation for all of the hash functions that glibc
actually supports, but not for the additional hash functions
supported by libxcrypt. If we're going to keep this manual section
around after the transition is more advanced, it would probably
make sense to add them then.
- There is much more detailed discussion of how to generate a salt,
and the failure behavior for crypt is documented. (Returning an
invalid hash on failure is what libxcrypt does; Solar Designer's
notes say that this was done "for compatibility with old programs
that assume crypt can never fail".)
- As far as I can tell, the header 'crypt.h' is entirely a GNU
invention, and never existed on any other Unix lineage. The
function 'crypt', however, was in Issue 1 of the SVID and is now
in the XSI component of POSIX. I tried to make all of the
@standards annotations consistent with this, but I'm not sure I got
them perfectly right.
- The genpass.c example has been improved to use getentropy instead
of the current time to generate the salt, and to use a SHA-256 hash
instead of MD5. It uses more random bytes than is strictly
necessary because I didn't want to complicate the code with proper
base64 encoding.
- The testpass.c example has three hardwired hashes now, to
demonstrate that different one-way functions produce different
hashes for the same input. It also demonstrates how DES hashing
only pays attention to the first eight characters of the input.
- There is new text explaining in more detail how a CSPRNG differs
from a regular random number generator, and how
getentropy/getrandom are not exactly a CSPRNG. I tried not to make
specific falsifiable claims here. I also tried to make the
blocking/cancellation/error behavior of both getentropy and
getrandom clearer.
The pre-allocation of the three scratch buffers increased the initial
stack size somewhat, but if retries are needed, the previous version
used more stack space if extend_alloca could not merge allocations.
Lack of alloca accounting also means could be problematic with
extremely large NSS responses, too.
[BZ #18023]
* nscd/aicache.c (addhstaiX): Use struct scratch_buffer instead
of extend_alloca.
As indicated by BZ#23178, concurrent access on some files read by nscd
may result non expected data send through service requisition. This is
due 'sendfile' Linux implementation where for sockets with zero-copy
support, callers must ensure the transferred portions of the the file
reffered by input file descriptor remain unmodified until the reader
on the other end of socket has consumed the transferred data.
I could not find any explicit documentation stating this behaviour on
Linux kernel documentation. However man-pages sendfile entry [1] states
in NOTES the aforementioned remark. It was initially pushed on man-pages
with an explicit testcase [2] that shows changing the file used in
'sendfile' call prior the socket input data consumption results in
previous data being lost.
From commit message it stated on tested Linux version (3.15) only TCP
socket showed this issues, however on recent kernels (4.4) I noticed the
same behaviour for local sockets as well.
Since sendfile on HURD is a read/write operation and the underlying
issue on Linux, the straightforward fix is just remove sendfile use
altogether. I am really skeptical it is hitting some hotstop (there
are indication over internet that sendfile is helpfull only for large
files, more than 10kb) here to justify that extra code complexity or
to pursuit other possible fix (through memory or file locks for
instance, which I am not sure it is doable).
Checked on x86_64-linux-gnu.
[BZ #23178]
* nscd/nscd-client.h (sendfileall): Remove prototype.
* nscd/connections.c [HAVE_SENDFILE] (sendfileall): Remove function.
(handle_request): Use writeall instead of sendfileall.
* nscd/aicache.c (addhstaiX): Likewise.
* nscd/grpcache.c (cache_addgr): Likewise.
* nscd/hstcache.c (cache_addhst): Likewise.
* nscd/initgrcache.c (addinitgroupsX): Likewise.
* nscd/netgroupcache.c (addgetnetgrentX, addinnetgrX): Likewise.
* nscd/pwdcache.c (cache_addpw): Likewise.
* nscd/servicescache.c (cache_addserv): Likewise.
* sysdeps/unix/sysv/linux/Makefile [$(subdir) == nscd]
(sysdep-CFLAGS): Remove -DHAVE_SENDFILE.
* sysdeps/unix/sysv/linux/kernel-features.h (__ASSUME_SENDFILE):
Remove define.
[1] http://man7.org/linux/man-pages/man2/sendfile.2.html
[2] 7b6a329977 (diff-efd6af3a70f0f07c578e85b51e83b3c3)
Contributed by
Agustina Arzille <avarzille@riseup.net>
Amos Jeffries <squid3@treenet.co.nz>
David Michael <fedora.dm0@gmail.com>
Marco Gerards <marco@gnu.org>
Marcus Brinkmann <marcus@gnu.org>
Neal H. Walfield <neal@gnu.org>
Pino Toscano <toscano.pino@tiscali.it>
Richard Braun <rbraun@sceen.net>
Roland McGrath <roland@gnu.org>
Samuel Thibault <samuel.thibault@ens-lyon.org>
Thomas DiModica <ricinwich@yahoo.com>
Thomas Schwinge <tschwinge@gnu.org>
* htl: New directory.
* sysdeps/htl: New directory.
* sysdeps/hurd/htl: New directory.
* sysdeps/i386/htl: New directory.
* sysdeps/mach/htl: New directory.
* sysdeps/mach/hurd/htl: New directory.
* sysdeps/mach/hurd/i386/htl: New directory.
* nscd/Depend, resolv/Depend, rt/Depend: Add htl dependency.
* sysdeps/mach/hurd/i386/Implies: Add mach/hurd/i386/htl imply.
* sysdeps/mach/hurd/i386/libpthread.abilist: New file.
Unlike other nscd caches, the netgroup cache contains two types of
records - those for "iterate through a netgroup" (i.e. setnetgrent())
and those for "is this user in this netgroup" (i.e. innetgr()),
i.e. full and partial records. The timeout code assumes these records
have the same key for the group name, so that the collection of records
that is "this netgroup" can be expired as a unit.
However, the keys are not the same, as the in-netgroup key is generated
by nscd rather than being passed to it from elsewhere, and is generated
without the trailing NUL. All other keys have the trailing NUL, and as
noted in the linked BZ, debug statements confirm that two keys for the
same netgroup are added to the cache with two different lengths.
The result of this is that as records in the cache expire, the purge
code only cleans out one of the two types of entries, resulting in
stale, possibly incorrect, and possibly inconsistent cache data.
The patch simply includes the existing NUL in the computation for the
key length ('key' points to the char after the NUL, and 'group' to the
first char of the group, so 'key-group' includes the first char to the
NUL, inclusive).
[BZ #22342]
* nscd/netgroupcache.c (addinnetgrX): Include trailing NUL in
key value.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP is Linux-only.
* nscd/connections.c (RWLOCK_INITIALIZER): Define to
PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP or
PTHREAD_RWLOCK_INITIALIZER if that is not available.
(dbs): Use RWLOCK_INITIALIZER instead of
PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP.
400669754d ('hurd: Fix nscd build') had the side effect of making
libc's freeaddrinfo expose freeifaddrs through __check_pf. We can just
move the renames to gai.c itself, along others.
* sysdeps/mach/hurd/check_pf.c (__getifaddrs, __freeifaddrs): Do not
define macros.
* nscd/gai.c (__getifaddrs): Define macro to getifaddrs.
(__freeifaddrs): Define macro to freeifaddrs.
Current GCC mainline detects that nscd calls readlink with the same
buffer for both input and output, which is not valid (those arguments
are both restrict-qualified in POSIX). This patch makes it use a
separate buffer for readlink's input (with a size that is sufficient
to avoid truncation, so there should be no problems with warnings
about possible truncation, though not strictly minimal, but much
smaller than the buffer for output) to avoid this problem.
Tested compilation for aarch64-linux-gnu with build-many-glibcs.py.
[BZ #22446]
* nscd/connections.c (handle_request) [SO_PEERCRED]: Use separate
buffers for readlink input and output.
Hide internal __nis_hash function to allow direct access within libc.so
and libc.a without using GOT nor PLT.
[BZ #18822]
* nscd/nscd_helper.c (__nis_hash): New prototype.
[BZ #22161]
* nscd/netgroupcache.c (addinnetgrX): Release read lock after
resetting timeout.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Andreas Schwab <schwab@suse.de>
A lock is held by mempool_allocate() when CACHEABLE is true; we
must release this lock if we exit early.
Mark internal nss symbols with attribute_hidden to allow direct access
within libc.so and libc.a without using GOT nor PLT.
Tested on x86-64 with and without --disable-nscd.
[BZ #18822]
* grp/initgroups.c (__nss_group_database): Removed.
(__nss_initgroups_database): Likewise.
* nscd/gai.c (__nss_hosts_database): Likewise.
* nss/XXX-lookup.c (DATABASE_NAME_SYMBOL): Likewise.
* posix/tst-rfc3484-2.c (__nss_hosts_database): Likewise.
* posix/tst-rfc3484-3.c (__nss_hosts_database): Likewise.
* posix/tst-rfc3484.c (__nss_hosts_database): Likewise.
* sysdeps/posix/getaddrinfo.c (__nss_hosts_database): Likewise.
* nss/getXXent.c (INTERNAL (REENTRANT_GETNAME)): Add
attribute_hidden.
* nss/nsswitch.c (__nss_database_custom): Define only if
USE_NSCD is defined.
(__nss_configure_lookup): Use __nss_database_custom only if
USE_NSCD is defined.
* nss/nsswitch.h (__nss_database_custom): Declare only if
USE_NSCD is defined. Add attribute_hidden.
(__nss_setent): Add attribute_hidden.
(__nss_endent): Likewise.
(__nss_getent_r): Likewise.
(__nss_getent): Likewise.
(DEFINE_DATABASE): Declare __nss_##arg##_database.
struct resolv_context objects provide a temporary resolver context
which does not change during a name lookup operation. Only when the
outmost context is created, the stub resolver configuration is
verified to be current (at present, only against previous res_init
calls). Subsequent attempts to obtain the context will reuse the
result of the initial verification operation.
struct resolv_context can also be extended in the future to store
data which needs to be deallocated during thread cancellation.
Many callers of __res_maybe_init also call _res_hconf_init.
Additional calls to the latter do not hurt because the function
does its work only once. (/etc/hosts.conf is not reloaded or
even checked for changes.) This means that we can simplify the
code by calling _res_hconf_init directly from __res_vinit.
cppflags-iterator.mk no longer has anything to do with CPPFLAGS; all
it does is set libof-$(foo) for a list of files. extra-modules.mk
does the same thing, but with a different input variable, and doesn't
let the caller control the module. Therefore, this patch gives
cppflags-iterator.mk a better name, removes extra-modules.mk, and
updates all uses of both.
* extra-modules.mk: Delete file.
* cppflags-iterator.mk: Rename to ...
* libof-iterator.mk: ...this. Adjust comments.
* Makerules, extra-lib.mk, benchtests/Makefile, elf/Makefile
* elf/rtld-Rules, iconv/Makefile, locale/Makefile, malloc/Makefile
* nscd/Makefile, sunrpc/Makefile, sysdeps/s390/Makefile:
Use libof-iterator.mk instead of cppflags-iterator.mk or
extra-modules.mk.
* benchtests/strcoll-inputs/filelist#en_US.UTF-8: Remove
extra-modules.mk and cppflags-iterator.mk, add libof-iterator.mk.
Simplify the Linux accept4 implementation based on the assumption
that it is available in some way. __ASSUME_ACCEPT4_SOCKETCALL was
previously unused, so remove it.
For ia64, the accept4 system call (and socket call) were backported
in kernel version 3.2.18. Reflect this in the installation
instructions.
posix/wordexp-test.c used libc-internal.h for PTR_ALIGN_DOWN; similar
to what was done with libc-diag.h, I have split the definitions of
cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and PTR_ALIGN_DOWN
to a new header, libc-pointer-arith.h.
It then occurred to me that the remaining declarations in libc-internal.h
are mostly to do with early initialization, and probably most of the
files including it, even in the core code, don't need it anymore. Indeed,
only 19 files actually need what remains of libc-internal.h. 23 others
need libc-diag.h instead, and 12 need libc-pointer-arith.h instead.
No file needs more than one of them, and 16 don't need any of them!
So, with this patch, libc-internal.h stops including libc-diag.h as
well as losing the pointer arithmetic macros, and all including files
are adjusted.
* include/libc-pointer-arith.h: New file. Define
cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and
PTR_ALIGN_DOWN here.
* include/libc-internal.h: Definitions of above macros
moved from here. Don't include libc-diag.h anymore either.
* posix/wordexp-test.c: Include stdint.h and libc-pointer-arith.h.
Don't include libc-internal.h.
* debug/pcprofile.c, elf/dl-tunables.c, elf/soinit.c, io/openat.c
* io/openat64.c, misc/ptrace.c, nptl/pthread_clock_gettime.c
* nptl/pthread_clock_settime.c, nptl/pthread_cond_common.c
* string/strcoll_l.c, sysdeps/nacl/brk.c
* sysdeps/unix/clock_settime.c
* sysdeps/unix/sysv/linux/i386/get_clockfreq.c
* sysdeps/unix/sysv/linux/ia64/get_clockfreq.c
* sysdeps/unix/sysv/linux/powerpc/get_clockfreq.c
* sysdeps/unix/sysv/linux/sparc/sparc64/get_clockfreq.c:
Don't include libc-internal.h.
* elf/get-dynamic-info.h, iconv/loop.c
* iconvdata/iso-2022-cn-ext.c, locale/weight.h, locale/weightwc.h
* misc/reboot.c, nis/nis_table.c, nptl_db/thread_dbP.h
* nscd/connections.c, resolv/res_send.c, soft-fp/fmadf4.c
* soft-fp/fmasf4.c, soft-fp/fmatf4.c, stdio-common/vfscanf.c
* sysdeps/ieee754/dbl-64/e_lgamma_r.c
* sysdeps/ieee754/dbl-64/k_rem_pio2.c
* sysdeps/ieee754/flt-32/e_lgammaf_r.c
* sysdeps/ieee754/flt-32/k_rem_pio2f.c
* sysdeps/ieee754/ldbl-128/k_tanl.c
* sysdeps/ieee754/ldbl-128ibm/k_tanl.c
* sysdeps/ieee754/ldbl-96/e_lgammal_r.c
* sysdeps/ieee754/ldbl-96/k_tanl.c, sysdeps/nptl/futex-internal.h:
Include libc-diag.h instead of libc-internal.h.
* elf/dl-load.c, elf/dl-reloc.c, locale/programs/locarchive.c
* nptl/nptl-init.c, string/strcspn.c, string/strspn.c
* malloc/malloc.c, sysdeps/i386/nptl/tls.h
* sysdeps/nacl/dl-map-segments.h, sysdeps/x86_64/atomic-machine.h
* sysdeps/unix/sysv/linux/spawni.c
* sysdeps/x86_64/nptl/tls.h:
Include libc-pointer-arith.h instead of libc-internal.h.
* elf/get-dynamic-info.h, sysdeps/nacl/dl-map-segments.h
* sysdeps/x86_64/atomic-machine.h:
Add multiple include guard.
