atomic_compare_and_exchange_bool_rel and
catomic_compare_and_exchange_bool_rel are removed and replaced with the
new C11-like atomic_compare_exchange_weak_release. The concurrent code
in nscd/cache.c has not been reviewed yet, so this patch does not add
detailed comments.
* nscd/cache.c (cache_add): Use new C11-like atomic operation instead
of atomic_compare_and_exchange_bool_rel.
* nptl/pthread_mutex_unlock.c (__pthread_mutex_unlock_full): Likewise.
* include/atomic.h (atomic_compare_and_exchange_bool_rel,
catomic_compare_and_exchange_bool_rel): Remove.
* sysdeps/aarch64/atomic-machine.h
(atomic_compare_and_exchange_bool_rel): Likewise.
* sysdeps/alpha/atomic-machine.h
(atomic_compare_and_exchange_bool_rel): Likewise.
* sysdeps/arm/atomic-machine.h
(atomic_compare_and_exchange_bool_rel): Likewise.
* sysdeps/mips/atomic-machine.h
(atomic_compare_and_exchange_bool_rel): Likewise.
* sysdeps/tile/atomic-machine.h
(atomic_compare_and_exchange_bool_rel): Likewise.
If a GETxxBYyy request (for passwd or group) is running in parallel to
an INVALIDATE request (for the same database) then in a particular order
of events the garbage collector is not properly marking all used memory
and fails an assertion:
GETGRBYNAME (root)
Haven't found "root" in group cache!
add new entry "root" of type GETGRBYNAME for group to cache (first)
handle_request: request received (Version = 2) from PID 7413
INVALIDATE (group)
pruning group cache; time 9223372036854775807
considering GETGRBYNAME entry "root", timeout 1456763027
add new entry "0" of type GETGRBYGID for group to cache
remove GETGRBYNAME entry "root"
nscd: mem.c:403: gc: Assertion `next_data == &he_data[db->head->nentries]' failed.
Here the first call to cache_add added the GETGRBYNAME entry, which is
immediately marked for collection by prune_cache. Then the GETGRBYGID
entry is added which shares the data packet with the first entry and
therefore is marked as !first, while the marking look in prune_cache has
already finished. When the garbage collector runs, it only considers
references by entries marked as first, missing the reference by the
secondary entry.
The only way to fix that is to prevent prune_cache from running while the
two related entries are added.
https://sourceware.org/glibc/wiki/Proposals/GroupMerging
== Justification ==
It is common today for users to rely on centrally-managed user stores for
handling their user accounts. However, much software existing today does
not have an innate understanding of such accounts. Instead, they commonly
rely on membership in known groups for managing access-control (for
example the "wheel" group on Fedora and RHEL systems or the "adm" group
on Debian-derived systems). In the present incarnation of nsswitch, the
only way to have such groups managed by a remote user store such as
FreeIPA or Active Directory would be to manually remove the groups from
/etc/group on the clients so that nsswitch would then move past nss_files
and into the SSSD, nss-ldap or other remote user database.
== Solution ==
With this patch, a new action is introduced for nsswitch:
NSS_ACTION_MERGE. To take advantage of it, one will add [SUCCESS=merge]
between two database entries in the nsswitch.conf file. When a group is
located in the first of the two group entries, processing will continue
on to the next one. If the group is also found in the next entry (and the
group name and GID are an exact match), the member list of the second
entry will be added to the group object to be returned.
== Implementation ==
After each DL_LOOKUP_FN() returns, the next action is checked. If the
function returned NSS_STATUS_SUCCESS and the next action is
NSS_ACTION_MERGE, a copy of the result buffer is saved for the next pass
through the loop. If on this next pass through the loop the database
returns another instance of a group matching both the group name and GID,
the member list is added to the previous list and it is returned as a
single object. If the following database does not contain the same group,
then the original is copied back into the destination buffer.
This patch implements merge functionality only for the group database.
For other databases, there is a default implementation that will return
the EINVAL errno if a merge is requested. The merge functionality can be
implemented for other databases at a later time if such is needed. Each
database must provide a unique implementation of the deep-copy and merge
functions.
If [SUCCESS=merge] is present in nsswitch.conf for a glibc version that
does not support it, glibc will process results up until that operation,
at which time it will return results if it has found them or else will
simply return an error. In practical terms, this ends up behaving like
the remainder of the nsswitch.conf line does not exist.
== Iterators ==
This feature does not modify the iterator functionality from its current
behavior. If getgrnam() or getgrgid() is called, glibc will iterate
through all entries in the `group` line in nsswitch.conf and display the
list of members without attempting to merge them. This is consistent with
the behavior of nss_files where if two separate lines are specified for
the same group in /etc/groups, getgrnam()/getgrgid() will display both.
Clients are already expected to handle this gracefully.
== No Premature Optimizations ==
The following is a list of places that might be eligible for
optimization, but were not overengineered for this initial contribution:
* Any situation where a merge may occur will result in one malloc() of
the same size as the input buffer.
* Any situation where a merge does occur will result in a second
malloc() to hold the list of pointers to member name strings.
* The list of members is simply concatenated together and is not tested
for uniqueness (which is identical to the behavior for nss_files,
which will simply return identical values if they both exist on the
line in the file. This could potentially be optimized to reduce space
usage in the buffer, but it is both complex and computationally
expensive to do so.
== Testing ==
I performed testing by running the getent utility against my newly-built
glibc and configuring /etc/nsswitch.conf with the following entry:
group: group: files [SUCCESS=merge] sss
In /etc/group I included the line:
wheel❌10:sgallagh
I then configured my local SSSD using the id_provider=local to respond
with:
wheel:*:10:localuser,localuser2
I then ran `getent group wheel` against the newly-built glibc in
multiple situations and received the expected output as described
above:
* When SSSD was running.
* When SSSD was configured in nsswitch.conf but the daemon was not
running.
* When SSSD was configured in nsswitch.conf but nss_sss.so.2 was not
installed on the system.
* When the order of 'sss' and 'files' was reversed.
* All of the above with the [SUCCESS=merge] removed (to ensure no
regressions).
* All of the above with `getent group 10`.
* All of the above with `getent group` with and without
`enumerate=true` set in SSSD.
* All of the above with and without nscd enabled on the system.
With gcc-4.9, a new -fstack-protector-strong flag is available that is
between -fstack-protector (pretty weak) and -fstack-protector-all (pretty
strong) that provides good trade-offs between overhead but still providing
good coverage. Update the places in glibc that use ssp to use this flag
when it's available.
This also kills off the indirection of hardcoding the flag name in the
Makefiles and adding it based on a have-ssp boolean. Instead, the build
always expands the $(stack-protector) variable to the best ssp setting.
This makes the build logic a bit simpler and allows people to easily set
to a diff flag like:
make stack-protector=-fstack-protector-all
Building nscd w/selinux enabled yields a warning which yields an error:
In file included from selinux.c:32:0:
/usr/include/selinux/flask.h:5:2: error:
#warning "Please remove any #include's of this header in your source code."
I've done just that and it builds cleanly with libselinux-2.4.
This patch combines BUSY_WAIT_NOP and atomic_delay into a new
atomic_spin_nop function and adjusts all clients. The new function is
put into atomic.h because what is best done in a spin loop is
architecture-specific, and atomics must be used for spinning. The
function name is meant to tell users that this has no effect on
synchronization semantics but is a performance aid for spinning.
* sysdeps/mach/hurd/Makefile ($(common-objpfx)errnos.d): Depend on
libc-modules.h
* sysdeps/mach/hurd/i386/trampoline.c (_hurd_setup_sighandler): Remove
unused declaration of _hurd_intr_rpc_msg_in_trap.
* mach/mach_init.c (__mach_init): Test whether HAVE_HOST_PAGE_SIZE is
defined instead of whether it is non-zero.
* sysdeps/mach/hurd/i386/intr-msg.h (INTR_MSG_TRAP): Use "+m"
input constraint instead of both input and output constraint. Use ecx
clobber instead of %ecx.
* sysdeps/mach/hurd/malloc-machine.h (mutex_init, mutex_lock,
mutex_unlock): Use a statement expression instead of an expression list.
* sysdeps/mach/hurd/setitimer.c (_hurd_itimer_thread_stack_size): Set
type to vm_size_t instead of vm_address_t.
* sysdeps/mach/hurd/fork.c (__fork): Test whether STACK_GROWTH_UP is
defined instead of whether it is non-zero.
* hurd/hurd/ioctl.h (_hurd_locked_install_cttyid): New declaration.
* sysdeps/mach/hurd/setsid.c: Include <hurd/ioctl.h>.
* sysdeps/mach/hurd/mmap.c (__mmap): Use 0 instead of NULL for
comparisons with mapaddr.
* nscd/nscd-client.h: Include <time.h>.
* sysdeps/mach/hurd/dl-sysdep.c (fmh): Pass vm_offset_t dummy
9th parameter to __vm_region instead of int.
In bug 14906 the user complains that the inotify support in nscd
is not sufficient when it comes to detecting changes in the
configurationfiles that should be watched for the various databases.
The current nscd implementation uses inotify to watch for changes in
the configuration files, but adds watches only for IN_DELETE_SELF and
IN_MODIFY. These watches are insufficient to cover even the most basic
uses by a system administrator. For example using emacs or vim to edit
a configuration file should trigger a reload but it might not if
the editors use move to atomically update the file. This atomic update
changes the inode and thus removes the notification on the file (as
inotify is based on inodes). Thus the inotify support in nscd for
configuration files is insufficient to account for the average use
cases of system administrators and users.
The inotify support is significantly enhanced and described here:
https://www.sourceware.org/ml/libc-alpha/2015-02/msg00504.html
Tested on x86_64 with and without inotify support.
The padding bytes in the statsdata struct are not initialized, due to
which valgrind throws a warning:
==11384== Memcheck, a memory error detector
==11384== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==11384== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==11384== Command: nscd -d
==11384==
Fri 25 Apr 2014 10:34:53 AM CEST - 11384: handle_request: request received (Version = 2) from PID 11396
Fri 25 Apr 2014 10:34:53 AM CEST - 11384: GETSTAT
==11384== Thread 6:
==11384== Syscall param socketcall.sendto(msg) points to uninitialised byte(s)
==11384== at 0x4E4ACDC: send (in /lib64/libpthread-2.12.so)
==11384== by 0x11AF6B: send_stats (in /usr/sbin/nscd)
==11384== by 0x112F75: nscd_run_worker (in /usr/sbin/nscd)
==11384== by 0x4E439D0: start_thread (in /lib64/libpthread-2.12.so)
==11384== by 0x599AB6C: clone (in /lib64/libc-2.12.so)
==11384== Address 0x15708395 is on thread 6's stack
Fix the warning by initializing the structure.
This patch avoids warnings for unused results of setuid and setgid in
nscd/connections.c using an ignore_value macro along the lines
suggested by Paul in
<https://sourceware.org/ml/libc-alpha/2014-11/msg00733.html>.
Tested for x86_64.
* include/libc-internal.h (ignore_value): New macro.
* nscd/connections.c (restart): Wrap calls to setuid and setgid
with ignore_value.
The current scheme to identify which module a translation unit is
built in depends on defining multiple macros IS_IN_* and also defining
NOT_IN_libc if we're building a non-libc module. In addition, there
is an IN_LIB macro that does effectively the same thing, but for
different modules (notably the systemtap probes). This macro scheme
unifies both ideas to use just one macro IN_MODULE and assign it a
value depending on the module it is being built into. If the module
is not defined, it defaults to MODULE_libc.
Patches that follow will replace uses of IS_IN_* variables with the
IS_IN() macro. libc-symbols.h has been converted already to give an
example of how such a transition will look.
Verified that there are no relevant binary changes. One source change
that will crop up repeatedly is that of nscd_stat, since it uses the
build timestamp as a constant in its logic.
* Makeconfig (in-module): Get value of libof set for the
translation unit.
(CPPFLAGS): Use $(in-module).
* Makerules: Don't suffix routine names for nonlib.
* include/libc-modules.h: New file.
* include/libc-symbols.h: Include libc-modules.h
(IS_IN): New macro to replace IS_IN_* macros.
* elf/Makefile: Set libof-* for each routine.
* elf/rtld-Rules: Likewise.
* extra-modules.mk: Likewise.
* iconv/Makefile: Likewise.
* iconvdata/Makefile: Likewise.
* locale/Makefile: Likewise.
* malloc/Makefile: Likewise.
* nss/Makefile: Likewise.
* sysdeps/gnu/Makefile: Likewise.
* sysdeps/ieee754/ldbl-opt/Makefile: Likewise.
* sysdeps/unix/sysv/linux/Makefile: Likewise.
* sysdeps/s390/s390-64/Makefile: Likewise.
* nscd/Makefile: Set libof-* for each routine. Set CFLAGS and
CPPFLAGS for nscd instead of nonlib.
qsort_r is defined in the same file as qsort, but is not an ISO C
function, so should be a weak alias for __qsort_r. The uses in
getaddrinfo should also call __qsort_r, since getaddrinfo is a POSIX
function and qsort_r isn't. This patch implements this. Because nscd
uses the getaddrinfo sources outside libc, as do the tst-rfc3484
tests, a #define of __qsort_r to qsort_r is added there alongside the
similar defines for other libc-internal symbols used in getaddrinfo.
Tested for x86_64 (testsuite, and that disassembly of installed shared
libraries is unchanged by the patch).
[BZ #17571]
* stdlib/msort.c (qsort_r): Rename to __qsort_r and define as weak
alias of __qsort_r.
(qsort): Call __qsort_r instead of qsort_r.
* include/stdlib.h (qsort_r): Do not call libc_hidden_proto.
(__qsort_r): Declare. Call libc_hidden_proto.
* sysdeps/posix/getaddrinfo.c (getaddrinfo): Call __qsort_r
instead of qsort_r.
* nscd/gai.c (__qsort_r): Define to qsort_r.
* posix/tst-rfc3484.c (__qsort_r): Likewise.
* posix/tst-rfc3484-2.c (__qsort_r): Likewise.
* posix/tst-rfc3484-3.c (__qsort_r): Likewise.
The nscd parent process returns the result of a `wait' call rather
than the exit status of the child it waits for. These two aren't
exactly the same. In my case (and probably on most machines), the exit
status is in the 2nd LSB of the result of `wait', and so:
e.g. if the nscd child process returns 1, the parent returns 1 << 8,
which Bash happily reports as 0.
The attached patch removed the unused ‘thread_info_t’ typedef and the
‘thread_info’ variable from nscd.c. The former conflicts with a GNU Mach
typedef, and the latter conflicts with a GNU Mach function declaration:
<https://lists.gnu.org/archive/html/bug-hurd/2014-06/msg00101.html>.
Tested on x86_64-linux-gnu.
This patch makes files using __ASSUME_* macros include
<kernel-features.h> explicitly, rather than relying on some other
header (such as tls.h, lowlevellock.h or pthreadP.h) to include it
implicitly. (I omitted cases where I've already posted or am testing
the patch that stops the file from needing __ASSUME_* at all.) This
accords with the general principle of making source files include the
headers for anything they use, and also helps make it safe to remove
<kernel-features.h> includes from any file that doesn't use
__ASSUME_* (some of those may be stray includes left behind after
increasing the minimum kernel version, others may never have been
needed or may have become obsolete after some other change).
Tested x86_64 that the disassembly of installed shared libraries is
unchanged by this patch.
* nptl/pthread_cond_wait.c: Include <kernel-features.h>.
* nptl/pthread_rwlock_timedrdlock.c: Likewise.
* nptl/pthread_rwlock_timedwrlock.c: Likewise.
* nptl/sysdeps/unix/sysv/linux/lowlevelrobustlock.c: Likewise.
* nscd/nscd.c: Likewise.
* sysdeps/i386/nptl/tcb-offsets.sym: Likewise.
* sysdeps/powerpc/nptl/tcb-offsets.sym: Likewise.
* sysdeps/sh/nptl/tcb-offsets.sym: Likewise.
* sysdeps/x86_64/nptl/tcb-offsets.sym: Likewise.
The netgroups nss modules in the glibc tree use NSS_STATUS_UNAVAIL
(with errno as ERANGE) when the supplied buffer does not have
sufficient space for the result. This is wrong, because the canonical
way to indicate insufficient buffer is to set the errno to ERANGE and
the status to NSS_STATUS_TRYAGAIN, as is used by all other modules.
This fixes nscd behaviour when the nss_ldap module returns
NSS_STATUS_TRYAGAIN to indicate that a netgroup entry is too long to
fit into the supplied buffer.
I noticed that some of the Depend files, used to determine the
subdirectory build order in sysd-sorted, still mentioned linuxthreads,
although it hasn't been supported for many years. This patch removes
those references. In the case of nscd, it substitutes an nptl
reference, since I believe there is a fact a thread library dependence
there; the others already mentioned nptl.
Note that I am not at all confident in the completeness of these
Depend files.
Note also that references to linuxthreads remain in a comment in
sysdeps/unix/sysv/linux/ia64/Versions, and in manual/maint.texi,
manual/signal.texi and scripts/documented.sh.
Tested x86_64 that the installed shared libraries are unchanged by the
patch (as is sysd-sorted).
* nscd/Depend (linuxthreads): Remove.
(nptl): Add.
* resolv/Depend (linuxthreads): Remove.
* rt/Depend (linuxthreads): Remove.
The datahead structure has an unused padding field that remains
uninitialized. Valgrind prints out a warning for it on querying a
netgroups entry. This is harmless, but is a potential data leak since
it would result in writing out an uninitialized byte to the cache
file. Besides, this happens only when there is a cache miss, so we're
not adding computation to any fast path.
This patch consolidates the code to initialize the header of a dataset
into a single set of functions (one for positive and another for
negative datasets) primarily to reduce repetition of code. The
secondary reason is to simplify Patch 2/2 which fixes the problem of
an uninitialized byte in the header by initializing an unused field in
the structure and hence preventing a possible data leak into the cache
file.
The SELinux team has indicated to me that glibc's SELinux checks
in nscd are not being carried out as they would expect the API
to be used today. They would like to move away from static header
defines for class and permissions and instead use dynamic checks
at runtime that provide an answer which is dependent on the runtime
status of SELinux i.e. more dynamic.
The following patch is a minimal change that moves us forward in
this direction.
It does the following:
* Stop checking for SELinux headers that define NSCD__SHMEMHOST.
Check only for the presence or absence of the library.
* Don't encode the specific SELinux permission constants into a
table at build time, and instead use the symbolic name for the
permission as expected.
* Lookup the "What do we do if we don't know this permission?"
policy and use that if we find SELinux's policy is older than
the glibc policy e.g. we make a request for a permission that
SELinux doesn't know about.
* Lastly, translate the class and permission and then make
the permission check. This is done every time we lookup
a permission, and this is the expected way to use the API.
SELinux will optimize this for us, and we expect the network
latencies to hide these extra library calls.
Tested on x86, x86-64, and via Fedora Rawhide since November 2013.
See:
https://sourceware.org/ml/libc-alpha/2014-04/msg00179.html
This patch defines _STRING_ARCH_unaligned to 0 on default bits/string.h
header to avoid undefined compiler warnings on platforms that do not
define it. It also make adjustments in code where tests checked if macro
existed or not.
Calls to stpcpy from nscd netgroups code will have overlapping source
and destination when all three values in the returned triplet are
non-NULL and in the expected (host,user,domain) order. This is seen
in valgrind as:
==3181== Source and destination overlap in stpcpy(0x19973b48, 0x19973b48)
==3181== at 0x4C2F30A: stpcpy (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==3181== by 0x12567A: addgetnetgrentX (string3.h:111)
==3181== by 0x12722D: addgetnetgrent (netgroupcache.c:665)
==3181== by 0x11114C: nscd_run_worker (connections.c:1338)
==3181== by 0x4E3C102: start_thread (pthread_create.c:309)
==3181== by 0x59B81AC: clone (clone.S:111)
==3181==
Fix this by using memmove instead of stpcpy.
nscd works correctly when the request in innetgr is a wildcard,
i.e. when one or more of host, user or domain parameters is NULL.
However, it does not work when the the triplet in the netgroup
definition has a wildcard. This is easy to reproduce for a triplet
defined as follows:
foonet (,foo,)
Here, an innetgr call that looks like this:
innetgr ("foonet", "foohost", "foo", NULL);
should succeed and so should:
innetgr ("foonet", NULL, "foo", "foodomain");
It does succeed with nscd disabled, but not with nscd enabled. This
fix adds this additional check for all three parts of the triplet so
that it gives the correct result.
[BZ #16758]
* nscd/netgroupcache.c (addinnetgrX): Succeed if triplet has
blank values.
The buffer to query netgroup entries is allocated sufficient space for
the netgroup entries and the key to be appended at the end, but it
sends in an incorrect available length to the NSS netgroup query
functions, resulting in overflow of the buffer in some special cases.
The fix here is to factor in the key length when sending the available
buffer and buffer length to the query functions.
Currently the nscd service is installed in systemd as a simple
service, which means that it is able to handle its own errors and does
not quit. Since nscd does not fit that description, i.e. it can exit
on errors like, say, failing to parse nscd.conf, it should be declared
as forking instead.
Currently, the nscd parent process parses commandline options and
configuration, forks on startup and immediately exits with a success.
If the child process encounters some error after this, it goes
undetected and any services started up after it may have to repeatedly
check to make sure that the nscd service did actually start up and is
serving requests.
To make this process more reliable, I have added a pipe between the
parent and child process, through which the child process sends a
notification to the parent informing it of its status. The parent
waits for this status and once it receives it, exits with the
corresponding exit code. So if the child service sends a success
status (0), the parent exits with a success status. Similarly for
error conditions, the child sends the non-zero status code, which the
parent passes on as the exit code.
This, along with setting the nscd service type to forking in its
systemd configuration file, allows systemd to be certain that the nscd
service is ready and is accepting connections.
The _nss_*_getnetgrent_r query populates the netgroup results in the
allocated buffer and then sets the result triplet to point to strings
in the buffer. This is a problem when the buffer is reallocated since
the pointers to the triplet strings are no longer valid. The pointers
need to be adjusted so that they now point to strings in the
reallocated buffer.
addgetnetgrentX has a buffer which is grown as per the needs of the
requested size either by using alloca or by falling back to malloc if
the size is larger than 1K. There are two problems with the alloca
bits: firstly, it doesn't really extend the buffer since it does not
use the return value of the extend_alloca macro, which is the location
of the reallocated buffer. Due to this the buffer does not actually
extend itself and hence a subsequent write may overwrite stuff on the
stack.
The second problem is more subtle - the buffer growth on the stack is
discontinuous due to block scope local variables. Combine that with
the fact that unlike realloc, extend_alloca does not copy over old
content and you have a situation where the buffer just has garbage in
the space where it should have had data.
This could have been fixed by adding code to copy over old data
whenever we call extend_alloca, but it seems unnecessarily
complicated. This code is not exactly a performance hotspot (it's
called when there is a cache miss, so factors like network lookup or
file reads will dominate over memory allocation/reallocation), so this
premature optimization is unnecessary.
Thanks Brad Hubbard <bhubbard@redhat.com> for his help with debugging
the problem.
Usage output for option --invalidate=TABLE is not helpful without
list of tables. The list is also missing from nscd(8) manual which
made it pretty difficult to know what are the tables.
