I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 7061 files FOO.
I then removed trailing white space from math/tgmath.h,
support/tst-support-open-dev-null-range.c, and
sysdeps/x86_64/multiarch/strlen-vec.S, to work around the following
obscure pre-commit check failure diagnostics from Savannah. I don't
know why I run into these diagnostics whereas others evidently do not.
remote: *** 912-#endif
remote: *** 913:
remote: *** 914-
remote: *** error: lines with trailing whitespace found
...
remote: *** error: sysdeps/unix/sysv/linux/statx_cp.c: trailing lines
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date. Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.
Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions. These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.
The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively. These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:
https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dchttps://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
This is a major rewrite of the description of 'crypt', 'getentropy',
and 'getrandom'.
A few highlights of the content changes:
- Throughout the manual, public headers, and user-visible messages,
I replaced the term "password" with "passphrase", the term
"password database" with "user database", and the term
"encrypt(ion)" with "(one-way) hashing" whenever it was applied to
passphrases. I didn't bother making this change in internal code
or tests. The use of the term "password" in ruserpass.c survives,
because that refers to a keyword in netrc files, but it is adjusted
to make this clearer.
There is a note in crypt.texi explaining that they were
traditionally called passwords but single words are not good enough
anymore, and a note in users.texi explaining that actual passphrase
hashes are found in a "shadow" database nowadays.
- There is a new short introduction to the "Cryptographic Functions"
section, explaining how we do not intend to be a general-purpose
cryptography library, and cautioning that there _are_, or have
been, legal restrictions on the use of cryptography in many
countries, without getting into any kind of detail that we can't
promise to keep up to date.
- I added more detail about what a "one-way function" is, and why
they are used to obscure passphrases for storage. I removed the
paragraph saying that systems not connected to a network need no
user authentication, because that's a pretty rare situation
nowadays. (It still says "sometimes it is necessary" to
authenticate the user, though.)
- I added documentation for all of the hash functions that glibc
actually supports, but not for the additional hash functions
supported by libxcrypt. If we're going to keep this manual section
around after the transition is more advanced, it would probably
make sense to add them then.
- There is much more detailed discussion of how to generate a salt,
and the failure behavior for crypt is documented. (Returning an
invalid hash on failure is what libxcrypt does; Solar Designer's
notes say that this was done "for compatibility with old programs
that assume crypt can never fail".)
- As far as I can tell, the header 'crypt.h' is entirely a GNU
invention, and never existed on any other Unix lineage. The
function 'crypt', however, was in Issue 1 of the SVID and is now
in the XSI component of POSIX. I tried to make all of the
@standards annotations consistent with this, but I'm not sure I got
them perfectly right.
- The genpass.c example has been improved to use getentropy instead
of the current time to generate the salt, and to use a SHA-256 hash
instead of MD5. It uses more random bytes than is strictly
necessary because I didn't want to complicate the code with proper
base64 encoding.
- The testpass.c example has three hardwired hashes now, to
demonstrate that different one-way functions produce different
hashes for the same input. It also demonstrates how DES hashing
only pays attention to the first eight characters of the input.
- There is new text explaining in more detail how a CSPRNG differs
from a regular random number generator, and how
getentropy/getrandom are not exactly a CSPRNG. I tried not to make
specific falsifiable claims here. I also tried to make the
blocking/cancellation/error behavior of both getentropy and
getrandom clearer.
As indicated by BZ#23178, concurrent access on some files read by nscd
may result non expected data send through service requisition. This is
due 'sendfile' Linux implementation where for sockets with zero-copy
support, callers must ensure the transferred portions of the the file
reffered by input file descriptor remain unmodified until the reader
on the other end of socket has consumed the transferred data.
I could not find any explicit documentation stating this behaviour on
Linux kernel documentation. However man-pages sendfile entry [1] states
in NOTES the aforementioned remark. It was initially pushed on man-pages
with an explicit testcase [2] that shows changing the file used in
'sendfile' call prior the socket input data consumption results in
previous data being lost.
From commit message it stated on tested Linux version (3.15) only TCP
socket showed this issues, however on recent kernels (4.4) I noticed the
same behaviour for local sockets as well.
Since sendfile on HURD is a read/write operation and the underlying
issue on Linux, the straightforward fix is just remove sendfile use
altogether. I am really skeptical it is hitting some hotstop (there
are indication over internet that sendfile is helpfull only for large
files, more than 10kb) here to justify that extra code complexity or
to pursuit other possible fix (through memory or file locks for
instance, which I am not sure it is doable).
Checked on x86_64-linux-gnu.
[BZ #23178]
* nscd/nscd-client.h (sendfileall): Remove prototype.
* nscd/connections.c [HAVE_SENDFILE] (sendfileall): Remove function.
(handle_request): Use writeall instead of sendfileall.
* nscd/aicache.c (addhstaiX): Likewise.
* nscd/grpcache.c (cache_addgr): Likewise.
* nscd/hstcache.c (cache_addhst): Likewise.
* nscd/initgrcache.c (addinitgroupsX): Likewise.
* nscd/netgroupcache.c (addgetnetgrentX, addinnetgrX): Likewise.
* nscd/pwdcache.c (cache_addpw): Likewise.
* nscd/servicescache.c (cache_addserv): Likewise.
* sysdeps/unix/sysv/linux/Makefile [$(subdir) == nscd]
(sysdep-CFLAGS): Remove -DHAVE_SENDFILE.
* sysdeps/unix/sysv/linux/kernel-features.h (__ASSUME_SENDFILE):
Remove define.
[1] http://man7.org/linux/man-pages/man2/sendfile.2.html
[2] 7b6a329977 (diff-efd6af3a70f0f07c578e85b51e83b3c3)
If a GETxxBYyy request (for passwd or group) is running in parallel to
an INVALIDATE request (for the same database) then in a particular order
of events the garbage collector is not properly marking all used memory
and fails an assertion:
GETGRBYNAME (root)
Haven't found "root" in group cache!
add new entry "root" of type GETGRBYNAME for group to cache (first)
handle_request: request received (Version = 2) from PID 7413
INVALIDATE (group)
pruning group cache; time 9223372036854775807
considering GETGRBYNAME entry "root", timeout 1456763027
add new entry "0" of type GETGRBYGID for group to cache
remove GETGRBYNAME entry "root"
nscd: mem.c:403: gc: Assertion `next_data == &he_data[db->head->nentries]' failed.
Here the first call to cache_add added the GETGRBYNAME entry, which is
immediately marked for collection by prune_cache. Then the GETGRBYGID
entry is added which shares the data packet with the first entry and
therefore is marked as !first, while the marking look in prune_cache has
already finished. When the garbage collector runs, it only considers
references by entries marked as first, missing the reference by the
secondary entry.
The only way to fix that is to prevent prune_cache from running while the
two related entries are added.
This patch consolidates the code to initialize the header of a dataset
into a single set of functions (one for positive and another for
negative datasets) primarily to reduce repetition of code. The
secondary reason is to simplify Patch 2/2 which fixes the problem of
an uninitialized byte in the header by initializing an unused field in
the structure and hence preventing a possible data leak into the cache
file.
The nscd/*cache.c files contain assert()s, writeall() and sendfileall() calls
that invalidly use together &dataset->resp and total where either dataset or
dataset->head.recsize should be used instead one of the components. In the
writeall() and sendfileall() cases, it is unlikely to matter in practice, but
the assertions can fail sometimes without a proper reason.
The commit 20e498bd removes the pthread_mutex_rdlock() calls, but not the
corresponding pthread_mutex_unlock() calls. Also, the database lock is never
unlocked in one branch of the mempool_alloc() if.
I think unreproducible random assert(dh->usable) crashes in prune_cache() were
caused by this. But an easy way to make nscd threads hang with the broken
locking was.
2009-02-13 Ulrich Drepper <drepper@redhat.com>
[BZ #5381]
* nscd/nscd.h: Remove definitions and declarations for mem_in_flight.
Change mempool_alloc prototype.
* nscd/mem.c (gc): Don't handle mem_in_flight.
(mempool_alloc): Third parameter now only indicates whether this is the
first call (to allocate data) or not. If it is, get db rdlock.
Release it on error. Don't handle mem_in_flight.
* nscd/aicache.c (addhstaiX): Mark he parameter as const.
Adjust third parameter of mempool_alloc calls.
Nothing to do here in case mempool_alloc fails.
Avoid local variable shadowing parameter. No need to get db rdlock
before calling cache_add.
* nscd/cache.c (cache_add): Adjust call to mempool_alloc. There is
no mem_in_flight array anymore.
* nscd/connections.c: Remove definition and handling of mem_in_flight.
* nscd/grpcache.c (cache_addgr): Adjust third parameter of
mempool_alloc calls. Mark he parameter as const. Nothing to do here
in case mempool_alloc fails. No need to get db rdlock before calling
cache_add.
* nscd/hstcache.c (cache_addhst): Likewise.
* nscd/initgrcache.c (addinitgroupsX): Likewise.
* nscd/servicescache.c (cache_addserv): Likewise.
* nscd/pwdcache.c (cache_addpw): Likewise. Remove some debugging code.
whether this is in response of a cache refill. Check alignment
of package data. Revamp waking of pruning thread.
(prune_cache): Small optimization.
* nscd/nscd.h: Adjust cache_add prototypes.
* nscd/aicache.c: Adjust cache_add calls.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
* nscd/servicescache.c: Likewise.
* nscd/connections.c (restart): Really disable cache use before
exec attempt. If it fails, reenable cache.
(nscd_run_prune): Initialize wakeup_time. After wakeup, set wakeup
time to max to be able to notice concurrent cache additions. Unlock
prune_lock while performing gc. Afterwards compute wakeup time with
current wakeup_time value in mind.
is the first use of the record, mark it as unusable.
* nscd/aicache.c: Don't touch the dataset after cache_add returns
reporting a failure.
* nscd/grpcache.c: Likewise
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
* nscd/servicecache.c: Likewise.
2008-04-15 Ulrich Drepper <drepper@redhat.com>
[BZ #5209]
* sysdeps/unix/sysv/linux/times.c: New file.
[BZ #5381]
* nscd/nscd.h: Define enum in_flight, mem_in_flight, and
mem_in_flight_list variables. Add new parameter to mempool_alloc
prototype.
* nscd/mem.c (mempool_alloc): Take additional parameter. Initialize
appropriate mem_in_flight element.
(gc): Take allocations which have not yet been committed to the
database into account.
* nscd/cache.c (cache_add): Add new parameter to mempool_alloc call.
Reset mem_in_flight before returning.
* nscd/connections.c (nscd_run_worker): Initialize mem_in_flight and
cue it up in mem_in_flight_list.
* nscd/aicache.c: Adjust mempool_alloc call.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
* nscd/servicescache.c: Likewise.
* nscd/Makefile (nscd-flags): Until ld is fixed, use -fpic instead
of -fpie.
* nscd/connections.c (handle_request): Provide better error message
in case SELinux forbids the service.
* version.h (VERSION): Bump to 2.8.90.
* nscd/nscd_conf.c (nscd_parse_file): Parse auto-propagate lines.
* nscd/nscd.conf: Add auto-propagate lines.
* nscd/connections.c (dbs): Initialize .propagate fields.
* nscd/grpcache.c (cache_addgr): Do not add ID entry for name lookups
and vice versa if propagation is disabled for the database.
* nscd/pwdcache.c (cache_addpw): Likewise.
* ncsd/aicache.c: Don't use sendfile for records on the stack.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
Patch by dmueller@suse.com.
(struct shmid_ds): Use it for shm_nattch field.
2005-11-18 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/futimesat.c (futimesat): If FILE is NULL,
set access and modification times of the file referenced by FD.
* sysdeps/generic/futimesat.c (futimesat): Don't return EINVAL if
FILE is NULL. Don't check FD if FILE is absolute path.
2005-11-19 Ulrich Drepper <drepper@redhat.com>
* nscd/nscd_gethst_r.c (nscd_gethst_r): Avoid unnecesary read call
if there are no aliases.
* sysdeps/unix/sysv/linux/Makefile (CFLAGS-connections.c,
CFLAGS-pwdcache.c, CFLAGS-grpcache.c, CFLAGS-hstcache.c,
CFLAGS-aicache.c, CFLAGS-initgrcache.c): Add -DHAVE_SENDFILE.
* sysdeps/unix/sysv/linux/kernel-features.h (__ASSUME_SENDFILE):
Define.
* nscd/pwdcache.c [HAVE_SENDFILE]: Include <sys/sendfile.h> and
<kernel-features.h>.
[HAVE_SENDFILE] (cache_addpw): Use sendfile to transmit positive
result.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/aicache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/connectionc.c: Likewise.
(dbs): Initialize max_db_size fields.
(nscd_init): When mapping the database, use max_db_size as the
mapping size even if it is bigger than the file size.
* nscd/mem.c (mempool_alloc): When resizing the file make sure the
limit in max_db_size is not exceeded. Don't use mremap, just
posix_fallocate is enough (according to Linus). Use posix_fallocate
correctly.
* nscd/nscd.conf: Add max-db-size parameters.
* nscd/nscd.h (struct database_dyn): Add max_db_size field.
Define DEFAULT_MAX_DB_SIZE and DEFAULT_DATASIZE_PER_BUCKET.
Temporarily define TEMP_FAILURE_RETRY_VAL here.
* nscd/nscd_conf.c (nscd_parse_file): Parse max-db-size parameter
and add sanity checks for it.
* nscd/aicache.c (addhstaiX): Use send with MSG_NOSIGNAL not write to
send reply.
* nscd/connection.c (writeall): Likewise.
(handle_request): Likewise.
* nscd/grpcache.c (cache_addgr): Likewise.
* nscd/hstcache.c (cache_addhst): Likewise.
* nscd/initgrcache.c (addinitgroupsX): Likewise.
* nscd/nscd.c (parse_opt): Likewise.
* nscd/nscd_stat.c (send_stats): Likewise.
(receive_print_stats): Likewise.
* nscd/pwdcache.c (cache_addpw): Likewise.
NETDB_INTERNAL if buffer is too small.
* nscd/hstcache.c (INCR): Remove.
(addhstbyX): Double buflen in each iteration rather than add INCR.
* nscd/grpcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
