Remove the dependencies from the ICU library code on static constructors
that were introduced by using std::mutex and condition variables. The
mutexes are lazily initialized by embedding them as local static variables
in getter functions, and relying on the C++ compiler/runtime to do thread
safe initialization of them.
- see also ICU-20062
- add a `-B` option to the two python invocations on Windows
- set PYTHONDONTWRITEBYTECODE in configure.ac and icudefs.mk.in
Co-authored-by: Fredrik Roubert <roubert@google.com>
- If icu/source/data/locales/root.txt missing, skip
python rules.mk generation.
- Also, create build directories properly as needed
- Also includes noise changes to configure
(configure was probably generated using unreleased
autoconf 2.70 or 2.69 + patches)
- eac8f4b31a did not
regen configure properly, so BUILDTOOL_OPTS is now
ICU_DATA_BUILDTOOL_OPTS
- If icu/source/data/locales/root.txt missing, skip
python rules.mk generation.
- Also, create build directories properly as needed
- Also includes noise changes to configure
(configure was probably generated using unreleased
autoconf 2.70 or 2.69 + patches)
- eac8f4b31a did not
regen configure properly, so BUILDTOOL_OPTS is now
ICU_DATA_BUILDTOOL_OPTS
- Fixes filterrb.cpp to check for wildcard when at a leaf.
- Adds additional verbose logging to genrb.
- Fixes filtration to add deps to dep_targets instead of dep_files.
- Separates dep_files to common_dep_files and specific_dep_files.
- use git instead of svn for versioning
- update generated file names to match official downloads
Example filenames:
- icu4c-63_1-c5749aaf94-src.tgz (includes git hash or tag)
- icu4c-63_1-src.tgz (hard link to above)
- icu4c-src.tgz (symlink)
- icu4c-63_1-c5749aaf94-src.zip (includes git hash or tag)
- icu4c-63_1-src.zip (hard link to above)
- icu4c-src.zip (symlink)
- icu4c-63_1-c5749aaf94-data.zip (includes git hash or tag)
- icu4c-63_1-data.zip (hard link to above)
- icu4c-data.zip (symlink)
- icu4c-63_1-c5749aaf94-docs.zip (includes git hash or tag)
- icu4c-63_1-docs.zip (hard link to above)
- icu4c-docs.zip (symlink)
Methods implementead as 'inline' but not declared 'inline' cause clang++
to throw compilation warnings on Windows. This adds 'inline' to the
relevant method declarations.
This eliminates the need for scratch buffers in any code path that ends
with these functions and also eliminates the need for counting bytes,
something that ByteSink will now handle correctly when needed.
Existing calls to uloc_addLikelySubtags() and uloc_minimizeSubtags()
throughout ICU4C implementation code are also updated to instead use
either the Locale or ulocimp_* functions with the new API.
None of this should have any externally visible effect, it's all about
cleaning up implementation internals.
from assumed UTF-8 resulted in an extremely large percentage of Unicode
replacement characters in the data passed to the API under test.
ICU-20217 Uses fuzzer generated bytes to make random selection of locales, converters,
etc., replacing the random number generator. This way the fuzzer can control
the selections.
ICU-20217 Minor follow-ups from code review.
Removes fuzzer target break_iterator_utf32_fuzzer which does not perform
anything useful what the regular break iterator fuzzer target already performs.
ICU-20217 Fixes for-loop body.
ICU-20217 Uses am allocated buffer to pass head-truncated fuzzer data to the
API under test. The fuzzer may otherwise not detect buffer underflow.
by
ICU-20217 Typing fix.
ICU-20217 Fixing typing.
ICU-20217 Improve fuzzer targets, move truncated fuzzer data into a
new buffer to prevent that buffer underflow goes undetected.
ICU-20217 Fixes buffer management of fuzzer-provided data.
ICU-20217 Factor in PR review comments.
Cherry-pick cldrbug 11492: Bad symbols for NaN in sv, ksh, kl, se locales.
Cherry-pick cldrbug 11491: sd, month name for July uses character not in exemplars.
Updated the various ICU4J *.jar files as well.
- Changes Java DecimalFormat boolean get* methods to is*.
- Makes the new draft methods non-virtual.
- Removes obsolete template class in header file.
- Adds proper U_HIDE tags in unum.h and decimfmt.h
If a file with an input line larger than INT32_MAX (i.e. 2 GB) contains
an UTF8 character after that limit, escapesrc crashes on 64 bit systems
or does not remove incomplete files on 32 bit systems.
The issue is that an unchecked cast from size_t to int32_t can turn
negative, which results in negative offsets during array access.
This will eventually lead to an out of boundary read, which most likely
crashes the tool.
This patch sets a fixed limit on 1 GB to make sure that no side effects
occur if the line is exactly INT32_MAX or a few bytes less. It should
still be way more than anyone would really need.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
If gencnval encounters an empty input file the function resolveAlias
triggers an out of boundary write due to uniqueAliasArr pointing to
a 0 byte reserved memory address.
This patch protects the code in question with a check for
knownAliasesCount being not 0.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
- Adds first "span" field category
- Re-implements DateIntervalFormat#fallbackFormat to use FieldPositionHandler
- New temporary wiring in SimpleFormatter
- Adds additional logic to NumberStringBuilder.
- Extends logic of number::impl::Field type.
- Adds tests for RBNF support.
- Adds tests from ftang's original PR.
- CreateFileMappingW is marked for both desktop and UWP apps, so we can call that in both code paths.
- We can use the W version of the CreateFileMapping API instead of A version since we pass a NULL for the name anyways.
- We can call the same API CreateFile[A|W] from both the UWP and Win32 versions of the code, reducing one of the UWP forks.
- Add a work-around for older versions of the Windows 10 SDK UWP headers.
- Remove the code that was creating a custom security descriptor (but setting everything to NULL) and pass null to the API directly. This way we will get the default security descriptor instead of the NULL dacl.
- Change to use nullptr instead of NULL in C++ code.
- Use move assignment for fields->formatter (LocalizedNumberFormatter) instead of creating new heap object every time.
- Add test cases for DecimalFormat object in invalid state.
- Protect against self-assignment in assignment operator.
- Fix segmentation fault when attempting to compare valid and invalid DecimalFormat objects.
- Changes based on review feedback from Shane.
- Fix minor typos in the public header file.
The problem is that Docker receives zip files only as LFS links when
cloning ICU from GitHub. Converting the txt files into zip files, which
is the required corpus format for the fuzzer, will be done by the oss-fuzz
build script.
ICU-20217 Adds fuzzer seed corpus files to the list of files that don't have
copyright notice.
the buildtools directory it needs when running the ICU4C unit tests in an
out-of-source installation.
The change is a quick workaround for now for an issue that can have wide impact.
Moved the macro from platform.h to uassert.h.
Removed any "unreachable" code that previously occurred after the UPRV_UNREACHABLE macro is used.
Changes based on review from Andy.
Co-authored-by: Daniel Ju <daju@microsoft.com>
- Wires up FormattedNumber[Range] in applicable languages.
- Adds new header files and tests, with minor cleanup to old tests.
- Adds code to guarantee terminating NUL in FormattedNumber[Range].
- Cleanup of API docs for inherited methods in FormattedNumber[Range].
- Reads, parses, and applies the filter file syntax.
- Removes unused keys from the resource bundle.
- Adds sample filter txt file with test in intltest.
- Reads filters.json or .hjson from ICU_DATA_FILTER_FILE environment variable
- Adds DepTarget for dependency semantics, and warns for missing deps.
- Fixes certain tests that crash with sliced locale data.
- Includes support for 4 filter types.
* Note: the diff will show changes to "runstatedir". This is
a feature which ICU does not use, and it is not in any released
autoconf version. Perhaps some distributions patched autoconf
locally.
https://lists.gnu.org/archive/html/autoconf/2018-07/msg00006.html
The API documentation is perfectly clear about this, an empty string for
the value means that the keyword should be removed:
@param keywordValue value of the keyword to be set. If 0-length or
NULL, will result in the keyword being removed. No error is given if
that keyword does not exist.
Remove all POSIX and Win32 specific mutex, atomic and threading implementations
in favor of C++11 std library functions.
Move the related (internal) ICU types and functions into the icu namespace.
Adds some plumbing to allow MutablePatternModifier to set fields, and otherwise builds upon the infrastructure from the previous commit to add the MEASURE_UNIT field.
- Creates new Python package in icu4c/data/buildtool
- Creates BUILRDULES.py in icu4c/data and icu4c/test/testdata, unified between Unix/Windows
- Removes most data build orchestration rules from makedata.mak, testdata.mak, data/Makefile.in, and test/testdata/Makefile.in
- Removes pool.res files and builds them on the fly instead
For historical reasons (commit 3b12074b40),
all C++ code (even #include statements) in cmemory.h is contained in an
#ifdef block at the end of the file. A recent bugfix inadvertently added
an additional #ifdef __cplusplus block at the beginning of the file to
add a new #include statement so that C++ #include statements now are
found in two places (commit 1bad36b91a).
- fastpath for UnicodeSet.add(new last range)
- fewer UnicodeSet memory allocations:
initial internal list array, exponential array growth,
allocate strings list/set only when first one is added
- faster CodePointTrie.getRange(): fewer calls to filter function
- revert UnicodeSet(intprop=value) from trie ranges to range starts + lookup
- cache per-int-prop range starts: fewer lookups
This resolves the immediate problem of brittle memory management
in the error handling code.
An obvious future improvement would be to replace the old C style
"plain struct with pointers" VariantListEntry, AttributeListEntry
and ExtensionListEntry with contemporary C++ style containers that
take care of ownership and memory management.
The shared templated helper class MemoryPool can be used to replace the
local helper class CodePointMatcherWarehouse, reducing the amount of
specialized code needed.
This eliminates the need for a scratch buffer in Locale::forLanguageTag()
and also the need for counting bytes required in uloc_forLanguageTag(),
something that ByteSink will now handle correctly.
Instead of _appendLDMLExtensionAsKeywords() requiring to receive a pre-
allocated buffer of sufficient size to store all the temporary strings
it needs to store, have it use a MemoryPool<CharString> to allocate
storage space as needed.
Storing strings as individual CharString objects, instead of as NUL
delimited substrings in a contiguous memory area, also eliminates the
need for keeping track of string boundaries and NUL terminators.
By moving the required call to uhash_close() into the destructor of
LocExtKeyData and using CharString instead of raw chunks of bytes
allocated with uprv_malloc(), it becomes easier to guarantee that
memory handling is correct, without leaks or overflows.
With the need for custom deleter functions removed, the code doesn't use
any of the additional functionality provided by UVector, it just needs a
simple way to keep track of allocated objects to delete them after it's
done using them, which MemoryPool does in a simpler and typesafe way.
CalendarDataSink doesn't use any of the additional functionality
provided by UVector, it just needs a simple way to keep track of
allocated objects to delete them after it's done using them.
The shared templated helper class MemoryPool is a drop-in replacement
for the local helper class CharStringPool, with a simpler implementation
and an interface that allows parameters to be passed to the constructor.
This will be used first to replace existing locally defined memory
management helper classes such as CharStringPool (uloc_tag.cpp) and
CodePointMatcherWarehouse (numparse_affixes.h), then in new code.
This eliminates the need for a scratch buffer in Locale::toLanguageTag()
and also the need for counting bytes required in uloc_toLanguageTag(),
something that ByteSink will now handle correctly and thereby
eliminating the bug where too few bytes required was returned.
Using temporary variables for the two values to be compared here makes
GCC compile the code just like we expect it to. (What it really is that
it otherwise does on some architechtures remains a mystery.)
This will make the tests pass as expected also on IA-32 with GCC.
It'll also make it possible to revert the old workaround for SPARC
introduced by commit 5b0592af79.
Tested:
Linux gcc45 3.16.0-5-686-pae #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) i686 GNU/Linux
Debian clang version 3.5.0-10 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
g++ (Debian 4.9.2-10+deb8u1) 4.9.2
Linux gcc202 4.16.0-1-sparc64-smp #1 SMP Debian 4.16.5-1 (2018-04-29) sparc64 GNU/Linux
clang version 4.0.1-10+sparc64 (tags/RELEASE_401/final)
g++ (Debian 8.2.0-7) 8.2.0
* ICU-20205 Add locale test for RelativeDateTimeFormatter.
* ICU-20205 Fix error in pt relative date data. Improve error handling in code.
* ICU-20205 Add instantiation test & regen data from ICU4C
* ICU-20205 Added DateFormatSymbols error check per jefgen's comments.
