Moved the macro from platform.h to uassert.h.
Removed any "unreachable" code that previously occurred after the UPRV_UNREACHABLE macro is used.
Changes based on review from Andy.
Co-authored-by: Daniel Ju <daju@microsoft.com>
- Wires up FormattedNumber[Range] in applicable languages.
- Adds new header files and tests, with minor cleanup to old tests.
- Adds code to guarantee terminating NUL in FormattedNumber[Range].
- Cleanup of API docs for inherited methods in FormattedNumber[Range].
- Reads, parses, and applies the filter file syntax.
- Removes unused keys from the resource bundle.
- Adds sample filter txt file with test in intltest.
- Reads filters.json or .hjson from ICU_DATA_FILTER_FILE environment variable
- Adds DepTarget for dependency semantics, and warns for missing deps.
- Fixes certain tests that crash with sliced locale data.
- Includes support for 4 filter types.
* Note: the diff will show changes to "runstatedir". This is
a feature which ICU does not use, and it is not in any released
autoconf version. Perhaps some distributions patched autoconf
locally.
https://lists.gnu.org/archive/html/autoconf/2018-07/msg00006.html
The API documentation is perfectly clear about this, an empty string for
the value means that the keyword should be removed:
@param keywordValue value of the keyword to be set. If 0-length or
NULL, will result in the keyword being removed. No error is given if
that keyword does not exist.
Remove all POSIX and Win32 specific mutex, atomic and threading implementations
in favor of C++11 std library functions.
Move the related (internal) ICU types and functions into the icu namespace.
Adds some plumbing to allow MutablePatternModifier to set fields, and otherwise builds upon the infrastructure from the previous commit to add the MEASURE_UNIT field.
- Creates new Python package in icu4c/data/buildtool
- Creates BUILRDULES.py in icu4c/data and icu4c/test/testdata, unified between Unix/Windows
- Removes most data build orchestration rules from makedata.mak, testdata.mak, data/Makefile.in, and test/testdata/Makefile.in
- Removes pool.res files and builds them on the fly instead
For historical reasons (commit 3b12074b40),
all C++ code (even #include statements) in cmemory.h is contained in an
#ifdef block at the end of the file. A recent bugfix inadvertently added
an additional #ifdef __cplusplus block at the beginning of the file to
add a new #include statement so that C++ #include statements now are
found in two places (commit 1bad36b91a).
- fastpath for UnicodeSet.add(new last range)
- fewer UnicodeSet memory allocations:
initial internal list array, exponential array growth,
allocate strings list/set only when first one is added
- faster CodePointTrie.getRange(): fewer calls to filter function
- revert UnicodeSet(intprop=value) from trie ranges to range starts + lookup
- cache per-int-prop range starts: fewer lookups
This resolves the immediate problem of brittle memory management
in the error handling code.
An obvious future improvement would be to replace the old C style
"plain struct with pointers" VariantListEntry, AttributeListEntry
and ExtensionListEntry with contemporary C++ style containers that
take care of ownership and memory management.
The shared templated helper class MemoryPool can be used to replace the
local helper class CodePointMatcherWarehouse, reducing the amount of
specialized code needed.
This eliminates the need for a scratch buffer in Locale::forLanguageTag()
and also the need for counting bytes required in uloc_forLanguageTag(),
something that ByteSink will now handle correctly.
Instead of _appendLDMLExtensionAsKeywords() requiring to receive a pre-
allocated buffer of sufficient size to store all the temporary strings
it needs to store, have it use a MemoryPool<CharString> to allocate
storage space as needed.
Storing strings as individual CharString objects, instead of as NUL
delimited substrings in a contiguous memory area, also eliminates the
need for keeping track of string boundaries and NUL terminators.
By moving the required call to uhash_close() into the destructor of
LocExtKeyData and using CharString instead of raw chunks of bytes
allocated with uprv_malloc(), it becomes easier to guarantee that
memory handling is correct, without leaks or overflows.
With the need for custom deleter functions removed, the code doesn't use
any of the additional functionality provided by UVector, it just needs a
simple way to keep track of allocated objects to delete them after it's
done using them, which MemoryPool does in a simpler and typesafe way.
CalendarDataSink doesn't use any of the additional functionality
provided by UVector, it just needs a simple way to keep track of
allocated objects to delete them after it's done using them.
The shared templated helper class MemoryPool is a drop-in replacement
for the local helper class CharStringPool, with a simpler implementation
and an interface that allows parameters to be passed to the constructor.
This will be used first to replace existing locally defined memory
management helper classes such as CharStringPool (uloc_tag.cpp) and
CodePointMatcherWarehouse (numparse_affixes.h), then in new code.
This eliminates the need for a scratch buffer in Locale::toLanguageTag()
and also the need for counting bytes required in uloc_toLanguageTag(),
something that ByteSink will now handle correctly and thereby
eliminating the bug where too few bytes required was returned.
Using temporary variables for the two values to be compared here makes
GCC compile the code just like we expect it to. (What it really is that
it otherwise does on some architechtures remains a mystery.)
This will make the tests pass as expected also on IA-32 with GCC.
It'll also make it possible to revert the old workaround for SPARC
introduced by commit 5b0592af79.
Tested:
Linux gcc45 3.16.0-5-686-pae #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) i686 GNU/Linux
Debian clang version 3.5.0-10 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
g++ (Debian 4.9.2-10+deb8u1) 4.9.2
Linux gcc202 4.16.0-1-sparc64-smp #1 SMP Debian 4.16.5-1 (2018-04-29) sparc64 GNU/Linux
clang version 4.0.1-10+sparc64 (tags/RELEASE_401/final)
g++ (Debian 8.2.0-7) 8.2.0
* ICU-20205 Add locale test for RelativeDateTimeFormatter.
* ICU-20205 Fix error in pt relative date data. Improve error handling in code.
* ICU-20205 Add instantiation test & regen data from ICU4C
* ICU-20205 Added DateFormatSymbols error check per jefgen's comments.
* ICU-20119 ICU4C 63 API promotion, work in progress.
* ICU-20119 ICU4C 63 API promotion, work in progress.
* ICU-20119 ICU 63 API promotion & change report
* ICU-20119 ICU4C 63 API change report, new draft, still work in progress.
* ICU-20119 ICU4C 63 API change report, another new draft, still work in progress.
* ICU-20119 ICU4C 63 API change report, another new draft.
* ICU-20119 ICU4C 63 API change report, after @preview revert. Hopefully the final rev.
* ICU-20119 ICU4C 63 API change report, clean up a few minor html errors.
uloc_forLanguageTag has a few mapping tables to map grandfathered
language tags and deprecated language subtags to their preferred or
modern values.
Update them based on the latest version of the IANA
language subtag registry. [1]
Five grandfathered tags without a preferred value are still mapped to
what ICU has mapped them to for backward compatibility until the
wisdom of continuing to do so is reviewed.
In addition, map redundant language tags to their preferred values
regardless of whether they're followed by other subtags or not. (e.g.
zh-yue vs zh-yue-u-co-pinyin) .
Similary, ja-latn-hepburn-heploc is mapped to ja-latn-alaic97 (the
variant subtag 'hepburn-helploc' with the prefix 'ja-latn' has the
preferred value, 'alaic97') .
Update the mapping for deprecated language subtags (e.g. 'jw' to
'jv' and a bunch of 3-letter language codes).
Add a new table for deprecated region subtags to map them to their
modern values. (e.g. 'DD' to 'DE').
Add a new test case for deprecated language and region mapping and
a few more cases for updated grandfathered and redundant tag mapping.
[1]
https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
* ICU-20119 Update and fixes for the following BRS tasks:
- Update urename.h
- Test uconfig.h variations
* ICU-20119 Updates copyright scanner script exclusions: don't scan ./git/*.
* ICU-20119 Changes in reply to comments for pull requst #165.
* Fix MSVC C4251: Need to export explicit template instantiation for std::atomic<int32_t> when building DLLs
* Some more warning fixes for MSVC as well.
* Can't use static_cast in C file.
This gets rid of those fixed buffers that caused ICU-13417 to be filed
in the first place, those that prevent handling language tags with very
large amounts of keywords.
A number of fixed buffers will still remain in uloc_tag.cpp (and
elsewhere in the locale handling code) for the time being, but this
change is a necessary first step in cleaning up this code and will
alleviate the most pressing problem encountered by ICU4C users.
An off-by-one error in _getKeywords() caused uloc_canonicalize() to not
write out the final keyword value in case the result would fill up the
buffer exactly, resulting in U_STRING_NOT_TERMINATED_WARNING.
In an effort to reduce the number of warnings that building ICU emits, we should set the following define for ICU4C library code when building using MSVC/VisualStudio: _HAS_EXCEPTIONS=0
This tells the MSVC implementation of the STL that exceptions should not be used - which is fine for ICU4C library code, as the library code does not make use of exceptions at all.
- Use stack allocated UResouceBundle to reduce number of calls to malloc (in a method that can't report back an error if Out-Of-Memory [OOM] happens).
- Use LocalUResourcePointer for automatic clean-up of UResouceBundle.
- Use uprv_strdup instead of calloc + strcpy.
- Changes comments, formatting, etc.
* ICU-20104 Fix lazy init in Indian Calendar. Now matches other calendars.
* ICU-20104 export class IndianCalendar for testing.
* ICU-20104 shot-in-the-dark Windows build experiment.
* ICU-20104 fix memory leak in added test.
The ICU4C sample "date" program just uses the "default" ICU locale. This change lets you pass in an explicit locale argument for testing on platforms like Windows that don't have/use the environment variable "LC_ALL".
- Enable UWP version of ICU to use Environment variable ICU_ENABLE_TENTATIVE_ERA for testing placeholder era names.
- Use LocalArray<int32_t> for the Era Start Dates to simply memory management, so that goto can be removed.
- Also fix some minor typos in header file.
* ICU-20140 Allow duplicated keys in U-extension per RFC 6067
RFC 6067 [1] does allow duplicate keywords, but ICU4C's
uloc_forLanguageCode rejects it as invalid.
Change it to accept duplicate keywords and honor only the
1st one while ignoring subsequent ones per RFC 6067.
[1] Unicode extension to BCP 47:
https://tools.ietf.org/html/rfc6067
* ICU-20140 Add ICU4J test and tweak ICU4C test
ICU4J test diverges from ICU4C tests:
1. Handling of duplicate variants in ICU4J seem to be wrong:
https://unicode-org.atlassian.net/browse/ICU-20148
2. ULocale.forLanguageTag only throws NullPointException so
that ICU4C's test for duplicate attributes cannot be ported.
- Follow the spec to calculate the mimimum significant digits in engineering notation
- The bug is regression since ICU 58. The new test still passes on
ICU58-based DecimalFormat
- Maximum significant digits is not changed
> Clang comes with __has_declspec_attribute to detect whether the name of an attribute is implemented as a MS style __declspec.
> This adds __has_declspect_attribute to the list of clang compatibility macros and then uses that check to determine if __declspec(dllimport) and __declspec(dllexport) can be used.
Also removes status code from two of the other methods; the status code was used only by MutablePatternModifier for the case of a malformed pattern; this error is better handled directly in the apply() method.
Organizing the implementation like this instead will (hopefully) make it
more clear what's being done and make it possible to use analogous
control flow in the copy and move implementations of operator=().
They are C++ template wrappers around Locale::createKeywords() and
Locale::createUnicodeKeywords() respectively, that write to any
container for which an STL style output iterator can be provided.
The simplest imaginable usage would just look like this:
std::string keys[16];
l.getKeywords<std::string>(keys, status);
The unit test has a more elaborate invocation, writing to an std::set<>.
* Updated era data format in supplementalData.
* Include tentative era names in data. Implemented Japanese era loaded from CLDR data in ICU4J.
* ICU4C implementation, ICU4C refactoring. WIP.
* VS project updates and some bug fixes
Also added API comments.
* Review feedback and bug fixes
- NULL to nullptr
- use of LocalUResourceBundlePointer
- TYPO "name" to "named"
- env var checking stricmp() == 0
* API comment correction based on feedback
* Duplicate the comment in ucal.h to calendar.h
* Fixed spelling errors in API comment
Follow-up from pull request #117:
Specify buffer size in only one place, explicitly check status before
proceeding and set status = U_MEMORY_ALLOCATION_ERROR if new fails.
Currently U_FALLTHROUGH is not defined for GCC, meaning that any switch case statements with fall-throughs will generate warnings when building ICU4C.
In GCC 7, they added support for: "_attribute_((fallthrough))" which we can now use for U_FALLTHROUGH, which eliminates many warnings when building with GCC.
This is a wrapper of Locale::createKeywords() that under the hood calls
uloc_toUnicodeLocaleKey() for each key before returning it, so that the
caller won't have to do this.
Enable the Windows UWP version of ICU4C to use TZ update/override files (.res files) for out-of-band timezone data updates. Also use the *W version of the Windows APIs to avoid code-page conversion.
* ICU-20098 Fix the validty check for extlang in uloc_forLanguageTag
BCP 47 has the following for language. extlang subtag can only be
preceded by 2*3ALPHA. Add a check for the length of language subtag
before extlang subtag.
language = 2*3ALPHA ; shortest ISO 639 code
["-" extlang] ; sometimes followed by
; extended language subtags
/ 4ALPHA ; or reserved for future use
/ 5*8ALPHA ; or registered language subtag
extlang = 3ALPHA ; selected ISO 639 codes
*2("-" 3ALPHA) ; permanently reserved}}
With this change, 'hant-cmn-CN' would drop '-cmn-CN' keeping only
'hant'.
* ICU-20098 Fix the validty check for extlang for ICU4J
* ICU-20098 Fix the compiler failure for ICU4J
* ICU-20098 Fix a compile error and test.
* ICU-20098 Add a test for invalid private use singleton
ICU4C's check for private use singleton subtag ('x') is wrong and
treats invalid language tags as valid.
ICU4J's check is correct and does not require any change.
Fix that and add tests to both ICU4C and ICU4J.
ICU does not report Out-Of-Memory (OOM) if it fails to memory map the data file(s) when calling the various platform API(s) to do so.
When you are using ICU with memory-mapped data file(s), and ICU fails to map the data file due to being out-of-memory, it does not bubble this failure up to the API that was called. You will instead get back the error U_MISSING_RESOURCE_ERROR, rather than U_MEMORY_ALLOCATION_ERROR, which might be a bit surprising to the caller of the API. This can lead to the application thinking that there are no resources for "en_US" or "en" (or even "root").
This change modifies ICU4C so that it will report back U_MEMORY_ALLOCATION_ERROR if OOM happens when attempting to load the data files.
* ICU-13530 test & fix cptrie.getRange() when small null data block matches the start of a non-null fast data block
* ICU-13530 fix test bug
* ICU-13530 test & fix bug calculating worst-case data array capacity at start of compaction
* ICU-13530 docs: clarify buildImmutable() discards upper bits, trie then returns unsigned ints; range iteration slower than if ranges were stored directly
* ICU-13530 accept feedback on docs
Use GetDynamicTimeZoneInformation instead of accessing the Windows registries so that it does not rely on COM in wintz. So that UWP apps could also use this code.
