- fastpath for UnicodeSet.add(new last range)
- fewer UnicodeSet memory allocations:
initial internal list array, exponential array growth,
allocate strings list/set only when first one is added
- faster CodePointTrie.getRange(): fewer calls to filter function
- revert UnicodeSet(intprop=value) from trie ranges to range starts + lookup
- cache per-int-prop range starts: fewer lookups
This eliminates the need for a scratch buffer in Locale::toLanguageTag()
and also the need for counting bytes required in uloc_toLanguageTag(),
something that ByteSink will now handle correctly and thereby
eliminating the bug where too few bytes required was returned.
Using temporary variables for the two values to be compared here makes
GCC compile the code just like we expect it to. (What it really is that
it otherwise does on some architechtures remains a mystery.)
This will make the tests pass as expected also on IA-32 with GCC.
It'll also make it possible to revert the old workaround for SPARC
introduced by commit 5b0592af79.
Tested:
Linux gcc45 3.16.0-5-686-pae #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) i686 GNU/Linux
Debian clang version 3.5.0-10 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
g++ (Debian 4.9.2-10+deb8u1) 4.9.2
Linux gcc202 4.16.0-1-sparc64-smp #1 SMP Debian 4.16.5-1 (2018-04-29) sparc64 GNU/Linux
clang version 4.0.1-10+sparc64 (tags/RELEASE_401/final)
g++ (Debian 8.2.0-7) 8.2.0
* ICU-20205 Add locale test for RelativeDateTimeFormatter.
* ICU-20205 Fix error in pt relative date data. Improve error handling in code.
* ICU-20205 Add instantiation test & regen data from ICU4C
* ICU-20205 Added DateFormatSymbols error check per jefgen's comments.
uloc_forLanguageTag has a few mapping tables to map grandfathered
language tags and deprecated language subtags to their preferred or
modern values.
Update them based on the latest version of the IANA
language subtag registry. [1]
Five grandfathered tags without a preferred value are still mapped to
what ICU has mapped them to for backward compatibility until the
wisdom of continuing to do so is reviewed.
In addition, map redundant language tags to their preferred values
regardless of whether they're followed by other subtags or not. (e.g.
zh-yue vs zh-yue-u-co-pinyin) .
Similary, ja-latn-hepburn-heploc is mapped to ja-latn-alaic97 (the
variant subtag 'hepburn-helploc' with the prefix 'ja-latn' has the
preferred value, 'alaic97') .
Update the mapping for deprecated language subtags (e.g. 'jw' to
'jv' and a bunch of 3-letter language codes).
Add a new table for deprecated region subtags to map them to their
modern values. (e.g. 'DD' to 'DE').
Add a new test case for deprecated language and region mapping and
a few more cases for updated grandfathered and redundant tag mapping.
[1]
https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
* ICU-20119 Update and fixes for the following BRS tasks:
- Update urename.h
- Test uconfig.h variations
* ICU-20119 Updates copyright scanner script exclusions: don't scan ./git/*.
* ICU-20119 Changes in reply to comments for pull requst #165.
* Fix MSVC C4251: Need to export explicit template instantiation for std::atomic<int32_t> when building DLLs
* Some more warning fixes for MSVC as well.
* Can't use static_cast in C file.
This gets rid of those fixed buffers that caused ICU-13417 to be filed
in the first place, those that prevent handling language tags with very
large amounts of keywords.
A number of fixed buffers will still remain in uloc_tag.cpp (and
elsewhere in the locale handling code) for the time being, but this
change is a necessary first step in cleaning up this code and will
alleviate the most pressing problem encountered by ICU4C users.
An off-by-one error in _getKeywords() caused uloc_canonicalize() to not
write out the final keyword value in case the result would fill up the
buffer exactly, resulting in U_STRING_NOT_TERMINATED_WARNING.
* ICU-20104 Fix lazy init in Indian Calendar. Now matches other calendars.
* ICU-20104 export class IndianCalendar for testing.
* ICU-20104 shot-in-the-dark Windows build experiment.
* ICU-20104 fix memory leak in added test.
* ICU-20140 Allow duplicated keys in U-extension per RFC 6067
RFC 6067 [1] does allow duplicate keywords, but ICU4C's
uloc_forLanguageCode rejects it as invalid.
Change it to accept duplicate keywords and honor only the
1st one while ignoring subsequent ones per RFC 6067.
[1] Unicode extension to BCP 47:
https://tools.ietf.org/html/rfc6067
* ICU-20140 Add ICU4J test and tweak ICU4C test
ICU4J test diverges from ICU4C tests:
1. Handling of duplicate variants in ICU4J seem to be wrong:
https://unicode-org.atlassian.net/browse/ICU-20148
2. ULocale.forLanguageTag only throws NullPointException so
that ICU4C's test for duplicate attributes cannot be ported.
- Follow the spec to calculate the mimimum significant digits in engineering notation
- The bug is regression since ICU 58. The new test still passes on
ICU58-based DecimalFormat
- Maximum significant digits is not changed
Also removes status code from two of the other methods; the status code was used only by MutablePatternModifier for the case of a malformed pattern; this error is better handled directly in the apply() method.
They are C++ template wrappers around Locale::createKeywords() and
Locale::createUnicodeKeywords() respectively, that write to any
container for which an STL style output iterator can be provided.
The simplest imaginable usage would just look like this:
std::string keys[16];
l.getKeywords<std::string>(keys, status);
The unit test has a more elaborate invocation, writing to an std::set<>.
* Updated era data format in supplementalData.
* Include tentative era names in data. Implemented Japanese era loaded from CLDR data in ICU4J.
* ICU4C implementation, ICU4C refactoring. WIP.
* VS project updates and some bug fixes
Also added API comments.
* Review feedback and bug fixes
- NULL to nullptr
- use of LocalUResourceBundlePointer
- TYPO "name" to "named"
- env var checking stricmp() == 0
* API comment correction based on feedback
* Duplicate the comment in ucal.h to calendar.h
* Fixed spelling errors in API comment
This is a wrapper of Locale::createKeywords() that under the hood calls
uloc_toUnicodeLocaleKey() for each key before returning it, so that the
caller won't have to do this.
* ICU-20098 Fix the validty check for extlang in uloc_forLanguageTag
BCP 47 has the following for language. extlang subtag can only be
preceded by 2*3ALPHA. Add a check for the length of language subtag
before extlang subtag.
language = 2*3ALPHA ; shortest ISO 639 code
["-" extlang] ; sometimes followed by
; extended language subtags
/ 4ALPHA ; or reserved for future use
/ 5*8ALPHA ; or registered language subtag
extlang = 3ALPHA ; selected ISO 639 codes
*2("-" 3ALPHA) ; permanently reserved}}
With this change, 'hant-cmn-CN' would drop '-cmn-CN' keeping only
'hant'.
* ICU-20098 Fix the validty check for extlang for ICU4J
* ICU-20098 Fix the compiler failure for ICU4J
* ICU-20098 Fix a compile error and test.
* ICU-20098 Add a test for invalid private use singleton
ICU4C's check for private use singleton subtag ('x') is wrong and
treats invalid language tags as valid.
ICU4J's check is correct and does not require any change.
Fix that and add tests to both ICU4C and ICU4J.
* ICU-13530 test & fix cptrie.getRange() when small null data block matches the start of a non-null fast data block
* ICU-13530 fix test bug
* ICU-13530 test & fix bug calculating worst-case data array capacity at start of compaction
* ICU-13530 docs: clarify buildImmutable() discards upper bits, trie then returns unsigned ints; range iteration slower than if ranges were stored directly
* ICU-13530 accept feedback on docs
* ICU-13530 copy C/C++ files UTrie2 -> UTrie3
X-SVN-Rev: 40754
* ICU-13530 UTrie3 new files copied from UTrie2: rename types/functions/macros
X-SVN-Rev: 40755
* ICU-13530 debug-print building each UTrie2
X-SVN-Rev: 40756
* ICU-13530 remove two-byte-UTF-8 errorValue block; move highValue from end of data array into header; add errorValue to header
X-SVN-Rev: 40762
* ICU-13530 UTrie3 U16_NEXT/PREV: errorValue for unpaired surrogates
X-SVN-Rev: 40763
* ICU-13530 no more separate values for lead surrogate code units
X-SVN-Rev: 40764
* ICU-13530 change from 11:5 trie bits to 10:6 for simpler UTF-8 code
X-SVN-Rev: 40766
* ICU-13530 UTrie2 build UTrie3 as well, print sizes
X-SVN-Rev: 40767
* ICU-13530 debug-print countSame, sumOverlaps, countInitial
X-SVN-Rev: 40768
* ICU-13530 debug-print whether trie is for CanonIterData
X-SVN-Rev: 40769
* ICU-13530 no index-shift for BMP data, no separate index-2 for 2-byte UTF-8; builder changes incomplete
X-SVN-Rev: 40777
* ICU-13530 remove errorValue and highStart from UNewTrie3
X-SVN-Rev: 40778
* ICU-13530 rewrite UTrie3 builder code
X-SVN-Rev: 40783
* ICU-13530 UTrie3 bug fixes
X-SVN-Rev: 40788
* ICU-13530 fully re-inline _UTRIE3_U8_NEXT()
X-SVN-Rev: 40790
* ICU-13530 find most common all-same data block for dataNullBlock and initialValue
X-SVN-Rev: 40792
* ICU-13530 UTrie3 iterator functions take start and return the end of a range, rather than callback call for each range
X-SVN-Rev: 40800
* ICU-13530 mask off unused data value bits before building a UTrie3 with values less than 32 bits wide
X-SVN-Rev: 40803
* ICU-13530 split utrie3builder.h out of utrie3.h
X-SVN-Rev: 40804
* ICU-13530 separate types UTrie3 vs. UTrie3Builder, implement builder as wrapper over C++ class Trie3Builder in .cpp
X-SVN-Rev: 40809
* ICU-13530 function to make a UTrie3Builder from a UTrie3
X-SVN-Rev: 40810
* ICU-13530 debug-print some data; some cleanup
X-SVN-Rev: 40865
* ICU-13530 BMP 10:6 but supplementary 10:6:4
X-SVN-Rev: 40984
* ICU-13530 move errorValue & highValue to the end of the data table, minimal padding to 4 bytes
X-SVN-Rev: 41011
* ICU-13530 index-1 table gap of index-2 null blocks
X-SVN-Rev: 41018
* ICU-13530 test with more than 128k compacted data
X-SVN-Rev: 41034
* ICU-13530 supplementary bits 11:5:4 saves a little space
X-SVN-Rev: 41039
* ICU-13530 supplementary bits 6:5:5:4 instead of gap: about same size but simpler
X-SVN-Rev: 41050
* ICU-13530 remove unnecessary utrie3_clone(built trie)
X-SVN-Rev: 41058
* ICU-13530 remove unnecessary UTrie3StringIterator
X-SVN-Rev: 41059
* ICU-13530 back to UTRIE3_GET...() macros *returning* data values
X-SVN-Rev: 41060
* ICU-13530 fast vs. small
X-SVN-Rev: 41066
* ICU-13530 always load NFC data, add simple normalization performance test
X-SVN-Rev: 41110
* ICU-13530 change normalization main trie to UTrie3 with special values for lead surrogates; forbid non-inert surrogate code *points* because unable to store values different from code *units*; runtime code work around that for code point lookup and iteration; adjust UTS 46 for normalization no longer mapping unpaired surrogates to U+FFFD
X-SVN-Rev: 41122
* ICU-13530 simplenormperf bug fix and NFC base line
X-SVN-Rev: 41126
* ICU-13530 move normalization getRange skipping lead surrogates to API getRangeSkipLead()
X-SVN-Rev: 41182
* ICU-13530 switch CanonIterData and gennorm2 Norms to UTrie3
X-SVN-Rev: 41183
* ICU-13530 remove unused overwrite parameter from setRange()
X-SVN-Rev: 41184
* ICU-13530 getRange skip lead -> fixed surrogates
X-SVN-Rev: 41219
* ICU-13530 minor cleanup
X-SVN-Rev: 41221
* ICU-13530 UTS 46 code map unpaired surrogates to U+FFFD before normalization
X-SVN-Rev: 41224
* ICU-13530 minor internal-docs cleanup
X-SVN-Rev: 41225
* ICU-13530 rename UTrie3 to UCPTrie, and other name changes
X-SVN-Rev: 41226
* ICU-13530 add 8-bit data option; add type-any & valueBits-any for fromBinary(); macros consistently source type then data width
X-SVN-Rev: 41234
* ICU-13530 scrub the API docs for the proposal
X-SVN-Rev: 41319
* ICU-13530 tag internal definitions as such, or move them to an internal header
X-SVN-Rev: 41320
* ICU-13530 Java API skeleton
X-SVN-Rev: 41326
* ICU-13530 API feedback: ValueWidth, MutableCodePointTrie, base CodePointMap, ...
X-SVN-Rev: 41382
* ICU-13530 add UCPTrie valueWidth field and padding, and combine data pointers into a union
X-SVN-Rev: 41408
* ICU-13530 switch some macros to using dataAccess parameter: separate index vs. data lookups, no macro variant for each value width
X-SVN-Rev: 41409
* ICU-13530 StringIterator is no longer a java.util.Iterator (bad fit)
X-SVN-Rev: 41455
* ICU-13530 CodePointTrie.java code complete
X-SVN-Rev: 41518
* ICU-13530 finish Java port incl test; keep C++ parallel
* ICU-13530 adjust API for feedback: rename HandleValue to FilterValue, change getRange+getRangeFixedSurr(bool allSurr) to enum RangeOption+getRange(enum option); change remaining C macros to use dataAccess for 16/32/8-bit value widths; fix/clarify some API docs
* ICU-13530 add javadoc
* ICU-13530 document UCPTrie binary data format
* ICU-13530 update .nrm formatVersion 3->4, document change in surrogate handling with new trie
* ICU-13530 re-hardcode NFC data
* move trie swapper code into new file; add new files to Windows project files; turn off trie debugging
* ICU-13530 minor cleanup
* ICU-13530 test more range starts; fix a C test leak
* ICU-13530 regenerate Java data from scratch
* ICU-13530 review feedback changes: API docs typos, more @internal, C++11 field initializers, fix potential leak in MutableCodePointTrie::fromUCPTrie()
* ICU-13530 rename interface FilterValue to ValueFilter
Includes all line break tailorings.
Corresponding updates to monkey test rules.
State table builder, fix missed table optimization, uncovered by new rule.
This will prevent littering the source tree with spurious .pyc files.
The potential faster execution when re-running a script that has an
up-to-date .pyc file is negligible.
