Commit Graph

200 Commits

Author SHA1 Message Date
Markus Scherer
0565894534 ICU-20497 Unicode 12.1 2019-04-04 10:23:24 -07:00
Markus Scherer
98589d9cc7 ICU-20203 Unicode 12 final data (only trivial changes) 2019-03-13 08:57:05 -07:00
Markus Scherer
ac4387a374 ICU-20203 Unicode 12 data 20190214 2019-02-15 11:37:34 -08:00
Markus Scherer
d2e3a8847d ICU-20111 move text layout properties data into a new ulayout.icu data file 2019-02-14 08:30:57 -08:00
Markus Scherer
ea7c030961 ICU-20203 update ICU to Unicode 12 beta
- data as of 2018-nov-26
- API constants for new blocks & scripts
- sync RBBIMonkeyTest.java test data with C++
2018-11-28 23:13:07 +01:00
Markus Scherer
d2ec8987a7
ICU-8966 ICU-12850 add API/data/code for text layout properties InPC, InSC, vo (#92)
ICU-8966: Indic_Positional_Category & Indic_Syllabic_Category

ICU-12850: Vertical_Orientation
2018-09-27 14:27:39 -07:00
Markus Scherer
fe3eb3ed5c
ICU-13530 add UCPTrie/CodePointTrie, switch normalization to use it (#48)
* ICU-13530 copy C/C++ files UTrie2 -> UTrie3

X-SVN-Rev: 40754

* ICU-13530 UTrie3 new files copied from UTrie2: rename types/functions/macros

X-SVN-Rev: 40755

* ICU-13530 debug-print building each UTrie2

X-SVN-Rev: 40756

* ICU-13530 remove two-byte-UTF-8 errorValue block; move highValue from end of data array into header; add errorValue to header

X-SVN-Rev: 40762

* ICU-13530 UTrie3 U16_NEXT/PREV: errorValue for unpaired surrogates

X-SVN-Rev: 40763

* ICU-13530 no more separate values for lead surrogate code units

X-SVN-Rev: 40764

* ICU-13530 change from 11:5 trie bits to 10:6 for simpler UTF-8 code

X-SVN-Rev: 40766

* ICU-13530 UTrie2 build UTrie3 as well, print sizes

X-SVN-Rev: 40767

* ICU-13530 debug-print countSame, sumOverlaps, countInitial

X-SVN-Rev: 40768

* ICU-13530 debug-print whether trie is for CanonIterData

X-SVN-Rev: 40769

* ICU-13530 no index-shift for BMP data, no separate index-2 for 2-byte UTF-8; builder changes incomplete

X-SVN-Rev: 40777

* ICU-13530 remove errorValue and highStart from UNewTrie3

X-SVN-Rev: 40778

* ICU-13530 rewrite UTrie3 builder code

X-SVN-Rev: 40783

* ICU-13530 UTrie3 bug fixes

X-SVN-Rev: 40788

* ICU-13530 fully re-inline _UTRIE3_U8_NEXT()

X-SVN-Rev: 40790

* ICU-13530 find most common all-same data block for dataNullBlock and initialValue

X-SVN-Rev: 40792

* ICU-13530 UTrie3 iterator functions take start and return the end of a range, rather than callback call for each range

X-SVN-Rev: 40800

* ICU-13530 mask off unused data value bits before building a UTrie3 with values less than 32 bits wide

X-SVN-Rev: 40803

* ICU-13530 split utrie3builder.h out of utrie3.h

X-SVN-Rev: 40804

* ICU-13530 separate types UTrie3 vs. UTrie3Builder, implement builder as wrapper over C++ class Trie3Builder in .cpp

X-SVN-Rev: 40809

* ICU-13530 function to make a UTrie3Builder from a UTrie3

X-SVN-Rev: 40810

* ICU-13530 debug-print some data; some cleanup

X-SVN-Rev: 40865

* ICU-13530 BMP 10:6 but supplementary 10:6:4

X-SVN-Rev: 40984

* ICU-13530 move errorValue & highValue to the end of the data table, minimal padding to 4 bytes

X-SVN-Rev: 41011

* ICU-13530 index-1 table gap of index-2 null blocks

X-SVN-Rev: 41018

* ICU-13530 test with more than 128k compacted data

X-SVN-Rev: 41034

* ICU-13530 supplementary bits 11:5:4 saves a little space

X-SVN-Rev: 41039

* ICU-13530 supplementary bits 6:5:5:4 instead of gap: about same size but simpler

X-SVN-Rev: 41050

* ICU-13530 remove unnecessary utrie3_clone(built trie)

X-SVN-Rev: 41058

* ICU-13530 remove unnecessary UTrie3StringIterator

X-SVN-Rev: 41059

* ICU-13530 back to UTRIE3_GET...() macros *returning* data values

X-SVN-Rev: 41060

* ICU-13530 fast vs. small

X-SVN-Rev: 41066

* ICU-13530 always load NFC data, add simple normalization performance test

X-SVN-Rev: 41110

* ICU-13530 change normalization main trie to UTrie3 with special values for lead surrogates; forbid non-inert surrogate code *points* because unable to store values different from code *units*; runtime code work around that for code point lookup and iteration; adjust UTS 46 for normalization no longer mapping unpaired surrogates to U+FFFD

X-SVN-Rev: 41122

* ICU-13530 simplenormperf bug fix and NFC base line

X-SVN-Rev: 41126

* ICU-13530 move normalization getRange skipping lead surrogates to API getRangeSkipLead()

X-SVN-Rev: 41182

* ICU-13530 switch CanonIterData and gennorm2 Norms to UTrie3

X-SVN-Rev: 41183

* ICU-13530 remove unused overwrite parameter from setRange()

X-SVN-Rev: 41184

* ICU-13530 getRange skip lead -> fixed surrogates

X-SVN-Rev: 41219

* ICU-13530 minor cleanup

X-SVN-Rev: 41221

* ICU-13530 UTS 46 code map unpaired surrogates to U+FFFD before normalization

X-SVN-Rev: 41224

* ICU-13530 minor internal-docs cleanup

X-SVN-Rev: 41225

* ICU-13530 rename UTrie3 to UCPTrie, and other name changes

X-SVN-Rev: 41226

* ICU-13530 add 8-bit data option; add type-any & valueBits-any for fromBinary(); macros consistently source type then data width

X-SVN-Rev: 41234

* ICU-13530 scrub the API docs for the proposal

X-SVN-Rev: 41319

* ICU-13530 tag internal definitions as such, or move them to an internal header

X-SVN-Rev: 41320

* ICU-13530 Java API skeleton

X-SVN-Rev: 41326

* ICU-13530 API feedback: ValueWidth, MutableCodePointTrie, base CodePointMap, ...

X-SVN-Rev: 41382

* ICU-13530 add UCPTrie valueWidth field and padding, and combine data pointers into a union

X-SVN-Rev: 41408

* ICU-13530 switch some macros to using dataAccess parameter: separate index vs. data lookups, no macro variant for each value width

X-SVN-Rev: 41409

* ICU-13530 StringIterator is no longer a java.util.Iterator (bad fit)

X-SVN-Rev: 41455

* ICU-13530 CodePointTrie.java code complete

X-SVN-Rev: 41518

* ICU-13530 finish Java port incl test; keep C++ parallel

* ICU-13530 adjust API for feedback: rename HandleValue to FilterValue, change getRange+getRangeFixedSurr(bool allSurr) to enum RangeOption+getRange(enum option); change remaining C macros to use dataAccess for 16/32/8-bit value widths; fix/clarify some API docs

* ICU-13530 add javadoc

* ICU-13530 document UCPTrie binary data format

* ICU-13530 update .nrm formatVersion 3->4, document change in surrogate handling with new trie

* ICU-13530 re-hardcode NFC data

* move trie swapper code into new file; add new files to Windows project files; turn off trie debugging

* ICU-13530 minor cleanup

* ICU-13530 test more range starts; fix a C test leak

* ICU-13530 regenerate Java data from scratch

* ICU-13530 review feedback changes: API docs typos, more @internal, C++11 field initializers, fix potential leak in MutableCodePointTrie::fromUCPTrie()

* ICU-13530 rename interface FilterValue to ValueFilter
2018-09-27 14:27:38 -07:00
Markus Scherer
c6c8b7201a ICU-13630 update spoof checker inclusion/recommended sets from Unicode 9 to Unicode 11
X-SVN-Rev: 41466
2018-05-25 23:39:01 +00:00
Markus Scherer
ebca759ea1 ICU-13630 Unicode 11 update from near-final data 20180521
X-SVN-Rev: 41426
2018-05-22 01:56:20 +00:00
Markus Scherer
ee37808752 ICU-13630 minor instructions update for numbering systems in CLDR
X-SVN-Rev: 41197
2018-04-05 21:29:58 +00:00
Markus Scherer
f7276c5c59 ICU-13630 finish updating ICU4J except RBBI tests
X-SVN-Rev: 41195
2018-04-05 03:37:21 +00:00
Markus Scherer
03303a6cb6 ICU-13630 Unicode 11 beta data apr02 (security apr03), fix ICU4C tests except RBBI
X-SVN-Rev: 41191
2018-04-03 23:09:49 +00:00
Markus Scherer
af6a771267 ICU-13630 implement, test, use emoji property Extended_Pictographic
X-SVN-Rev: 41094
2018-03-12 05:53:02 +00:00
Markus Scherer
b3aec18a3c ICU-13630 ucase.icu formatVersion 4: more compressible exceptions, and more room for future exceptions growth
X-SVN-Rev: 41093
2018-03-12 00:15:40 +00:00
Markus Scherer
1752b5c8c9 ICU-13630 Unicode 11 beta data mar06, API constants for new property values
X-SVN-Rev: 41092
2018-03-09 23:53:02 +00:00
Markus Scherer
45da4f810d ICU-12985 final? Unicode 10 data from 20170605
X-SVN-Rev: 40153
2017-06-06 02:46:10 +00:00
Markus Scherer
acf2b4cc82 ICU-13186 stop prepending UTF-8 BOM to some Unicode files
X-SVN-Rev: 40149
2017-06-02 22:52:19 +00:00
Markus Scherer
b2ead3e2e1 ICU-8130 UTS 46 conformance test using Unicode IdnaTest.txt
X-SVN-Rev: 40130
2017-05-23 04:44:58 +00:00
Markus Scherer
4f0f0cb395 ICU-12985 changes.txt section Unihan collators: cross-link with Unicode tools site
X-SVN-Rev: 40106
2017-05-03 23:45:33 +00:00
Markus Scherer
cb722b8bcd ICU-12985 Unicode 10 minor data update
X-SVN-Rev: 40104
2017-05-03 23:26:47 +00:00
Markus Scherer
20bee936b1 ICU-12985 ppucd.txt more readable unassigned ranges; block compaction by size savings not value plurality reduces clutter
X-SVN-Rev: 40096
2017-05-02 22:53:28 +00:00
Markus Scherer
761c994436 ICU-12985 pre-parse VerticalOrientation.txt
X-SVN-Rev: 40086
2017-04-28 20:29:22 +00:00
Markus Scherer
eb57bf7c90 ICU-12985 implement the binary Prepended_Concatenation_Mark property
X-SVN-Rev: 40084
2017-04-27 21:11:01 +00:00
Markus Scherer
6ce7f348a3 ICU-12985 implement the binary Emoji_Component property for emoji 5
X-SVN-Rev: 40082
2017-04-26 23:58:36 +00:00
Markus Scherer
edce2be62c ICU-12985 Unicode 10 data 20170418, new property values, adjust tools & tests
X-SVN-Rev: 40079
2017-04-26 21:17:13 +00:00
Markus Scherer
1982037316 ICU-12900 change ppucd.txt for copyright scanner patterns
X-SVN-Rev: 39921
2017-03-23 17:30:41 +00:00
Markus Scherer
e9ae3ff964 ICU-12900 copyright scanner has been fixed, remove the extra ICU-style copyright lines from Unicope data files
X-SVN-Rev: 39920
2017-03-23 17:01:58 +00:00
Markus Scherer
466a569c58 ICU-12900 mostly still Unicode 9.0 but Unicode 10 beta (20170322) segmentation & bidi data and draft emoji 5.0 (also 20170322)
X-SVN-Rev: 39915
2017-03-23 02:14:00 +00:00
Markus Scherer
962c595814 ICU-12669 draft Emoji 4.0 properties data
X-SVN-Rev: 39067
2016-08-18 21:08:23 +00:00
Steven R. Loomis
0c5b2b597d ICU-12455 BRS - BOM fix
X-SVN-Rev: 38915
2016-07-01 16:59:16 +00:00
Michael Ow
8bfa8c576b ICU-12564 Fix some more bom issues with updated copyright notice
X-SVN-Rev: 38850
2016-06-16 05:17:05 +00:00
Michael Ow
61607c2773 ICU-12564 Update copyright notice in trunk
X-SVN-Rev: 38848
2016-06-15 18:58:17 +00:00
Markus Scherer
1a6d70fa44 ICU-12526 Unicode 9 data 2016-jun-09
X-SVN-Rev: 38820
2016-06-10 00:01:46 +00:00
Markus Scherer
5e69db5c2f ICU-12526 integrate Unicode 9 beta
X-SVN-Rev: 38753
2016-05-19 22:48:18 +00:00
Markus Scherer
997ed6a7cd ICU-11764 8 new script codes for Unicode 9 & CLDR 29
X-SVN-Rev: 38608
2016-04-08 22:23:30 +00:00
Michael Ow
a197e45227 ICU-12083 Fix bom in icu4c 57m1
X-SVN-Rev: 38246
2016-02-01 17:14:55 +00:00
Markus Scherer
c10fdc6e29 ICU-11802 add 4 Emoji properties from emoji-data.txt 2.0
X-SVN-Rev: 38183
2016-01-21 04:39:21 +00:00
Markus Scherer
7c3234e250 ICU-11574 Unicode 8.0 final data
X-SVN-Rev: 37548
2015-06-12 11:10:10 +00:00
Markus Scherer
fc290db4c1 ICU-11574 near-final Unicode 8 data with new block API constants, code adjustments & fixes; svn merge --reintegrate branches/markus/uni80
X-SVN-Rev: 37473
2015-05-28 23:16:00 +00:00
Markus Scherer
05a29b34c9 ICU-11562 improve script code API docs
X-SVN-Rev: 37135
2015-03-04 22:23:14 +00:00
Andy Heninger
74157ec338 ICU-11446 Spoof Checker data update: use MA table only.
X-SVN-Rev: 37071
2015-02-26 01:38:14 +00:00
Markus Scherer
a9d7c3e4bd ICU-11449 reorder single scripts not groups, scripts/groups can start on top-16-bit boundaries, data formatVersion 5 for new scripts data and optional reorderRanges appended to reorderCodes
X-SVN-Rev: 36924
2015-01-07 03:37:11 +00:00
Markus Scherer
590a85cbf1 ICU-11448 collation common tertiary weights for normal Hiragana, change builder to cope with below-common tertiary weights
X-SVN-Rev: 36906
2014-12-27 00:38:34 +00:00
Markus Scherer
030eff56d3 ICU-10829 simplify U+FFFE collation: U+FFFE use common non-primary weights, adjust code for that, test order of U+FFFE equivalent to ucol_mergeSortkeys() but not necessarily same sort keys, omit case level if lowerFirst and only common weights
X-SVN-Rev: 36856
2014-12-11 17:04:32 +00:00
Michael Ow
990392306a ICU-11313 Fix bom in FractionalUCA file
X-SVN-Rev: 36847
2014-12-10 21:41:07 +00:00
Markus Scherer
5d211663e6 ICU-11375 ICU root collation: remove Cyrillic contractions
X-SVN-Rev: 36735
2014-11-17 00:41:41 +00:00
Andy Heninger
afe9d5350e ICU-11262 Spoof check data refresh.
X-SVN-Rev: 36553
2014-09-17 23:27:40 +00:00
Scott Russell
74d781408e ICU-10992 BRS: ensure correct UTF-8 file properties
X-SVN-Rev: 36438
2014-09-10 18:56:09 +00:00
Markus Scherer
0f78abc7ee ICU-9644 re-hardcode some normalization data: nfc.nrm
X-SVN-Rev: 36384
2014-09-08 03:05:56 +00:00
Markus Scherer
ec55298e96 ICU-11042 Han radical-stroke order data in the root collator instead of CJK tailorings, adjusted code & tests (merged from branches/markus/rootunihan)
X-SVN-Rev: 36162
2014-08-14 18:47:27 +00:00