...and remove the outdated QUnicodeTables::Script enum.
QFontEngineData now has one extra slot that never used
(engines[QChar::Script_Inherited]). engines[QChar::Script_Unknown],
if accessed, would be set with a Box engine instance, and could be used
as a minor optimization some time later.
In order to preserve the existing behavior, we map all scripts up to Latin to Common.
Change-Id: Ide4182a0f8447b4bf25713ecc3fe8097b8fed040
Reviewed-by: Pierre Rossi <pierre.rossi@gmail.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
...where the values are not aliased to Common script.
The old QUnicodeTables::Script enum was retained for compatibility reasons
until Qt internals are updated to use QChar::script().
Using QChar::Script instead of QUnicodeTables::Script would improve both
the text analysis (itemization, boundary finding) and the text shaping quality.
This also a required step for switching to Hurfbuzz-NG.
/* This adds 6668 more .rodata bytes */
Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5
Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
Some of them were incorrectly spelled (for a while)
Change-Id: I871968e3bbdd2172f4c4dfb6e74729c05e7e8e01
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
As of Unicode 5.1, some punctuation marks were mapped to MidLetter and MidNumLet
for better URL and abbreviations handling which caused "hi.there" to be treated
like if it were just a single word;
until we have the Unicode Text Segmentation tailoring mechanism, retain
the old behavior by remapping (some of) those characters back to their old values.
Change-Id: I49dea6064f2ea40a82fc0b1bc3c4f0b4e803919f
Reviewed-by: David Faure <david.faure@kdab.com>
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
...just like described in http://www.unicode.org/reports/tr35/#Likely_Subtags.
This is much more effective than current "guessing" algorithm
+ makes it possible to instantiate a locale by the script or territory code only.
Change-Id: I674f8476e65b01c56960b6e83a1a346df0715274
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
This patch adds some missing codes (two-letter ones only),
removes an outdated ones, and updates some names.
The legacy language codes are handled in QLocalePrivate::codeToLanguage()
(fortunately, there are only 4 of such codes).
Change-Id: Iff50aecd1c762b6399cd151aebb955f341d366c6
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
...with a generated one in a way similar to what
http://www.unicode.org/reports/tr35/#Likely_Subtags suggests.
The supplemental/likelySubtags.xml contains all the required data.
This changes some default countries to a most-expected ones.
Change-Id: I920a5623601d8661a943e78197d3bcc838191483
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
as their meaning, in fact, is unknown (or default) country/script.
Change-Id: Id75a70d4b33c2092de414f3ac357f6bcb627ba47
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Version 6.2 of the Unicode Standard is a special release
dedicated to the early publication of the newly encoded Turkish lira sign.
In addition, there are some significant changes to the Unicode algorithms
for text segmentation and line breaking to improve breaking for emoji symbols.
For more details, see http://www.unicode.org/versions/Unicode6.2.0/
Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Change copyrights and license headers from Nokia to Digia
Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
Command line apps that use stdout need the CONSOLE subsystem.
Change-Id: Ic220f45119ea70372e27b4a364f7264cc649726d
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@nokia.com>
Reviewed-by: Daniel Teske <daniel.teske@nokia.com>
enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to
GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively,
their values has been renamed to contain a '_' as logical enum-value separator
(just like many other nums in Qt, e.g. LineBreakClass);
*BreakFormat has been replaced with *Break_Extend (some format characters are
kind of subtype of the extender characters, not vice versa).
Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
This fixes the blocks and memory consumption reports, the whitespace issues
and makes the code a bit cleaner.
Since I'm the only one who does change this code, such a no-op commit
could not hurt anyone or even git blame ;)
Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Instead of expanding the scripts table with script values for the code points
>= 0x10000, it has been merged with the properties table in order to
increase perfomance of the script itemization code (not affected yet).
(Stats: the properties table grew up in 97428-89800 = 7628 bytes;
the old scripts table was of size 7680 bytes)
The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed
(they were just empty, the "corrigendum" script corrections should be applied
to Scripts.txt directly, *no customization allowed*!).
More script testcases has been added - at least one per supported script.
Task-number: QTBUG-6530
Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
+ QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro
replace uses of hardcoded values with the new API; remove leftovers
Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
since the entire range of a valid unicode code points is in use, QHash
is suboptimal and could be replaced with QList;
taking the value by ref and not inserting it back to the map + not calculating
the default value over and over gains us up to 60% performance boost!
Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
in order to reduce code duplication and prepare the ground for upcoming changes
Change-Id: I980244149f65384c9484bbec4682de8b7b848b08
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
> http://www.unicode.org/versions/Unicode5.2.0/
D. Character Additions:
There are three new characters in the newly-encoded Kaithi script that will
require changes in implementations which make hard-coded assumptions about
composition during normalization. Most new characters added to the standard
with decompositions cannot be generated by the operations toNFC() or toNFKC),
but these three can. Implementers should check their code carefully
to ensure that it handles these three characters correctly.
U+1109A KAITHI LETTER DDDHA
U+1109C KAITHI LETTER RHA
U+110AB KAITHI LETTER VA
UCD 6.1 adds two more of them:
U+1112E CHAKMA VOWEL SIGN O
U+1112F CHAKMA VOWEL SIGN AU
Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
This operation should be a no-op anyway, since at this point in time,
the fromAscii and toAscii functions simply call their fromLatin1 and
toLatin1 counterparts.
Task-number: QTBUG-21872
Change-Id: Ib1eaf42679ab5db4005192c3d00ba79e43edfcca
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
better memory usage report;
an additional asserts with conditions the implementation is depends on;
a namespace for the internal static data;
styling fixes
Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
there are two such codepoints were added in the Unicode 5.1:
U+1D79 LATIN SMALL LETTER INSULAR G
U+A77D LATIN CAPITAL LETTER INSULAR G
two more of them were added in the Unicode 6.0:
U+0265 LATIN SMALL LETTER TURNED H
U+A78D LATIN CAPITAL LETTER TURNED H
and two more were added in the Unicode 6.1:
U+0266 LATIN SMALL LETTER H WITH HOOK
U+A7AA LATIN CAPITAL LETTER H WITH HOOK
we map them like special cases with length == 1
(note: all are in BMP which is checked explicitly in the generator)
Change-Id: I8a34164eb3ee2e575b7799cc12d4b96ad5bcd9c6
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
In Qt4 QLocalePrivate is a struct returned by a d() method. This will be
unsuitable for the planned change to use ICU and may cause BIC issues.
This change makes QLocalePrivate a class and creates a new struct
QLocaleData to hold the data index. Further clean-ups are possible
but are left for later.
Change-Id: Ie316a07790f74674a3b520b735dff72695cc4060
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
http://unicode.org/versions/corrigendum6.html:
> in Unicode 5.0, the list of characters with the Bidi_Mirrored property
> was made consistent for brackets and quotation marks, in preparation for
> new constraints on bidi mirroring. However, after publication of
> Unicode 5.0.0 it was discovered that this change adversely affected
> several quotation mark characters in deployed data.
Task-number: QTBUG-25169
Change-Id: Id49caf401af2d5a1e6dbcc32b2f350aa20b7f901
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
let's don't hardcode the latests affected version value and simply use
the one parsed from NormalizationCorrections.txt
Change-Id: I37021e8238d77deada4c5ba7a2d160c87186b9dd
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
The implementation of the various qHash overloads offered by
Qt can change at any time for any reason
(speed, quality, security, ...).
Therefore, relying on the fact that qHash will always give
an identical result across Qt versions (... across different
processes, etc.), given identical input, is wrong.
Note that this also implies that one cannot rely on QHash
having a stable ordering (even without the random qHash seed).
For such use cases, one must use f.i. a private hash function
that will never change outside his own control.
This patch adds a private hash function for QStrings,
which is identical to the Qt(4) qHash(QString) implementation.
A couple of spots in Qt where the results of a qHash call were
actually saved on disk are ported to use the new function,
and a bit of documentation is added to QHash docs.
Change-Id: Ia3731ea26ac68649b535b95e9f36fbec3df693c8
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Robin Burchell <robin+qt@viroteck.net>
Month Names and Standalone Month Names are stored separately, but for
majority of locales the names are the same and so storage is duplicated.
By storing both sets of names in the same array 50KB is saved in
libQtCore.so on Linux.
Depends on change Ic84bbc82 in branch api_review for CLDR 1.9.1
Change-Id: I83224ebc2180ee6de69797fa50d38348acc94107
Reviewed-by: Denis Dzyubenko <denis.dzyubenko@nokia.com>
remove "header" and assignmets which are defaults or bogus,
reorder some assignments.
Change-Id: I67403872168c890ca3b696753ceb01c605d19be7
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
from now, QUnicodeTables::specialCaseMap[] starts with a placeholder; so,
if somethingCaseSpecial is true, then somethingCaseDiff is always greater than 0
Change-Id: Ibb1870512836eee71b1521564c0745096c05b2f9
Merge-request: 70
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
Reviewed-by: Olivier
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
The strange indentation has made maintenance of license headers
difficult on several occasions.
Change-Id: Ib29a7e5275e2c7a2c13a3f32fd2479f34d3353ca
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
As in the past, to avoid rewriting various autotests that contain
line-number information, an extra blank line has been inserted at the
end of the license text to ensure that this commit does not change the
total number of lines in the license header.
Change-Id: I311e001373776812699d6efc045b5f742890c689
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
