Commit Graph

97 Commits

Author SHA1 Message Date
Konstantin Ritt
a98b541f26 Update Unicode data files to v8.0
Change-Id: I0aa368cb07353924031a9af4f0bdc33692eb1053
Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-05 08:24:58 +00:00
Edward Welbourne
e4d736b6de Fix a #! first line.
An executable .sh file should have its #! be the first two bytes.

Change-Id: I22c9eee6d349df743b02996bef0e093df3f42eb5
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@theqtcompany.com>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@theqtcompany.com>
2015-09-30 18:26:06 +00:00
Konstantin Ritt
dec451cf67 Update Unicode data up to v7.0
* Two newly adopted currency symbols:
  the Azerbaijan manat and the Russia ruble
* Pictographic symbols (including many emoji), geometric symbols,
  arrows, and ornaments originating from the Wingdings and Webdings sets
* Twenty-three new lesser-used and historic scripts
  extending support for written languages of North America, China, India,
  other Asian countries, and Africa
* Letters used in Teuthonista and other transcriptional systems,
  and a new notational set, Duployan

For more details, see http://www.unicode.org/versions/Unicode7.0.0/

The Properties struct's .*Diff members were narrowed down
to signed 15 bits and the unicodeVersion has been expanded to 8 bits.

[ChangeLog][QtCore] Unicode data updated to v.7.0

Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-27 16:40:06 +00:00
Konstantin Ritt
ecdd5648bd Update UCD source files to v7.0
Change-Id: I47277963c926128ad0c4ac5141835e767bb440a7
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-27 16:39:53 +00:00
Konstantin Ritt
31cd326bec Report a correct Unicode version used by Qt
The version bump was forgotten in edfce46a6c

Change-Id: I99501bb314baab68f037417db4b3b14e11f2de02
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 07:02:25 +00:00
Jani Heikkinen
cb95fff1c7 Fixed license headers
Change-Id: Ibebe1318d1c2de97601aa07269705c87737083ee
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@theqtcompany.com>
2015-02-17 18:46:09 +00:00
Jani Heikkinen
83a5694dc2 Update copyright headers
Qt copyrights are now in The Qt Company, so we could update the source
code headers accordingly. In the same go we should also fix the links to
point to qt.io.

Outdated header.LGPL removed (use header.LGPL21 instead)

Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing
combination. New header.LGPL-COMM taken in the use file which were
using old header.LGPL3 (src/plugins/platforms/android/extract.cpp)

Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license
combination

Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe
Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
2015-02-11 06:49:51 +00:00
Marc Mutz
a627c8628f QUnicodeTables: mark all functions as nothrow
Yes, this is necessary. The noexcept operator looks for noexcept tagging,
not at the contents of the function to determine whether to return true.

The more conditionally-noexcept functions are used, the more important it
becomes that low-level functions are correctly marked noexcept. In that, it
is like constexpr.

Change-Id: I4bca178444d1fd7caf3a92f996b1536eebdb5014
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2015-01-10 08:42:55 +01:00
Matti Paaso
974c210835 Update license headers and add new license files
- Renamed LICENSE.LGPL to LICENSE.LGPLv21
- Added LICENSE.LGPLv3
- Removed LICENSE.GPL

Change-Id: Iec3406e3eb3f133be549092015cefe33d259a3f2
Reviewed-by: Iikka Eklund <iikka.eklund@digia.com>
2014-09-24 12:26:19 +02:00
Thiago Macieira
2fe90a61bd Mark all QChar static functions as [[gnu::const]]
The GCC documentation says that a const function is not allowed to read
global memory. This needs to be clarified: it's not allowed to read RW
global memory. It's fine to read read-only memory, as that is equivalent
to just pure code.

The QChar static out-of-line functions only lookup a property of the
given Unicode character and always return the same value.

The only exception is the decomposition() function, which returns a
QString and is therefore not allowed to be marked const.

Change-Id: Id36b2f84a1b8ff9db5acf1d4e59e8b3811068cff
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Giuseppe D'Angelo <giuseppe.dangelo@kdab.com>
2014-07-07 03:51:43 +02:00
Konstantin Ritt
b80fcbdba6 Introduce QChar::JoiningType enum and QChar::joiningType() method
This aimed to disctinct joining types "L", "T", and "U" from just "U".
Unicode 6.3.0 has introduced a character with joining type "L" and
Unicode 7.0 will add a few more characters of joining type "L", so
we'll have to deal with it anyways.

[ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType()
 method that deprecates the old QChar::Joining enum and joining() method.

Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-29 23:19:47 +01:00
Konstantin Ritt
edfce46a6c Update the Unicode Data and Algorithms up to Unicode 6.3.0
* Mongolian and Phags-pa characters have been given a Joining_Type
  classification for contextual shaping. As a part of these additions,
  one Phags-pa character has the Joining_Type value of L (Left Joining),
  which no character had been assigned before.
* The unassigned code points in the Currency Symbols block have been
  given the Bidi_Class property value ET and the Line_Break property
  value PR, to help implementations support new currency symbols,
  when they are encoded.
* Hebrew letters and basic punctuation marks have been assigned
  the newly introduced Word_Break property values Hebrew_Letter,
  Single_Quote, and Double_Quote.
* The Bidi_Class property has been extended with four new values
  for directional isolates.
For more details, see http://www.unicode.org/versions/Unicode6.3.0/

Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-14 15:38:57 +01:00
Konstantin Ritt
a6046be428 Update UCD source files up to Unicode 6.3.0
Change-Id: I9ab58a659af1e758b172a24aa95bce1fea89c33d
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-14 15:38:43 +01:00
Konstantin Ritt
339aff06f9 [1/2] Implement Unicode Normalization Form Quick Check (NF QC)
Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt
into account and generate NF QC bits.
\sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms

Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-13 03:51:07 +02:00
Frederik Gladhorn
c608ec8254 Merge remote-tracking branch 'origin/stable' into dev
Conflicts:
	src/corelib/io/qsavefile_p.h
	src/corelib/tools/qregularexpression.cpp
	src/gui/util/qvalidator.cpp
	src/gui/util/qvalidator.h

Change-Id: I58fdf0358bd86e2fad5d9ad0556f3d3f1f535825
2013-01-22 18:40:13 +01:00
Sergio Ahumada
48e0c4df23 Update copyright year in Digia's license headers
Change-Id: Ic804938fc352291d011800d21e549c10acac66fb
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-01-18 09:07:35 +01:00
Konstantin Ritt
9b0fab6b62 Update Qt internals to use QChar::Script
...and remove the outdated QUnicodeTables::Script enum.
QFontEngineData now has one extra slot that never used
(engines[QChar::Script_Inherited]). engines[QChar::Script_Unknown],
if accessed, would be set with a Box engine instance, and could be used
as a minor optimization some time later.

In order to preserve the existing behavior, we map all scripts up to Latin to Common.

Change-Id: Ide4182a0f8447b4bf25713ecc3fe8097b8fed040
Reviewed-by: Pierre Rossi <pierre.rossi@gmail.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-12-21 19:01:35 +01:00
Konstantin Ritt
f7639c0a6d Add QChar::Script enum
...where the values are not aliased to Common script.

The old QUnicodeTables::Script enum was retained for compatibility reasons
until Qt internals are updated to use QChar::script().
Using QChar::Script instead of QUnicodeTables::Script would improve both
the text analysis (itemization, boundary finding) and the text shaping quality.
This also a required step for switching to Hurfbuzz-NG.

/* This adds 6668 more .rodata bytes */

Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5
Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-20 14:48:32 +01:00
Konstantin Ritt
2fbb69a093 QTBF: Fix issue with no splitting the words at "." (FULL STOP)
As of Unicode 5.1, some punctuation marks were mapped to MidLetter and MidNumLet
for better URL and abbreviations handling which caused "hi.there" to be treated
like if it were just a single word;
until we have the Unicode Text Segmentation tailoring mechanism, retain
the old behavior by remapping (some of) those characters back to their old values.

Change-Id: I49dea6064f2ea40a82fc0b1bc3c4f0b4e803919f
Reviewed-by: David Faure <david.faure@kdab.com>
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-11-23 11:59:50 +01:00
Konstantin Ritt
2672c4fa91 Update the Unicode Data and Algorithms up to Unicode 6.2
Version 6.2 of the Unicode Standard is a special release
dedicated to the early publication of the newly encoded Turkish lira sign.
In addition, there are some significant changes to the Unicode algorithms
for text segmentation and line breaking to improve breaking for emoji symbols.

For more details, see http://www.unicode.org/versions/Unicode6.2.0/

Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-10-09 03:04:41 +02:00
Iikka Eklund
be15856f61 Change copyrights from Nokia to Digia
Change copyrights and license headers from Nokia to Digia

Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-22 19:20:11 +02:00
Konstantin Ritt
b57e2162ef QUnicodeTables: some internal API renamings
enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to
GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively,
their values has been renamed to contain a '_' as logical enum-value separator
(just like many other nums in Qt, e.g. LineBreakClass);
*BreakFormat has been replaced with *Break_Extend (some format characters are
kind of subtype of the extender characters, not vice versa).

Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-22 09:47:59 +02:00
Konstantin Ritt
c1329fba13 Clean-up the Unicode tables generator code and the generated header
This fixes the blocks and memory consumption reports, the whitespace issues
and makes the code a bit cleaner.

Since I'm the only one who does change this code, such a no-op commit
could not hurt anyone or even git blame ;)

Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-22 09:47:59 +02:00
Konstantin Ritt
d8c04d60db Make QUnicodeTables::script() support SMP code points
Instead of expanding the scripts table with script values for the code points
>= 0x10000, it has been merged with the properties table in order to
increase perfomance of the script itemization code (not affected yet).
(Stats: the properties table grew up in 97428-89800 = 7628 bytes;
        the old scripts table was of size 7680 bytes)

The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed
(they were just empty, the "corrigendum" script corrections should be applied
to Scripts.txt directly, *no customization allowed*!).

More script testcases has been added - at least one per supported script.

Task-number: QTBUG-6530

Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-14 05:22:13 +02:00
Konstantin Ritt
12e0319213 Line Breaking Algorithm: handle the Object Replacement Character
See http://www.unicode.org/reports/tr14/#CB
and http://www.unicode.org/reports/tr14/#LB20 for details

Change-Id: Ice0aa2b2ce81f6e39839a353240420436eddd754
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-10 15:58:13 +02:00
Konstantin Ritt
c9100bcce7 Update the Unicode data files up to v6.1.0
Change-Id: I20b94634b1f4ebff10757c2348cfdbbd906e8797
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-10 15:57:54 +02:00
Konstantin Ritt
60e1892d83 Update the qunicodetables generator to deal with UCD 6.1 files
Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-10 15:57:49 +02:00
Konstantin Ritt
2b15c1b30f Move ScriptSentinel enum from header to .cpp
Change-Id: Ic74e8e2471e92aa2014735f6ab0bb4f3b88de206
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-25 21:48:44 +02:00
Konstantin Ritt
ba300f42bd QChar: add isSurrogate() and isNonCharacter() to the public API
+ QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro
replace uses of hardcoded values with the new API; remove leftovers

Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2012-05-16 04:24:56 +02:00
Konstantin Ritt
3fe02eaa12 significant unicodetables generator performance optimization
since the entire range of a valid unicode code points is in use, QHash
is suboptimal and could be replaced with QList;
taking the value by ref and not inserting it back to the map + not calculating
the default value over and over gains us up to 60% performance boost!

Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-11 10:38:25 +02:00
Konstantin Ritt
8c0048a377 add some useful methods to QUnicodeTables::
in order to reduce code duplication and prepare the ground for upcoming changes

Change-Id: I980244149f65384c9484bbec4682de8b7b848b08
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-10 11:34:25 +02:00
Konstantin Ritt
46b78113b2 add support for non-BMP ligatures
> http://www.unicode.org/versions/Unicode5.2.0/
D. Character Additions:
There are three new characters in the newly-encoded Kaithi script that will
require changes in implementations which make hard-coded assumptions about
composition during normalization. Most new characters added to the standard
with decompositions cannot be generated by the operations toNFC() or toNFKC),
but these three can. Implementers should check their code carefully
to ensure that it handles these three characters correctly.
U+1109A KAITHI LETTER DDDHA
U+1109C KAITHI LETTER RHA
U+110AB KAITHI LETTER VA

UCD 6.1 adds two more of them:
U+1112E CHAKMA VOWEL SIGN O
U+1112F CHAKMA VOWEL SIGN AU

Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-04 15:24:52 +02:00
Konstantin Ritt
f948bb3c6c qunicodetables generator: improve the output and the generated code
better memory usage report;
an additional asserts with conditions the implementation is depends on;
a namespace for the internal static data;
styling fixes

Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-24 21:45:00 +02:00
Konstantin Ritt
ba0d752c2d workaround issue where casing diff overflows signed short
there are two such codepoints were added in the Unicode 5.1:
  U+1D79  LATIN SMALL LETTER INSULAR G
  U+A77D  LATIN CAPITAL LETTER INSULAR G
two more of them were added in the Unicode 6.0:
  U+0265  LATIN SMALL LETTER TURNED H
  U+A78D  LATIN CAPITAL LETTER TURNED H
and two more were added in the Unicode 6.1:
  U+0266  LATIN SMALL LETTER H WITH HOOK
  U+A7AA  LATIN CAPITAL LETTER H WITH HOOK

we map them like special cases with length == 1
(note: all are in BMP which is checked explicitly in the generator)

Change-Id: I8a34164eb3ee2e575b7799cc12d4b96ad5bcd9c6
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-24 21:45:00 +02:00
Konstantin Ritt
73b24486ed UCD-5.0: apply Corrigendum #6
http://unicode.org/versions/corrigendum6.html:
> in Unicode 5.0, the list of characters with the Bidi_Mirrored property
> was made consistent for brackets and quotation marks, in preparation for
> new constraints on bidi mirroring. However, after publication of
> Unicode 5.0.0 it was discovered that this change adversely affected
> several quotation mark characters in deployed data.

Task-number: QTBUG-25169
Change-Id: Id49caf401af2d5a1e6dbcc32b2f350aa20b7f901
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-15 03:53:07 +02:00
Konstantin Ritt
50fefebc84 replace hardcoded values with a surrogate handling methods
Change-Id: Iba079953c46a29404232d2dacbe0c90170097d51
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-11 01:42:12 +02:00
Konstantin Ritt
3b778df102 minor improvement for NormalizationCorrections
let's don't hardcode the latests affected version value and simply use
the one parsed from NormalizationCorrections.txt

Change-Id: I37021e8238d77deada4c5ba7a2d160c87186b9dd
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-11 01:42:12 +02:00
Oswald Buddenhagen
d91cf1e53b clean up qmake-generated projects
remove "header" and assignmets which are defaults or bogus,
reorder some assignments.

Change-Id: I67403872168c890ca3b696753ceb01c605d19be7
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-02-24 05:18:30 +01:00
Konstantin Ritt
9514138a5c optimize QString::toLower()/toUpper() for special cases, step 2
from now, QUnicodeTables::specialCaseMap[] starts with a placeholder; so,
if somethingCaseSpecial is true, then somethingCaseDiff is always greater than 0

Change-Id: Ibb1870512836eee71b1521564c0745096c05b2f9
Merge-request: 70
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
Reviewed-by: Olivier
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2012-02-21 22:31:00 +01:00
Konstantin Ritt
5f04962132 optimize QString::toLower()/toUpper() for special cases, step 1
reorganize QUnicodeTables::specialCaseMap as follows:
specialCaseMap contains sequence entries in form { length, a, b, .. }

Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7
Merge-request: 70
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
Reviewed-by: Olivier
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2012-02-21 22:31:00 +01:00
Jason McDonald
5635823e17 Remove "All rights reserved" line from license headers.
As in the past, to avoid rewriting various autotests that contain
line-number information, an extra blank line has been inserted at the
end of the license text to ensure that this commit does not change the
total number of lines in the license header.

Change-Id: I311e001373776812699d6efc045b5f742890c689
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-30 03:54:59 +01:00
Jason McDonald
629d6eda5c Update contact information in license headers.
Replace Nokia contact email address with Qt Project website.

Change-Id: I431bbbf76d7c27d8b502f87947675c116994c415
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-23 04:04:33 +01:00
Jason McDonald
1fdfc2abfe Update copyright year in license headers.
Change-Id: I02f2c620296fcd91d4967d58767ea33fc4e1e7dc
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
2012-01-05 06:36:56 +01:00
Ritt Konstantin
42402a1672 replace 'const QChar &' with 'QChar ' for QChar and QString
Merge-request: 69
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>

Change-Id: I61f5a54b783252029fcad95677958fa6a2130d01
Reviewed-by: Olivier Goffart <ogoffart@kde.org>
2011-10-26 19:59:36 +02:00
Ritt Konstantin
d17c76feee drop an obsolete QChar::NoCategory enum value
there is no such category in the Unicode specs. the QChar::NoCategory
was a subject of bugs since it was introduced. int 4.6 it's meaning was
limited to mention ucs4 > UNICODE_LAST_CODEPOINT only (which is useless anyways)
in order to preserve the old (wrong) behavior.
fix it now for qtbase

Change-Id: I630534824e071090b39772881e747c1fdb758719
Reviewed-on: http://codereview.qt.nokia.com/1584
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2011-07-13 13:31:13 +02:00
Jyri Tahtela
f9f395c28b Update licenseheader text in source files for qtbase Qt module
Updated version of LGPL and FDL licenseheaders.
Apply release phase licenseheaders for all source files.

Reviewed-by: Trust Me
2011-05-24 12:34:08 +03:00
Qt by Nokia
38be0d1383 Initial import from the monolithic Qt.
This is the beginning of revision history for this module. If you
want to look at revision history older than this, please refer to the
Qt Git wiki for how to use Git history grafting. At the time of
writing, this wiki is located here:

http://qt.gitorious.org/qt/pages/GitIntroductionWithQt

If you have already performed the grafting and you don't see any
history beyond this commit, try running "git log" with the "--follow"
argument.

Branched from the monolithic repo, Qt master branch, at commit
896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 12:05:43 +02:00