AuroraMiddleware/qt5base-lts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4ed483b0e2
qt5base-lts/util/unicode/main.cpp

3194 lines
109 KiB
C++
Raw Normal View History

Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
/****************************************************************************
**
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
** Copyright (C) 2019 The Qt Company Ltd.
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** Contact: https://www.qt.io/licensing/
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
**
** This file is part of the utils of the Qt Toolkit.
**
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** $QT_BEGIN_LICENSE:GPL-EXCEPT$
Change copyrights from Nokia to Digia Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-19 12:28:29 +00:00
** Commercial License Usage
** Licensees holding valid commercial Qt licenses may use this file in
** accordance with the commercial license agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
Update copyright headers Qt copyrights are now in The Qt Company, so we could update the source code headers accordingly. In the same go we should also fix the links to point to qt.io. Outdated header.LGPL removed (use header.LGPL21 instead) Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing combination. New header.LGPL-COMM taken in the use file which were using old header.LGPL3 (src/plugins/platforms/android/extract.cpp) Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license combination Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
2015-01-28 08:44:43 +00:00
** a written agreement between you and The Qt Company. For licensing terms
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** and conditions see https://www.qt.io/terms-conditions. For further
** information use the contact form at https://www.qt.io/contact-us.
Change copyrights from Nokia to Digia Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-19 12:28:29 +00:00
**
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
** GNU General Public License Usage
** Alternatively, this file may be used under the terms of the GNU
** General Public License version 3 as published by the Free Software
** Foundation with exceptions as appearing in the file LICENSE.GPL3-EXCEPT
** included in the packaging of this file. Please review the following
** information to ensure the GNU General Public License requirements will
** be met: https://www.gnu.org/licenses/gpl-3.0.html.
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
**
** $QT_END_LICENSE$
**
****************************************************************************/
#include <qbytearray.h>
#include <qchar.h>
#include <qdebug.h>
Use QList instead of QVector in util Task-number: QTBUG-84469 Change-Id: I077fb5c32456d438a457c1f73852313ea2ea9ae5 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-07 10:04:21 +00:00
#include <qfile.h>
#include <qhash.h>
#include <qlist.h>
#include <qstring.h>
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
#if 0
#include <private/qunicodetables_p.h>
#endif
Update UCD to Revision 26 Include WordBreakTest.html, since a test uses sample strings from it, albeit without actually reading the file. Had to comment out more of the new tests, as at Revision 24, pending an update to harfbuzz and the text boundary detection code. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Task-number: QTBUG-82747 Change-Id: I0082294b09d67ffdc6a9b5c15acf77ad3b86f65f Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2020-03-13 16:26:53 +00:00
#define DATA_VERSION_S "13.0"
#define DATA_VERSION_STR "QChar::Unicode_13_0"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static QHash<QByteArray, QChar::UnicodeVersion> age_map;
static void initAgeMap()
{
struct AgeMap {
const QChar::UnicodeVersion version;
const char *age;
} ageMap[] = {
{ QChar::Unicode_1_1, "1.1" },
{ QChar::Unicode_2_0, "2.0" },
{ QChar::Unicode_2_1_2, "2.1" },
{ QChar::Unicode_3_0, "3.0" },
{ QChar::Unicode_3_1, "3.1" },
{ QChar::Unicode_3_2, "3.2" },
{ QChar::Unicode_4_0, "4.0" },
{ QChar::Unicode_4_1, "4.1" },
{ QChar::Unicode_5_0, "5.0" },
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
{ QChar::Unicode_5_1, "5.1" },
{ QChar::Unicode_5_2, "5.2" },
{ QChar::Unicode_6_0, "6.0" },
{ QChar::Unicode_6_1, "6.1" },
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
{ QChar::Unicode_6_2, "6.2" },
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
{ QChar::Unicode_6_3, "6.3" },
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
{ QChar::Unicode_7_0, "7.0" },
Update Unicode data & algorithms up to v8.0 * Georgian lari currency symbol * A large collection of CJK unified ideographs * Emoji symbols and symbol modifiers * Letters to support the Ik language in Uganda, Kulango in the Côte d’Ivoire, and other languages of Africa * A set of lowercase Cherokee syllables, forming case pairs with the existing Cherokee characters * The Ahom script for support of the Tai Ahom language in India * Arabic letters to support Arwi—the Tamil language written in the Arabic script For more details, see http://www.unicode.org/versions/Unicode8.0.0/ [ChangeLog][QtCore] Unicode data updated to v.8.0 Change-Id: If255f95c9c45655b721369a116299da3cabbba0a Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-02 04:28:14 +00:00
{ QChar::Unicode_8_0, "8.0" },
Update most Unicode data to version 10.0 The text segmentation data is not being updated in this change, as it requires additional code changes. Updating those will come in a follow-up commit. Change-Id: I5d6b6bc96044e8dd0c25cf6f79756e7f68bf6e7c Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 08:47:53 +00:00
{ QChar::Unicode_9_0, "9.0" },
{ QChar::Unicode_10_0, "10.0" },
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
{ QChar::Unicode_11_0, "11.0" },
{ QChar::Unicode_12_0, "12.0" },
{ QChar::Unicode_12_1, "12.1" }, // UCD Revision 24
Update UCD to Revision 26 Include WordBreakTest.html, since a test uses sample strings from it, albeit without actually reading the file. Had to comment out more of the new tests, as at Revision 24, pending an update to harfbuzz and the text boundary detection code. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Task-number: QTBUG-82747 Change-Id: I0082294b09d67ffdc6a9b5c15acf77ad3b86f65f Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2020-03-13 16:26:53 +00:00
{ QChar::Unicode_13_0, "13.0" }, // UCD Revision 26
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ QChar::Unicode_Unassigned, 0 }
};
AgeMap *d = ageMap;
while (d->age) {
age_map.insert(d->age, d->version);
++d;
}
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
static QHash<QByteArray, QChar::Category> categoryMap;
static void initCategoryMap()
{
struct Cat {
QChar::Category cat;
const char *name;
} categories[] = {
{ QChar::Mark_NonSpacing, "Mn" },
{ QChar::Mark_SpacingCombining, "Mc" },
{ QChar::Mark_Enclosing, "Me" },
{ QChar::Number_DecimalDigit, "Nd" },
{ QChar::Number_Letter, "Nl" },
{ QChar::Number_Other, "No" },
{ QChar::Separator_Space, "Zs" },
{ QChar::Separator_Line, "Zl" },
{ QChar::Separator_Paragraph, "Zp" },
{ QChar::Other_Control, "Cc" },
{ QChar::Other_Format, "Cf" },
{ QChar::Other_Surrogate, "Cs" },
{ QChar::Other_PrivateUse, "Co" },
{ QChar::Other_NotAssigned, "Cn" },
{ QChar::Letter_Uppercase, "Lu" },
{ QChar::Letter_Lowercase, "Ll" },
{ QChar::Letter_Titlecase, "Lt" },
{ QChar::Letter_Modifier, "Lm" },
{ QChar::Letter_Other, "Lo" },
{ QChar::Punctuation_Connector, "Pc" },
{ QChar::Punctuation_Dash, "Pd" },
{ QChar::Punctuation_Open, "Ps" },
{ QChar::Punctuation_Close, "Pe" },
{ QChar::Punctuation_InitialQuote, "Pi" },
{ QChar::Punctuation_FinalQuote, "Pf" },
{ QChar::Punctuation_Other, "Po" },
{ QChar::Symbol_Math, "Sm" },
{ QChar::Symbol_Currency, "Sc" },
{ QChar::Symbol_Modifier, "Sk" },
{ QChar::Symbol_Other, "So" },
{ QChar::Other_NotAssigned, 0 }
};
Cat *c = categories;
while (c->name) {
categoryMap.insert(c->name, c->cat);
++c;
}
}
static QHash<QByteArray, QChar::Decomposition> decompositionMap;
static void initDecompositionMap()
{
struct Dec {
QChar::Decomposition dec;
const char *name;
} decompositions[] = {
{ QChar::Canonical, "<canonical>" },
{ QChar::Font, "<font>" },
{ QChar::NoBreak, "<noBreak>" },
{ QChar::Initial, "<initial>" },
{ QChar::Medial, "<medial>" },
{ QChar::Final, "<final>" },
{ QChar::Isolated, "<isolated>" },
{ QChar::Circle, "<circle>" },
{ QChar::Super, "<super>" },
{ QChar::Sub, "<sub>" },
{ QChar::Vertical, "<vertical>" },
{ QChar::Wide, "<wide>" },
{ QChar::Narrow, "<narrow>" },
{ QChar::Small, "<small>" },
{ QChar::Square, "<square>" },
{ QChar::Compat, "<compat>" },
{ QChar::Fraction, "<fraction>" },
{ QChar::NoDecomposition, 0 }
};
Dec *d = decompositions;
while (d->name) {
decompositionMap.insert(d->name, d->dec);
++d;
}
}
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
enum Direction {
DirL = QChar::DirL,
DirR = QChar::DirR,
DirEN = QChar::DirEN,
DirES = QChar::DirES,
DirET = QChar::DirET,
DirAN = QChar::DirAN,
DirCS = QChar::DirCS,
DirB = QChar::DirB,
DirS = QChar::DirS,
DirWS = QChar::DirWS,
DirON = QChar::DirON,
DirLRE = QChar::DirLRE,
DirLRO = QChar::DirLRO,
DirAL = QChar::DirAL,
DirRLE = QChar::DirRLE,
DirRLO = QChar::DirRLO,
DirPDF = QChar::DirPDF,
DirNSM = QChar::DirNSM,
DirBN = QChar::DirBN,
DirLRI = QChar::DirLRI,
DirRLI = QChar::DirRLI,
DirFSI = QChar::DirFSI,
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
DirPDI = QChar::DirPDI,
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
Dir_Unassigned
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
};
static QHash<QByteArray, Direction> directionMap;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
static void initDirectionMap()
{
struct Dir {
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
Direction dir;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
const char *name;
} directions[] = {
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
{ DirL, "L" },
{ DirR, "R" },
{ DirEN, "EN" },
{ DirES, "ES" },
{ DirET, "ET" },
{ DirAN, "AN" },
{ DirCS, "CS" },
{ DirB, "B" },
{ DirS, "S" },
{ DirWS, "WS" },
{ DirON, "ON" },
{ DirLRE, "LRE" },
{ DirLRO, "LRO" },
{ DirAL, "AL" },
{ DirRLE, "RLE" },
{ DirRLO, "RLO" },
{ DirPDF, "PDF" },
{ DirNSM, "NSM" },
{ DirBN, "BN" },
{ DirLRI, "LRI" },
{ DirRLI, "RLI" },
{ DirFSI, "FSI" },
{ DirPDI, "PDI" },
{ Dir_Unassigned, 0 }
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
};
Dir *d = directions;
while (d->name) {
directionMap.insert(d->name, d->dir);
++d;
}
}
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
enum JoiningType {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Joining_None,
Joining_Causing,
Joining_Dual,
Joining_Right,
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
Joining_Left,
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
Joining_Transparent,
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
Joining_Unassigned
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
static QHash<QByteArray, JoiningType> joining_map;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static void initJoiningMap()
{
struct JoiningList {
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
JoiningType joining;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
const char *name;
} joinings[] = {
{ Joining_None, "U" },
{ Joining_Causing, "C" },
{ Joining_Dual, "D" },
{ Joining_Right, "R" },
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
{ Joining_Left, "L" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ Joining_Transparent, "T" },
{ Joining_Unassigned, 0 }
};
JoiningList *d = joinings;
while (d->name) {
joining_map.insert(d->name, d->joining);
++d;
}
}
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
static const char *grapheme_break_class_string =
"enum GraphemeBreakClass {\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" GraphemeBreak_Any,\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
" GraphemeBreak_CR,\n"
" GraphemeBreak_LF,\n"
" GraphemeBreak_Control,\n"
" GraphemeBreak_Extend,\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" GraphemeBreak_ZWJ,\n"
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
" GraphemeBreak_RegionalIndicator,\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
" GraphemeBreak_Prepend,\n"
" GraphemeBreak_SpacingMark,\n"
" GraphemeBreak_L,\n"
" GraphemeBreak_V,\n"
" GraphemeBreak_T,\n"
" GraphemeBreak_LV,\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" GraphemeBreak_LVT,\n"
" Graphemebreak_E_Base,\n"
" Graphemebreak_E_Modifier,\n"
" Graphemebreak_Glue_After_Zwj,\n"
" Graphemebreak_E_Base_GAZ,\n"
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
"\n"
Add missing docs for UCD additions at 5.15 Also remove two stray commas pointed out in code-review and some others noticed on checking for similar. This amends commit c3eb521a0f10112df6b61d2592351c4eef2e1f9b. Change-Id: If20c5146b740defe8d25ff61d399031b5c66ded1 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-11-26 14:52:17 +00:00
" NumGraphemeBreakClasses\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
enum GraphemeBreakClass {
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
GraphemeBreak_Any,
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
GraphemeBreak_CR,
GraphemeBreak_LF,
GraphemeBreak_Control,
GraphemeBreak_Extend,
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
GraphemeBreak_ZWJ,
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
GraphemeBreak_RegionalIndicator,
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
GraphemeBreak_Prepend,
GraphemeBreak_SpacingMark,
GraphemeBreak_L,
GraphemeBreak_V,
GraphemeBreak_T,
GraphemeBreak_LV,
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
GraphemeBreak_LVT,
Graphemebreak_E_Base,
Graphemebreak_E_Modifier,
Graphemebreak_Glue_After_Zwj,
Graphemebreak_E_Base_GAZ,
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
GraphemeBreak_Unassigned
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
static QHash<QByteArray, GraphemeBreakClass> grapheme_break_map;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static void initGraphemeBreak()
{
struct GraphemeBreakList {
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
GraphemeBreakClass brk;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
const char *name;
} breaks[] = {
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ GraphemeBreak_Any, "Any" },
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
{ GraphemeBreak_CR, "CR" },
{ GraphemeBreak_LF, "LF" },
{ GraphemeBreak_Control, "Control" },
{ GraphemeBreak_Extend, "Extend" },
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ GraphemeBreak_ZWJ, "ZWJ" },
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
{ GraphemeBreak_RegionalIndicator, "Regional_Indicator" },
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
{ GraphemeBreak_Prepend, "Prepend" },
{ GraphemeBreak_SpacingMark, "SpacingMark" },
{ GraphemeBreak_L, "L" },
{ GraphemeBreak_V, "V" },
{ GraphemeBreak_T, "T" },
{ GraphemeBreak_LV, "LV" },
{ GraphemeBreak_LVT, "LVT" },
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ Graphemebreak_E_Base, "E_Base" },
{ Graphemebreak_E_Modifier, "E_Modifier" },
{ Graphemebreak_Glue_After_Zwj, "Glue_After_Zwj" },
{ Graphemebreak_E_Base_GAZ, "E_Base_GAZ" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ GraphemeBreak_Unassigned, 0 }
};
GraphemeBreakList *d = breaks;
while (d->name) {
grapheme_break_map.insert(d->name, d->brk);
++d;
}
}
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
static const char *word_break_class_string =
"enum WordBreakClass {\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" WordBreak_Any,\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
" WordBreak_CR,\n"
" WordBreak_LF,\n"
" WordBreak_Newline,\n"
" WordBreak_Extend,\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" WordBreak_ZWJ,\n"
" WordBreak_Format,\n"
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
" WordBreak_RegionalIndicator,\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
" WordBreak_Katakana,\n"
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
" WordBreak_HebrewLetter,\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
" WordBreak_ALetter,\n"
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
" WordBreak_SingleQuote,\n"
" WordBreak_DoubleQuote,\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
" WordBreak_MidNumLet,\n"
" WordBreak_MidLetter,\n"
" WordBreak_MidNum,\n"
" WordBreak_Numeric,\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" WordBreak_ExtendNumLet,\n"
" WordBreak_E_Base,\n"
" WordBreak_E_Modifier,\n"
" WordBreak_Glue_After_Zwj,\n"
" WordBreak_E_Base_GAZ,\n"
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
" WordBreak_WSegSpace,\n"
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
"\n"
Add missing docs for UCD additions at 5.15 Also remove two stray commas pointed out in code-review and some others noticed on checking for similar. This amends commit c3eb521a0f10112df6b61d2592351c4eef2e1f9b. Change-Id: If20c5146b740defe8d25ff61d399031b5c66ded1 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-11-26 14:52:17 +00:00
" NumWordBreakClasses\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
enum WordBreakClass {
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
WordBreak_Any,
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
WordBreak_CR,
WordBreak_LF,
WordBreak_Newline,
WordBreak_Extend,
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
WordBreak_ZWJ,
WordBreak_Format,
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
WordBreak_RegionalIndicator,
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
WordBreak_Katakana,
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
WordBreak_HebrewLetter,
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
WordBreak_ALetter,
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
WordBreak_SingleQuote,
WordBreak_DoubleQuote,
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
WordBreak_MidNumLet,
WordBreak_MidLetter,
WordBreak_MidNum,
WordBreak_Numeric,
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
WordBreak_ExtendNumLet,
WordBreak_E_Base,
WordBreak_E_Modifier,
WordBreak_Glue_After_Zwj,
WordBreak_E_Base_GAZ,
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
WordBreak_WSegSpace,
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
WordBreak_Unassigned
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
static QHash<QByteArray, WordBreakClass> word_break_map;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static void initWordBreak()
{
struct WordBreakList {
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
WordBreakClass brk;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
const char *name;
} breaks[] = {
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ WordBreak_Any, "Any" },
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
{ WordBreak_CR, "CR" },
{ WordBreak_LF, "LF" },
{ WordBreak_Newline, "Newline" },
{ WordBreak_Extend, "Extend" },
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ WordBreak_ZWJ, "ZWJ" },
{ WordBreak_Format, "Format" },
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
{ WordBreak_RegionalIndicator, "Regional_Indicator" },
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
{ WordBreak_Katakana, "Katakana" },
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
{ WordBreak_HebrewLetter, "Hebrew_Letter" },
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
{ WordBreak_ALetter, "ALetter" },
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
{ WordBreak_SingleQuote, "Single_Quote" },
{ WordBreak_DoubleQuote, "Double_Quote" },
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
{ WordBreak_MidNumLet, "MidNumLet" },
{ WordBreak_MidLetter, "MidLetter" },
{ WordBreak_MidNum, "MidNum" },
{ WordBreak_Numeric, "Numeric" },
{ WordBreak_ExtendNumLet, "ExtendNumLet" },
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ WordBreak_E_Base, "E_Base" },
{ WordBreak_E_Modifier, "E_Modifier" },
{ WordBreak_Glue_After_Zwj, "Glue_After_Zwj" },
{ WordBreak_E_Base_GAZ, "E_Base_GAZ" },
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
{ WordBreak_WSegSpace, "WSegSpace" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ WordBreak_Unassigned, 0 }
};
WordBreakList *d = breaks;
while (d->name) {
word_break_map.insert(d->name, d->brk);
++d;
}
}
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
static const char *sentence_break_class_string =
"enum SentenceBreakClass {\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" SentenceBreak_Any,\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
" SentenceBreak_CR,\n"
" SentenceBreak_LF,\n"
" SentenceBreak_Sep,\n"
" SentenceBreak_Extend,\n"
" SentenceBreak_Sp,\n"
" SentenceBreak_Lower,\n"
" SentenceBreak_Upper,\n"
" SentenceBreak_OLetter,\n"
" SentenceBreak_Numeric,\n"
" SentenceBreak_ATerm,\n"
" SentenceBreak_SContinue,\n"
" SentenceBreak_STerm,\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" SentenceBreak_Close,\n"
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
"\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" NumSentenceBreakClasses\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
enum SentenceBreakClass {
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
SentenceBreak_Any,
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
SentenceBreak_CR,
SentenceBreak_LF,
SentenceBreak_Sep,
SentenceBreak_Extend,
SentenceBreak_Sp,
SentenceBreak_Lower,
SentenceBreak_Upper,
SentenceBreak_OLetter,
SentenceBreak_Numeric,
SentenceBreak_ATerm,
SentenceBreak_SContinue,
SentenceBreak_STerm,
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
SentenceBreak_Close,
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
SentenceBreak_Unassigned
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
static QHash<QByteArray, SentenceBreakClass> sentence_break_map;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static void initSentenceBreak()
{
struct SentenceBreakList {
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
SentenceBreakClass brk;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
const char *name;
} breaks[] = {
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ SentenceBreak_Any, "Any" },
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
{ SentenceBreak_CR, "CR" },
{ SentenceBreak_LF, "LF" },
{ SentenceBreak_Sep, "Sep" },
{ SentenceBreak_Extend, "Extend" },
{ SentenceBreak_Extend, "Format" },
{ SentenceBreak_Sp, "Sp" },
{ SentenceBreak_Lower, "Lower" },
{ SentenceBreak_Upper, "Upper" },
{ SentenceBreak_OLetter, "OLetter" },
{ SentenceBreak_Numeric, "Numeric" },
{ SentenceBreak_ATerm, "ATerm" },
{ SentenceBreak_SContinue, "SContinue" },
{ SentenceBreak_STerm, "STerm" },
{ SentenceBreak_Close, "Close" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ SentenceBreak_Unassigned, 0 }
};
SentenceBreakList *d = breaks;
while (d->name) {
sentence_break_map.insert(d->name, d->brk);
++d;
}
}
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
static const char *line_break_class_string =
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
"// see http://www.unicode.org/reports/tr14/tr14-30.html\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"// we don't use the XX and AI classes and map them to AL instead.\n"
"enum LineBreakClass {\n"
" LineBreak_OP, LineBreak_CL, LineBreak_CP, LineBreak_QU, LineBreak_GL,\n"
" LineBreak_NS, LineBreak_EX, LineBreak_SY, LineBreak_IS, LineBreak_PR,\n"
" LineBreak_PO, LineBreak_NU, LineBreak_AL, LineBreak_HL, LineBreak_ID,\n"
" LineBreak_IN, LineBreak_HY, LineBreak_BA, LineBreak_BB, LineBreak_B2,\n"
" LineBreak_ZW, LineBreak_CM, LineBreak_WJ, LineBreak_H2, LineBreak_H3,\n"
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
" LineBreak_JL, LineBreak_JV, LineBreak_JT, LineBreak_RI, LineBreak_CB,\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" LineBreak_EB, LineBreak_EM, LineBreak_ZWJ,\n"
" LineBreak_SA, LineBreak_SG, LineBreak_SP,\n"
" LineBreak_CR, LineBreak_LF, LineBreak_BK,\n"
Unicode tables: minor prettification Put blank lines before the final Num*Classes entries in enums, to set them off visibly from the "real" members. Moved some oddly placed commas to the ends of preceding lines, so that later additions can just add lines (with comma on end) without having to modify the preceding line while doing so. Change-Id: I5188dc25af9e4c17a1882fd9dab070e88013060b Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-11-27 13:40:33 +00:00
"\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" NumLineBreakClasses\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
enum LineBreakClass {
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
LineBreak_OP, LineBreak_CL, LineBreak_CP, LineBreak_QU, LineBreak_GL,
LineBreak_NS, LineBreak_EX, LineBreak_SY, LineBreak_IS, LineBreak_PR,
LineBreak_PO, LineBreak_NU, LineBreak_AL, LineBreak_HL, LineBreak_ID,
LineBreak_IN, LineBreak_HY, LineBreak_BA, LineBreak_BB, LineBreak_B2,
LineBreak_ZW, LineBreak_CM, LineBreak_WJ, LineBreak_H2, LineBreak_H3,
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
LineBreak_JL, LineBreak_JV, LineBreak_JT, LineBreak_RI, LineBreak_CB,
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
LineBreak_EB, LineBreak_EM, LineBreak_ZWJ,
LineBreak_SA, LineBreak_SG, LineBreak_SP,
LineBreak_CR, LineBreak_LF, LineBreak_BK,
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
LineBreak_Unassigned
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
static QHash<QByteArray, LineBreakClass> line_break_map;
static void initLineBreak()
{
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
// ### Classes XX and AI are left out and mapped to AL for now.
// ### Class NL is mapped to BK.
// ### Treating characters of class CJ as class NS will give CSS strict line breaking;
// treating them as class ID will give CSS normal breaking.
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
struct LineBreakList {
LineBreakClass brk;
const char *name;
} breaks[] = {
{ LineBreak_BK, "BK" },
{ LineBreak_CR, "CR" },
{ LineBreak_LF, "LF" },
{ LineBreak_CM, "CM" },
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
{ LineBreak_BK, "NL" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ LineBreak_SG, "SG" },
{ LineBreak_WJ, "WJ" },
{ LineBreak_ZW, "ZW" },
{ LineBreak_GL, "GL" },
{ LineBreak_SP, "SP" },
{ LineBreak_B2, "B2" },
{ LineBreak_BA, "BA" },
{ LineBreak_BB, "BB" },
{ LineBreak_HY, "HY" },
Line Breaking Algorithm: handle the Object Replacement Character See http://www.unicode.org/reports/tr14/#CB and http://www.unicode.org/reports/tr14/#LB20 for details Change-Id: Ice0aa2b2ce81f6e39839a353240420436eddd754 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-31 10:04:32 +00:00
{ LineBreak_CB, "CB" },
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
{ LineBreak_NS, "CJ" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ LineBreak_CL, "CL" },
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
{ LineBreak_CP, "CP" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ LineBreak_EX, "EX" },
{ LineBreak_IN, "IN" },
{ LineBreak_NS, "NS" },
{ LineBreak_OP, "OP" },
{ LineBreak_QU, "QU" },
{ LineBreak_IS, "IS" },
{ LineBreak_NU, "NU" },
{ LineBreak_PO, "PO" },
{ LineBreak_PR, "PR" },
{ LineBreak_SY, "SY" },
{ LineBreak_AL, "AI" },
{ LineBreak_AL, "AL" },
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
{ LineBreak_HL, "HL" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ LineBreak_H2, "H2" },
{ LineBreak_H3, "H3" },
{ LineBreak_ID, "ID" },
{ LineBreak_JL, "JL" },
{ LineBreak_JV, "JV" },
{ LineBreak_JT, "JT" },
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
{ LineBreak_RI, "RI" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ LineBreak_SA, "SA" },
{ LineBreak_AL, "XX" },
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
{ LineBreak_EB, "EB" },
{ LineBreak_EM, "EM" },
{ LineBreak_ZWJ, "ZWJ" },
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
{ LineBreak_Unassigned, 0 }
};
LineBreakList *d = breaks;
while (d->name) {
line_break_map.insert(d->name, d->brk);
++d;
}
}
Add QChar::Script enum ...where the values are not aliased to Common script. The old QUnicodeTables::Script enum was retained for compatibility reasons until Qt internals are updated to use QChar::script(). Using QChar::Script instead of QUnicodeTables::Script would improve both the text analysis (itemization, boundary finding) and the text shaping quality. This also a required step for switching to Hurfbuzz-NG. /* This adds 6668 more .rodata bytes */ Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-08 03:36:49 +00:00
static QHash<QByteArray, QChar::Script> scriptMap;
static void initScriptMap()
{
struct Scrpt {
QChar::Script script;
const char *name;
} scripts[] = {
// general
{ QChar::Script_Unknown, "Unknown" },
{ QChar::Script_Inherited, "Inherited" },
{ QChar::Script_Common, "Common" },
// pre-4.0
{ QChar::Script_Latin, "Latin" },
{ QChar::Script_Greek, "Greek" },
{ QChar::Script_Cyrillic, "Cyrillic" },
{ QChar::Script_Armenian, "Armenian" },
{ QChar::Script_Hebrew, "Hebrew" },
{ QChar::Script_Arabic, "Arabic" },
{ QChar::Script_Syriac, "Syriac" },
{ QChar::Script_Thaana, "Thaana" },
{ QChar::Script_Devanagari, "Devanagari" },
{ QChar::Script_Bengali, "Bengali" },
{ QChar::Script_Gurmukhi, "Gurmukhi" },
{ QChar::Script_Gujarati, "Gujarati" },
{ QChar::Script_Oriya, "Oriya" },
{ QChar::Script_Tamil, "Tamil" },
{ QChar::Script_Telugu, "Telugu" },
{ QChar::Script_Kannada, "Kannada" },
{ QChar::Script_Malayalam, "Malayalam" },
{ QChar::Script_Sinhala, "Sinhala" },
{ QChar::Script_Thai, "Thai" },
{ QChar::Script_Lao, "Lao" },
{ QChar::Script_Tibetan, "Tibetan" },
{ QChar::Script_Myanmar, "Myanmar" },
{ QChar::Script_Georgian, "Georgian" },
{ QChar::Script_Hangul, "Hangul" },
{ QChar::Script_Ethiopic, "Ethiopic" },
{ QChar::Script_Cherokee, "Cherokee" },
{ QChar::Script_CanadianAboriginal, "CanadianAboriginal" },
{ QChar::Script_Ogham, "Ogham" },
{ QChar::Script_Runic, "Runic" },
{ QChar::Script_Khmer, "Khmer" },
{ QChar::Script_Mongolian, "Mongolian" },
{ QChar::Script_Hiragana, "Hiragana" },
{ QChar::Script_Katakana, "Katakana" },
{ QChar::Script_Bopomofo, "Bopomofo" },
{ QChar::Script_Han, "Han" },
{ QChar::Script_Yi, "Yi" },
{ QChar::Script_OldItalic, "OldItalic" },
{ QChar::Script_Gothic, "Gothic" },
{ QChar::Script_Deseret, "Deseret" },
{ QChar::Script_Tagalog, "Tagalog" },
{ QChar::Script_Hanunoo, "Hanunoo" },
{ QChar::Script_Buhid, "Buhid" },
{ QChar::Script_Tagbanwa, "Tagbanwa" },
{ QChar::Script_Coptic, "Coptic" },
// 4.0
{ QChar::Script_Limbu, "Limbu" },
{ QChar::Script_TaiLe, "TaiLe" },
{ QChar::Script_LinearB, "LinearB" },
{ QChar::Script_Ugaritic, "Ugaritic" },
{ QChar::Script_Shavian, "Shavian" },
{ QChar::Script_Osmanya, "Osmanya" },
{ QChar::Script_Cypriot, "Cypriot" },
{ QChar::Script_Braille, "Braille" },
// 4.1
{ QChar::Script_Buginese, "Buginese" },
{ QChar::Script_NewTaiLue, "NewTaiLue" },
{ QChar::Script_Glagolitic, "Glagolitic" },
{ QChar::Script_Tifinagh, "Tifinagh" },
{ QChar::Script_SylotiNagri, "SylotiNagri" },
{ QChar::Script_OldPersian, "OldPersian" },
{ QChar::Script_Kharoshthi, "Kharoshthi" },
// 5.0
{ QChar::Script_Balinese, "Balinese" },
{ QChar::Script_Cuneiform, "Cuneiform" },
{ QChar::Script_Phoenician, "Phoenician" },
{ QChar::Script_PhagsPa, "PhagsPa" },
{ QChar::Script_Nko, "Nko" },
// 5.1
{ QChar::Script_Sundanese, "Sundanese" },
{ QChar::Script_Lepcha, "Lepcha" },
{ QChar::Script_OlChiki, "OlChiki" },
{ QChar::Script_Vai, "Vai" },
{ QChar::Script_Saurashtra, "Saurashtra" },
{ QChar::Script_KayahLi, "KayahLi" },
{ QChar::Script_Rejang, "Rejang" },
{ QChar::Script_Lycian, "Lycian" },
{ QChar::Script_Carian, "Carian" },
{ QChar::Script_Lydian, "Lydian" },
{ QChar::Script_Cham, "Cham" },
// 5.2
{ QChar::Script_TaiTham, "TaiTham" },
{ QChar::Script_TaiViet, "TaiViet" },
{ QChar::Script_Avestan, "Avestan" },
{ QChar::Script_EgyptianHieroglyphs, "EgyptianHieroglyphs" },
{ QChar::Script_Samaritan, "Samaritan" },
{ QChar::Script_Lisu, "Lisu" },
{ QChar::Script_Bamum, "Bamum" },
{ QChar::Script_Javanese, "Javanese" },
{ QChar::Script_MeeteiMayek, "MeeteiMayek" },
{ QChar::Script_ImperialAramaic, "ImperialAramaic" },
{ QChar::Script_OldSouthArabian, "OldSouthArabian" },
{ QChar::Script_InscriptionalParthian, "InscriptionalParthian" },
{ QChar::Script_InscriptionalPahlavi, "InscriptionalPahlavi" },
{ QChar::Script_OldTurkic, "OldTurkic" },
{ QChar::Script_Kaithi, "Kaithi" },
// 6.0
{ QChar::Script_Batak, "Batak" },
{ QChar::Script_Brahmi, "Brahmi" },
{ QChar::Script_Mandaic, "Mandaic" },
// 6.1
{ QChar::Script_Chakma, "Chakma" },
{ QChar::Script_MeroiticCursive, "MeroiticCursive" },
{ QChar::Script_MeroiticHieroglyphs, "MeroiticHieroglyphs" },
{ QChar::Script_Miao, "Miao" },
{ QChar::Script_Sharada, "Sharada" },
{ QChar::Script_SoraSompeng, "SoraSompeng" },
{ QChar::Script_Takri, "Takri" },
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
// 7.0
{ QChar::Script_CaucasianAlbanian, "CaucasianAlbanian" },
{ QChar::Script_BassaVah, "BassaVah" },
{ QChar::Script_Duployan, "Duployan" },
{ QChar::Script_Elbasan, "Elbasan" },
{ QChar::Script_Grantha, "Grantha" },
{ QChar::Script_PahawhHmong, "PahawhHmong" },
{ QChar::Script_Khojki, "Khojki" },
{ QChar::Script_LinearA, "LinearA" },
{ QChar::Script_Mahajani, "Mahajani" },
{ QChar::Script_Manichaean, "Manichaean" },
{ QChar::Script_MendeKikakui, "MendeKikakui" },
{ QChar::Script_Modi, "Modi" },
{ QChar::Script_Mro, "Mro" },
{ QChar::Script_OldNorthArabian, "OldNorthArabian" },
{ QChar::Script_Nabataean, "Nabataean" },
{ QChar::Script_Palmyrene, "Palmyrene" },
{ QChar::Script_PauCinHau, "PauCinHau" },
{ QChar::Script_OldPermic, "OldPermic" },
{ QChar::Script_PsalterPahlavi, "PsalterPahlavi" },
{ QChar::Script_Siddham, "Siddham" },
{ QChar::Script_Khudawadi, "Khudawadi" },
{ QChar::Script_Tirhuta, "Tirhuta" },
{ QChar::Script_WarangCiti, "WarangCiti" },
Update Unicode data & algorithms up to v8.0 * Georgian lari currency symbol * A large collection of CJK unified ideographs * Emoji symbols and symbol modifiers * Letters to support the Ik language in Uganda, Kulango in the Côte d’Ivoire, and other languages of Africa * A set of lowercase Cherokee syllables, forming case pairs with the existing Cherokee characters * The Ahom script for support of the Tai Ahom language in India * Arabic letters to support Arwi—the Tamil language written in the Arabic script For more details, see http://www.unicode.org/versions/Unicode8.0.0/ [ChangeLog][QtCore] Unicode data updated to v.8.0 Change-Id: If255f95c9c45655b721369a116299da3cabbba0a Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-02 04:28:14 +00:00
// 8.0
{ QChar::Script_Ahom, "Ahom" },
{ QChar::Script_AnatolianHieroglyphs, "AnatolianHieroglyphs" },
{ QChar::Script_Hatran, "Hatran" },
{ QChar::Script_Multani, "Multani" },
{ QChar::Script_OldHungarian, "OldHungarian" },
{ QChar::Script_SignWriting, "SignWriting" },
Update most Unicode data to version 10.0 The text segmentation data is not being updated in this change, as it requires additional code changes. Updating those will come in a follow-up commit. Change-Id: I5d6b6bc96044e8dd0c25cf6f79756e7f68bf6e7c Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 08:47:53 +00:00
// 9.0
{ QChar::Script_Adlam, "Adlam" },
{ QChar::Script_Bhaiksuki, "Bhaiksuki" },
{ QChar::Script_Marchen, "Marchen" },
{ QChar::Script_Newa, "Newa" },
{ QChar::Script_Osage, "Osage" },
{ QChar::Script_Tangut, "Tangut" },
// 10.0
{ QChar::Script_MasaramGondi, "MasaramGondi" },
{ QChar::Script_Nushu, "Nushu" },
{ QChar::Script_Soyombo, "Soyombo" },
{ QChar::Script_ZanabazarSquare, "ZanabazarSquare" },
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
// 12.1
{ QChar::Script_Dogra, "Dogra" },
{ QChar::Script_GunjalaGondi, "GunjalaGondi" },
{ QChar::Script_HanifiRohingya, "HanifiRohingya" },
{ QChar::Script_Makasar, "Makasar" },
{ QChar::Script_Medefaidrin, "Medefaidrin" },
{ QChar::Script_OldSogdian, "OldSogdian" },
{ QChar::Script_Sogdian, "Sogdian" },
{ QChar::Script_Elymaic, "Elymaic" },
{ QChar::Script_Nandinagari, "Nandinagari" },
{ QChar::Script_NyiakengPuachueHmong, "NyiakengPuachueHmong" },
{ QChar::Script_Wancho, "Wancho" },
Update UCD to Revision 26 Include WordBreakTest.html, since a test uses sample strings from it, albeit without actually reading the file. Had to comment out more of the new tests, as at Revision 24, pending an update to harfbuzz and the text boundary detection code. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Task-number: QTBUG-82747 Change-Id: I0082294b09d67ffdc6a9b5c15acf77ad3b86f65f Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2020-03-13 16:26:53 +00:00
// 13.0
{ QChar::Script_Chorasmian, "Chorasmian" },
{ QChar::Script_DivesAkuru, "DivesAkuru" },
{ QChar::Script_KhitanSmallScript, "KhitanSmallScript" },
{ QChar::Script_Yezidi, "Yezidi" },
Update most Unicode data to version 10.0 The text segmentation data is not being updated in this change, as it requires additional code changes. Updating those will come in a follow-up commit. Change-Id: I5d6b6bc96044e8dd0c25cf6f79756e7f68bf6e7c Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 08:47:53 +00:00
Add QChar::Script enum ...where the values are not aliased to Common script. The old QUnicodeTables::Script enum was retained for compatibility reasons until Qt internals are updated to use QChar::script(). Using QChar::Script instead of QUnicodeTables::Script would improve both the text analysis (itemization, boundary finding) and the text shaping quality. This also a required step for switching to Hurfbuzz-NG. /* This adds 6668 more .rodata bytes */ Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-08 03:36:49 +00:00
// unhandled
{ QChar::Script_Unknown, 0 }
};
Scrpt *p = scripts;
while (p->name) {
scriptMap.insert(p->name, p->script);
++p;
}
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// Keep this one in sync with the code in createPropertyInfo
static const char *property_string =
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
"enum Case {\n"
" LowerCase,\n"
" UpperCase,\n"
" TitleCase,\n"
" CaseFold,\n"
"\n"
" NumCases\n"
"};\n"
"\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"struct Properties {\n"
" ushort category : 8; /* 5 used */\n"
" ushort direction : 8; /* 5 used */\n"
" ushort combiningClass : 8;\n"
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
" ushort joining : 3;\n"
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
" signed short digitValue : 5;\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
" signed short mirrorDiff : 16;\n"
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
" ushort unicodeVersion : 8; /* 5 used */\n"
" ushort nfQuickCheck : 8;\n" // could be narrowed
Wasm: Add workaround for Emscripten compiler again This work around was added directly to the generated files at some point, and never to the generator it seems. So to avoid removing the workaround again when we regenerate the next time, we need to add it. Task-number: QTBUG-74511 Change-Id: Ided1bd949234ba82df61c55891646823e7f72e80 Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
2019-03-18 07:27:22 +00:00
"#ifdef Q_OS_WASM\n"
" unsigned char : 0; //wasm 64 packing trick\n"
"#endif\n"
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
" struct {\n"
" ushort special : 1;\n"
" signed short diff : 15;\n"
" } cases[NumCases];\n"
Wasm: Add workaround for Emscripten compiler again This work around was added directly to the generated files at some point, and never to the generator it seems. So to avoid removing the workaround again when we regenerate the next time, we need to add it. Task-number: QTBUG-74511 Change-Id: Ided1bd949234ba82df61c55891646823e7f72e80 Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
2019-03-18 07:27:22 +00:00
"#ifdef Q_OS_WASM\n"
" unsigned char : 0; //wasm 64 packing trick\n"
"#endif\n"
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
" ushort graphemeBreakClass : 5; /* 5 used */\n"
" ushort wordBreakClass : 5; /* 5 used */\n"
" ushort lineBreakClass : 6; /* 6 used */\n"
QUnicodeTables: pack Properties struct GCC doesn't like the sequence : 5 : 5 : 8 : 6 : 8 and inserts a :6 padding between the :5 and the :8 and a :2 padding between the :6 and the :8, growing the bitfield by 8 bits of embedded padding and another byte to bring the struct back to sizeof % 2 == 0. Fix by reshuffling the elements and adding a static_assert for the next round. Saves ~5KiB in QtCore executable size. Change-Id: I4758a6f48ba389abc2aee92f60997d42ebb0e5b8 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-09-03 21:40:56 +00:00
" ushort sentenceBreakClass : 8; /* 4 used */\n"
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
" ushort script : 8;\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"};\n\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT const Properties * QT_FASTCALL properties(char32_t ucs4) noexcept;\n"
"Q_CORE_EXPORT const Properties * QT_FASTCALL properties(char16_t ucs2) noexcept;\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static const char *methods =
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT GraphemeBreakClass QT_FASTCALL graphemeBreakClass(char32_t ucs4) noexcept;\n"
Replace Q_DECL_NOTHROW with noexcept the remaining places The first replacement had missed objective-C++ code some places ourside the src dir. In C-files Q_DECL_NOTHROW is replaced with Q_DECL_NOEXCEPT as we still need to turn it off when compiled in C mode, but can get rid of the old NOTHROW moniker. Change-Id: I6370f57066679c5120d0265a69e7e378e09d4759 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-04-09 11:33:15 +00:00
"inline GraphemeBreakClass graphemeBreakClass(QChar ch) noexcept\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"{ return graphemeBreakClass(ch.unicode()); }\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT WordBreakClass QT_FASTCALL wordBreakClass(char32_t ucs4) noexcept;\n"
Replace Q_DECL_NOTHROW with noexcept the remaining places The first replacement had missed objective-C++ code some places ourside the src dir. In C-files Q_DECL_NOTHROW is replaced with Q_DECL_NOEXCEPT as we still need to turn it off when compiled in C mode, but can get rid of the old NOTHROW moniker. Change-Id: I6370f57066679c5120d0265a69e7e378e09d4759 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-04-09 11:33:15 +00:00
"inline WordBreakClass wordBreakClass(QChar ch) noexcept\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"{ return wordBreakClass(ch.unicode()); }\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT SentenceBreakClass QT_FASTCALL sentenceBreakClass(char32_t ucs4) noexcept;\n"
Replace Q_DECL_NOTHROW with noexcept the remaining places The first replacement had missed objective-C++ code some places ourside the src dir. In C-files Q_DECL_NOTHROW is replaced with Q_DECL_NOEXCEPT as we still need to turn it off when compiled in C mode, but can get rid of the old NOTHROW moniker. Change-Id: I6370f57066679c5120d0265a69e7e378e09d4759 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-04-09 11:33:15 +00:00
"inline SentenceBreakClass sentenceBreakClass(QChar ch) noexcept\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"{ return sentenceBreakClass(ch.unicode()); }\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT LineBreakClass QT_FASTCALL lineBreakClass(char32_t ucs4) noexcept;\n"
Replace Q_DECL_NOTHROW with noexcept the remaining places The first replacement had missed objective-C++ code some places ourside the src dir. In C-files Q_DECL_NOTHROW is replaced with Q_DECL_NOEXCEPT as we still need to turn it off when compiled in C mode, but can get rid of the old NOTHROW moniker. Change-Id: I6370f57066679c5120d0265a69e7e378e09d4759 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-04-09 11:33:15 +00:00
"inline LineBreakClass lineBreakClass(QChar ch) noexcept\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"{ return lineBreakClass(ch.unicode()); }\n"
"\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static const int SizeOfPropertiesStruct = 20;
QUnicodeTables: pack Properties struct GCC doesn't like the sequence : 5 : 5 : 8 : 6 : 8 and inserts a :6 padding between the :5 and the :8 and a :2 padding between the :6 and the :8, growing the bitfield by 8 bits of embedded padding and another byte to bring the struct back to sizeof % 2 == 0. Fix by reshuffling the elements and adding a static_assert for the next round. Saves ~5KiB in QtCore executable size. Change-Id: I4758a6f48ba389abc2aee92f60997d42ebb0e5b8 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-09-03 21:40:56 +00:00
static const QByteArray sizeOfPropertiesStructCheck =
Port Q_STATIC_ASSERT(_X) to static_assert There is no reason for keep using our macro now that we have C++17. The macro itself is left in for the moment being, as well as its detection logic, because it's needed for C code (not everything supports C11 yet). A few more cleanups will arrive in the next few patches. Note that this is a mere search/replace; some places were using double braces to work around the presence of commas in a macro, no attempt has been done to fix those. tst_qglobal had just some minor changes to keep testing the macro. Change-Id: I1c1c397d9f3e63db3338842bf350c9069ea57639 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2020-06-11 09:35:19 +00:00
"static_assert(sizeof(Properties) == " + QByteArray::number(SizeOfPropertiesStruct) + ");\n\n";
QUnicodeTables: pack Properties struct GCC doesn't like the sequence : 5 : 5 : 8 : 6 : 8 and inserts a :6 padding between the :5 and the :8 and a :2 padding between the :6 and the :8, growing the bitfield by 8 bits of embedded padding and another byte to bring the struct back to sizeof % 2 == 0. Fix by reshuffling the elements and adding a static_assert for the next round. Saves ~5KiB in QtCore executable size. Change-Id: I4758a6f48ba389abc2aee92f60997d42ebb0e5b8 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-09-03 21:40:56 +00:00
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
struct PropertyFlags {
Update the Unicode Data and Algorithms up to Unicode 6.2 Version 6.2 of the Unicode Standard is a special release dedicated to the early publication of the newly encoded Turkish lira sign. In addition, there are some significant changes to the Unicode algorithms for text segmentation and line breaking to improve breaking for emoji symbols. For more details, see http://www.unicode.org/versions/Unicode6.2.0/ Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-09-27 22:57:39 +00:00
bool operator==(const PropertyFlags &o) const {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
return (combiningClass == o.combiningClass
&& category == o.category
&& direction == o.direction
&& joining == o.joining
&& age == o.age
&& digitValue == o.digitValue
&& mirrorDiff == o.mirrorDiff
&& lowerCaseDiff == o.lowerCaseDiff
&& upperCaseDiff == o.upperCaseDiff
&& titleCaseDiff == o.titleCaseDiff
&& caseFoldDiff == o.caseFoldDiff
&& lowerCaseSpecial == o.lowerCaseSpecial
&& upperCaseSpecial == o.upperCaseSpecial
&& titleCaseSpecial == o.titleCaseSpecial
&& caseFoldSpecial == o.caseFoldSpecial
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
&& graphemeBreakClass == o.graphemeBreakClass
&& wordBreakClass == o.wordBreakClass
&& sentenceBreakClass == o.sentenceBreakClass
&& lineBreakClass == o.lineBreakClass
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
&& script == o.script
[1/2] Implement Unicode Normalization Form Quick Check (NF QC) Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt into account and generate NF QC bits. \sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-10 12:41:52 +00:00
&& nfQuickCheck == o.nfQuickCheck
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
);
}
// from UnicodeData.txt
uchar combiningClass : 8;
QChar::Category category : 5;
QChar::Direction direction : 5;
// from ArabicShaping.txt
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
QChar::JoiningType joining : 3;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// from DerivedAge.txt
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
QChar::UnicodeVersion age : 5;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int digitValue;
int mirrorDiff : 16;
int lowerCaseDiff;
int upperCaseDiff;
int titleCaseDiff;
int caseFoldDiff;
bool lowerCaseSpecial;
bool upperCaseSpecial;
bool titleCaseSpecial;
bool caseFoldSpecial;
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
GraphemeBreakClass graphemeBreakClass;
WordBreakClass wordBreakClass;
SentenceBreakClass sentenceBreakClass;
LineBreakClass lineBreakClass;
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
int script;
[1/2] Implement Unicode Normalization Form Quick Check (NF QC) Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt into account and generate NF QC bits. \sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-10 12:41:52 +00:00
// from DerivedNormalizationProps.txt
uchar nfQuickCheck;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
static QList<int> specialCaseMap;
static int appendToSpecialCaseMap(const QList<int> &map)
{
QList<int> utf16map;
for (int i = 0; i < map.size(); ++i) {
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
uint codepoint = map.at(i);
// if the condition below doesn't hold anymore we need to modify our special case mapping code
Q_ASSERT(!QChar::requiresSurrogates(codepoint));
if (QChar::requiresSurrogates(codepoint)) {
utf16map << QChar::highSurrogate(codepoint);
utf16map << QChar::lowSurrogate(codepoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
} else {
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
utf16map << codepoint;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
int length = utf16map.size();
utf16map.prepend(length);
optimize QString::toLower()/toUpper() for special cases, step 2 from now, QUnicodeTables::specialCaseMap[] starts with a placeholder; so, if somethingCaseSpecial is true, then somethingCaseDiff is always greater than 0 Change-Id: Ibb1870512836eee71b1521564c0745096c05b2f9 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:21 +00:00
if (specialCaseMap.isEmpty())
specialCaseMap << 0; // placeholder
int i = 1;
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
while (i < specialCaseMap.size()) {
int n = specialCaseMap.at(i);
if (n == length) {
int j;
for (j = 1; j <= n; ++j) {
if (specialCaseMap.at(i+j) != utf16map.at(j))
break;
}
if (j > n)
return i;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
i += n + 1;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
int pos = specialCaseMap.size();
specialCaseMap << utf16map;
return pos;
}
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
// DerivedCoreProperties.txt
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
static inline bool isDefaultIgnorable(uint ucs4)
{
// Default_Ignorable_Code_Point:
// Generated from
// Other_Default_Ignorable_Code_Point + Cf + Variation_Selector
// - White_Space - FFF9..FFFB (Annotation Characters)
// - 0600..0604, 06DD, 070F, 110BD (exceptional Cf characters that should be visible)
if (ucs4 <= 0xff)
return ucs4 == 0xad;
return ucs4 == 0x034f
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
|| ucs4 == 0x061c
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
|| (ucs4 >= 0x115f && ucs4 <= 0x1160)
|| (ucs4 >= 0x17b4 && ucs4 <= 0x17b5)
|| (ucs4 >= 0x180b && ucs4 <= 0x180d)
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
|| ucs4 == 0x180e
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
|| (ucs4 >= 0x200b && ucs4 <= 0x200f)
|| (ucs4 >= 0x202a && ucs4 <= 0x202e)
|| (ucs4 >= 0x2060 && ucs4 <= 0x206f)
|| ucs4 == 0x3164
|| (ucs4 >= 0xfe00 && ucs4 <= 0xfe0f)
|| ucs4 == 0xfeff
|| ucs4 == 0xffa0
|| (ucs4 >= 0xfff0 && ucs4 <= 0xfff8)
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
|| (ucs4 >= 0x1bca0 && ucs4 <= 0x1bca3)
|| (ucs4 >= 0x1d173 && ucs4 <= 0x1d17a)
|| (ucs4 >= 0xe0000 && ucs4 <= 0xe0fff);
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
struct UnicodeData {
UnicodeData(int codepoint = 0) {
p.category = QChar::Other_NotAssigned; // Cn
p.combiningClass = 0;
p.direction = QChar::DirL;
// DerivedBidiClass.txt
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
// The unassigned code points that default to AL are in the ranges:
// [U+0600..U+07BF, U+08A0..U+08FF, U+FB50..U+FDCF, U+FDF0..U+FDFF, U+FE70..U+FEFF, U+1EE00..U+1EEFF]
if ((codepoint >= 0x0600 && codepoint <= 0x07BF)
|| (codepoint >= 0x08A0 && codepoint <= 0x08FF)
|| (codepoint >= 0xFB50 && codepoint <= 0xFDCF)
|| (codepoint >= 0xFDF0 && codepoint <= 0xFDFF)
|| (codepoint >= 0xFE70 && codepoint <= 0xFEFF)
|| (codepoint >= 0x1EE00 && codepoint <= 0x1EEFF)) {
p.direction = QChar::DirAL;
}
// The unassigned code points that default to R are in the ranges:
// [U+0590..U+05FF, U+07C0..U+089F, U+FB1D..U+FB4F, U+10800..U+10FFF, U+1E800..U+1EDFF, U+1EF00..U+1EFFF]
else if ((codepoint >= 0x0590 && codepoint <= 0x05FF)
|| (codepoint >= 0x07C0 && codepoint <= 0x089F)
|| (codepoint >= 0xFB1D && codepoint <= 0xFB4F)
|| (codepoint >= 0x10800 && codepoint <= 0x10FFF)
|| (codepoint >= 0x1E800 && codepoint <= 0x1EDFF)
|| (codepoint >= 0x1EF00 && codepoint <= 0x1EFFF)) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
p.direction = QChar::DirR;
}
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
// The unassigned code points that default to ET are in the range:
// [U+20A0..U+20CF]
else if (codepoint >= 0x20A0 && codepoint <= 0x20CF) {
p.direction = QChar::DirET;
}
// The unassigned code points that default to BN have one of the following properties:
// Default_Ignorable_Code_Point
// Noncharacter_Code_Point
else if (QChar::isNonCharacter(codepoint) || isDefaultIgnorable(codepoint)) {
p.direction = QChar::DirBN;
}
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
p.lineBreakClass = LineBreak_AL; // XX -> AL
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
// LineBreak.txt
// The unassigned code points that default to "ID" include ranges in the following blocks:
Update Unicode data & algorithms up to v8.0 * Georgian lari currency symbol * A large collection of CJK unified ideographs * Emoji symbols and symbol modifiers * Letters to support the Ik language in Uganda, Kulango in the Côte d’Ivoire, and other languages of Africa * A set of lowercase Cherokee syllables, forming case pairs with the existing Cherokee characters * The Ahom script for support of the Tai Ahom language in India * Arabic letters to support Arwi—the Tamil language written in the Arabic script For more details, see http://www.unicode.org/versions/Unicode8.0.0/ [ChangeLog][QtCore] Unicode data updated to v.8.0 Change-Id: If255f95c9c45655b721369a116299da3cabbba0a Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-02 04:28:14 +00:00
// [U+3400..U+4DBF, U+4E00..U+9FFF, U+F900..U+FAFF, U+20000..U+2A6DF, U+2A700..U+2B73F, U+2B740..U+2B81F, U+2B820..U+2CEAF, U+2F800..U+2FA1F]
// and any other reserved code points on
// [U+20000..U+2FFFD, U+30000..U+3FFFD]
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
if ((codepoint >= 0x3400 && codepoint <= 0x4DBF)
|| (codepoint >= 0x4E00 && codepoint <= 0x9FFF)
|| (codepoint >= 0xF900 && codepoint <= 0xFAFF)
|| (codepoint >= 0x20000 && codepoint <= 0x2A6DF)
|| (codepoint >= 0x2A700 && codepoint <= 0x2B73F)
|| (codepoint >= 0x2B740 && codepoint <= 0x2B81F)
Update Unicode data & algorithms up to v8.0 * Georgian lari currency symbol * A large collection of CJK unified ideographs * Emoji symbols and symbol modifiers * Letters to support the Ik language in Uganda, Kulango in the Côte d’Ivoire, and other languages of Africa * A set of lowercase Cherokee syllables, forming case pairs with the existing Cherokee characters * The Ahom script for support of the Tai Ahom language in India * Arabic letters to support Arwi—the Tamil language written in the Arabic script For more details, see http://www.unicode.org/versions/Unicode8.0.0/ [ChangeLog][QtCore] Unicode data updated to v.8.0 Change-Id: If255f95c9c45655b721369a116299da3cabbba0a Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-02 04:28:14 +00:00
|| (codepoint >= 0x2B820 && codepoint <= 0x2CEAF)
Update the qunicodetables generator to deal with UCD 6.1 files Change-Id: If22018ff83cfc6b9c984f689648da038fce11d84 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-03 01:17:10 +00:00
|| (codepoint >= 0x2F800 && codepoint <= 0x2FA1F)
|| (codepoint >= 0x20000 && codepoint <= 0x2FFFD)
|| (codepoint >= 0x30000 && codepoint <= 0x3FFFD)) {
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
p.lineBreakClass = LineBreak_ID;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
// The unassigned code points that default to "PR" comprise a range in the following block:
// [U+20A0..U+20CF]
else if (codepoint >= 0x20A0 && codepoint <= 0x20CF) {
p.lineBreakClass = LineBreak_PR;
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
mirroredChar = 0;
decompositionType = QChar::NoDecomposition;
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
p.joining = QChar::Joining_None;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
p.age = QChar::Unicode_Unassigned;
p.mirrorDiff = 0;
p.digitValue = -1;
p.lowerCaseDiff = 0;
p.upperCaseDiff = 0;
p.titleCaseDiff = 0;
p.caseFoldDiff = 0;
p.lowerCaseSpecial = 0;
p.upperCaseSpecial = 0;
p.titleCaseSpecial = 0;
p.caseFoldSpecial = 0;
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
p.graphemeBreakClass = GraphemeBreak_Any;
p.wordBreakClass = WordBreak_Any;
p.sentenceBreakClass = SentenceBreak_Any;
Add QChar::Script enum ...where the values are not aliased to Common script. The old QUnicodeTables::Script enum was retained for compatibility reasons until Qt internals are updated to use QChar::script(). Using QChar::Script instead of QUnicodeTables::Script would improve both the text analysis (itemization, boundary finding) and the text shaping quality. This also a required step for switching to Hurfbuzz-NG. /* This adds 6668 more .rodata bytes */ Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-08 03:36:49 +00:00
p.script = QChar::Script_Unknown;
[1/2] Implement Unicode Normalization Form Quick Check (NF QC) Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt into account and generate NF QC bits. \sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-10 12:41:52 +00:00
p.nfQuickCheck = 0;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
propertyIndex = -1;
excludedComposition = false;
}
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
static UnicodeData &valueRef(int codepoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
PropertyFlags p;
// from UnicodeData.txt
QChar::Decomposition decompositionType;
QList<int> decomposition;
QList<int> specialFolding;
// from BidiMirroring.txt
int mirroredChar;
// DerivedNormalizationProps.txt
bool excludedComposition;
// computed position of unicode property set
int propertyIndex;
};
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
static QList<UnicodeData> unicodeData;
UnicodeData &UnicodeData::valueRef(int codepoint)
{
static bool initialized = false;
if (!initialized) {
QChar: add isSurrogate() and isNonCharacter() to the public API + QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro replace uses of hardcoded values with the new API; remove leftovers Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2012-05-15 17:48:20 +00:00
unicodeData.reserve(QChar::LastValidCodePoint + 1);
for (int uc = 0; uc <= QChar::LastValidCodePoint; ++uc)
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
unicodeData.append(UnicodeData(uc));
initialized = true;
}
Q_ASSERT(codepoint <= 0x10ffff);
return unicodeData[codepoint];
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static QHash<int, int> decompositionLength;
static int highestComposedCharacter = 0;
static int numLigatures = 0;
static int highestLigature = 0;
struct Ligature {
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
int u1;
int u2;
int ligature;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
// we need them sorted after the first component for fast lookup
bool operator < (const Ligature &l1, const Ligature &l2)
{ return l1.u1 < l2.u1; }
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
static QHash<int, QList<Ligature> > ligatureHashes;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
static QHash<int, int> combiningClassUsage;
static int maxLowerCaseDiff = 0;
static int maxUpperCaseDiff = 0;
static int maxTitleCaseDiff = 0;
static void readUnicodeData()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading UnicodeData.txt");
enum UniDataFields {
UD_Value,
UD_Name,
UD_Category,
UD_CombiningClass,
UD_BidiCategory,
UD_Decomposition,
UD_DecimalDigitValue,
UD_DigitValue,
UD_NumericValue,
UD_Mirrored,
UD_OldName,
UD_Comment,
UD_UpperCase,
UD_LowerCase,
UD_TitleCase
};
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/UnicodeData.txt");
if (!f.exists())
qFatal("Couldn't find UnicodeData.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.truncate(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
if (line.isEmpty())
continue;
QList<QByteArray> properties = line.split(';');
bool ok;
int codepoint = properties[UD_Value].toInt(&ok, 16);
Q_ASSERT(ok);
QChar: add isSurrogate() and isNonCharacter() to the public API + QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro replace uses of hardcoded values with the new API; remove leftovers Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2012-05-15 17:48:20 +00:00
Q_ASSERT(codepoint <= QChar::LastValidCodePoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int lastCodepoint = codepoint;
QByteArray name = properties[UD_Name];
if (name.startsWith('<') && name.contains("First")) {
QByteArray nextLine;
nextLine.resize(1024);
f.readLine(nextLine.data(), 1024);
QList<QByteArray> properties = nextLine.split(';');
Q_ASSERT(properties[UD_Name].startsWith('<') && properties[UD_Name].contains("Last"));
lastCodepoint = properties[UD_Value].toInt(&ok, 16);
Q_ASSERT(ok);
QChar: add isSurrogate() and isNonCharacter() to the public API + QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro replace uses of hardcoded values with the new API; remove leftovers Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2012-05-15 17:48:20 +00:00
Q_ASSERT(lastCodepoint <= QChar::LastValidCodePoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &data = UnicodeData::valueRef(codepoint);
drop an obsolete QChar::NoCategory enum value there is no such category in the Unicode specs. the QChar::NoCategory was a subject of bugs since it was introduced. int 4.6 it's meaning was limited to mention ucs4 > UNICODE_LAST_CODEPOINT only (which is useless anyways) in order to preserve the old (wrong) behavior. fix it now for qtbase Change-Id: I630534824e071090b39772881e747c1fdb758719 Reviewed-on: http://codereview.qt.nokia.com/1584 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2011-07-08 14:24:57 +00:00
data.p.category = categoryMap.value(properties[UD_Category], QChar::Other_NotAssigned);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
data.p.combiningClass = properties[UD_CombiningClass].toInt();
if (!combiningClassUsage.contains(data.p.combiningClass))
combiningClassUsage[data.p.combiningClass] = 1;
else
++combiningClassUsage[data.p.combiningClass];
Update the Unicode Data and Algorithms up to Unicode 6.3.0 * Mongolian and Phags-pa characters have been given a Joining_Type classification for contextual shaping. As a part of these additions, one Phags-pa character has the Joining_Type value of L (Left Joining), which no character had been assigned before. * The unassigned code points in the Currency Symbols block have been given the Bidi_Class property value ET and the Line_Break property value PR, to help implementations support new currency symbols, when they are encoded. * Hebrew letters and basic punctuation marks have been assigned the newly introduced Word_Break property values Hebrew_Letter, Single_Quote, and Double_Quote. * The Bidi_Class property has been extended with four new values for directional isolates. For more details, see http://www.unicode.org/versions/Unicode6.3.0/ Change-Id: Iad62d02edc58a8497898dcd6d6c70d5aece317ea Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2014-01-12 19:14:25 +00:00
Direction dir = directionMap.value(properties[UD_BidiCategory], Dir_Unassigned);
if (dir == Dir_Unassigned)
qFatal("unhandled direction value: %s", properties[UD_BidiCategory].constData());
data.p.direction = QChar::Direction(dir);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (!properties[UD_UpperCase].isEmpty()) {
int upperCase = properties[UD_UpperCase].toInt(&ok, 16);
Q_ASSERT(ok);
int diff = upperCase - codepoint;
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
// if the conditions below doesn't hold anymore we need to modify our upper casing code
Q_ASSERT(QChar::requiresSurrogates(codepoint) == QChar::requiresSurrogates(upperCase));
if (QChar::requiresSurrogates(codepoint)) {
Q_ASSERT(QChar::highSurrogate(codepoint) == QChar::highSurrogate(upperCase));
Q_ASSERT(QChar::lowSurrogate(codepoint) + diff == QChar::lowSurrogate(upperCase));
}
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
if (qAbs(diff) >= (1<<13)) {
workaround issue where casing diff overflows signed short there are two such codepoints were added in the Unicode 5.1: U+1D79 LATIN SMALL LETTER INSULAR G U+A77D LATIN CAPITAL LETTER INSULAR G two more of them were added in the Unicode 6.0: U+0265 LATIN SMALL LETTER TURNED H U+A78D LATIN CAPITAL LETTER TURNED H and two more were added in the Unicode 6.1: U+0266 LATIN SMALL LETTER H WITH HOOK U+A7AA LATIN CAPITAL LETTER H WITH HOOK we map them like special cases with length == 1 (note: all are in BMP which is checked explicitly in the generator) Change-Id: I8a34164eb3ee2e575b7799cc12d4b96ad5bcd9c6 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-23 15:11:57 +00:00
qWarning() << "upperCaseDiff exceeded (" << hex << codepoint << "->" << upperCase << "); map it for special case";
data.p.upperCaseSpecial = true;
data.p.upperCaseDiff = appendToSpecialCaseMap(QList<int>() << upperCase);
} else {
data.p.upperCaseDiff = diff;
maxUpperCaseDiff = qMax(maxUpperCaseDiff, qAbs(diff));
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
if (!properties[UD_LowerCase].isEmpty()) {
int lowerCase = properties[UD_LowerCase].toInt(&ok, 16);
Q_ASSERT(ok);
int diff = lowerCase - codepoint;
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
// if the conditions below doesn't hold anymore we need to modify our lower casing code
Q_ASSERT(QChar::requiresSurrogates(codepoint) == QChar::requiresSurrogates(lowerCase));
if (QChar::requiresSurrogates(codepoint)) {
Q_ASSERT(QChar::highSurrogate(codepoint) == QChar::highSurrogate(lowerCase));
Q_ASSERT(QChar::lowSurrogate(codepoint) + diff == QChar::lowSurrogate(lowerCase));
}
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
if (qAbs(diff) >= (1<<13)) {
workaround issue where casing diff overflows signed short there are two such codepoints were added in the Unicode 5.1: U+1D79 LATIN SMALL LETTER INSULAR G U+A77D LATIN CAPITAL LETTER INSULAR G two more of them were added in the Unicode 6.0: U+0265 LATIN SMALL LETTER TURNED H U+A78D LATIN CAPITAL LETTER TURNED H and two more were added in the Unicode 6.1: U+0266 LATIN SMALL LETTER H WITH HOOK U+A7AA LATIN CAPITAL LETTER H WITH HOOK we map them like special cases with length == 1 (note: all are in BMP which is checked explicitly in the generator) Change-Id: I8a34164eb3ee2e575b7799cc12d4b96ad5bcd9c6 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-23 15:11:57 +00:00
qWarning() << "lowerCaseDiff exceeded (" << hex << codepoint << "->" << lowerCase << "); map it for special case";
data.p.lowerCaseSpecial = true;
data.p.lowerCaseDiff = appendToSpecialCaseMap(QList<int>() << lowerCase);
} else {
data.p.lowerCaseDiff = diff;
maxLowerCaseDiff = qMax(maxLowerCaseDiff, qAbs(diff));
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
// we want toTitleCase to map to ToUpper in case we don't have any titlecase.
if (properties[UD_TitleCase].isEmpty())
properties[UD_TitleCase] = properties[UD_UpperCase];
if (!properties[UD_TitleCase].isEmpty()) {
int titleCase = properties[UD_TitleCase].toInt(&ok, 16);
Q_ASSERT(ok);
int diff = titleCase - codepoint;
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
// if the conditions below doesn't hold anymore we need to modify our title casing code
Q_ASSERT(QChar::requiresSurrogates(codepoint) == QChar::requiresSurrogates(titleCase));
if (QChar::requiresSurrogates(codepoint)) {
Q_ASSERT(QChar::highSurrogate(codepoint) == QChar::highSurrogate(titleCase));
Q_ASSERT(QChar::lowSurrogate(codepoint) + diff == QChar::lowSurrogate(titleCase));
}
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
if (qAbs(diff) >= (1<<13)) {
workaround issue where casing diff overflows signed short there are two such codepoints were added in the Unicode 5.1: U+1D79 LATIN SMALL LETTER INSULAR G U+A77D LATIN CAPITAL LETTER INSULAR G two more of them were added in the Unicode 6.0: U+0265 LATIN SMALL LETTER TURNED H U+A78D LATIN CAPITAL LETTER TURNED H and two more were added in the Unicode 6.1: U+0266 LATIN SMALL LETTER H WITH HOOK U+A7AA LATIN CAPITAL LETTER H WITH HOOK we map them like special cases with length == 1 (note: all are in BMP which is checked explicitly in the generator) Change-Id: I8a34164eb3ee2e575b7799cc12d4b96ad5bcd9c6 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-23 15:11:57 +00:00
qWarning() << "titleCaseDiff exceeded (" << hex << codepoint << "->" << titleCase << "); map it for special case";
data.p.titleCaseSpecial = true;
data.p.titleCaseDiff = appendToSpecialCaseMap(QList<int>() << titleCase);
} else {
data.p.titleCaseDiff = diff;
maxTitleCaseDiff = qMax(maxTitleCaseDiff, qAbs(diff));
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
if (!properties[UD_DigitValue].isEmpty())
data.p.digitValue = properties[UD_DigitValue].toInt();
// decompositition
QByteArray decomposition = properties[UD_Decomposition];
if (!decomposition.isEmpty()) {
highestComposedCharacter = qMax(highestComposedCharacter, codepoint);
QList<QByteArray> d = decomposition.split(' ');
if (d[0].contains('<')) {
data.decompositionType = decompositionMap.value(d[0], QChar::NoDecomposition);
if (data.decompositionType == QChar::NoDecomposition)
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
qFatal("unhandled decomposition type: %s", d[0].constData());
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
d.takeFirst();
} else {
data.decompositionType = QChar::Canonical;
}
for (int i = 0; i < d.size(); ++i) {
data.decomposition.append(d[i].toInt(&ok, 16));
Q_ASSERT(ok);
}
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
++decompositionLength[data.decomposition.size()];
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
for (int i = codepoint; i <= lastCodepoint; ++i)
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
unicodeData[i] = data;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
static int maxMirroredDiff = 0;
static void readBidiMirroring()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading BidiMirroring.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/BidiMirroring.txt");
if (!f.exists())
qFatal("Couldn't find BidiMirroring.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
if (line.isEmpty())
continue;
line = line.replace(" ", "");
QList<QByteArray> pair = line.split(';');
Q_ASSERT(pair.size() == 2);
bool ok;
int codepoint = pair[0].toInt(&ok, 16);
Q_ASSERT(ok);
int mirror = pair[1].toInt(&ok, 16);
Q_ASSERT(ok);
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(codepoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
d.mirroredChar = mirror;
d.p.mirrorDiff = d.mirroredChar - codepoint;
maxMirroredDiff = qMax(maxMirroredDiff, qAbs(d.p.mirrorDiff));
}
}
static void readArabicShaping()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading ArabicShaping.txt");
Fix a bug in the generation of the Unicode joining property data The code that was supposed to initialize the joining property of characters to the correct defaults was actually applied after reading in the data from the Unicode file, and was in a couple of cases overwriting explicitly specified data in ArabicShaping.txt Task-number: QTBUG-63191 Change-Id: Ie35261039b2211a827322ca11afacd9555ccefc7 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2017-12-11 10:28:11 +00:00
// Initialize defaults:
// Code points that are not explicitly listed in ArabicShaping.txt are either of joining type T or U:
// - Those that not explicitly listed that are of General Category Mn, Me, or Cf have joining type T.
// - All others not explicitly listed have joining type U.
for (int codepoint = 0; codepoint <= QChar::LastValidCodePoint; ++codepoint) {
UnicodeData &d = UnicodeData::valueRef(codepoint);
if (d.p.joining == QChar::Joining_None) {
if (d.p.category == QChar::Mark_NonSpacing || d.p.category == QChar::Mark_Enclosing || d.p.category == QChar::Other_Format)
d.p.joining = QChar::Joining_Transparent;
}
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/ArabicShaping.txt");
if (!f.exists())
qFatal("Couldn't find ArabicShaping.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
line = line.trimmed();
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
Q_ASSERT(l.size() == 4);
bool ok;
int codepoint = l[0].toInt(&ok, 16);
Q_ASSERT(ok);
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
UnicodeData &d = UnicodeData::valueRef(codepoint);
JoiningType joining = joining_map.value(l[2].trimmed(), Joining_Unassigned);
switch (joining) {
case Joining_Unassigned:
qFatal("%x: unassigned or unhandled joining type: %s", codepoint, l[2].constData());
break;
case Joining_Transparent:
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
switch (d.p.category) {
case QChar::Mark_Enclosing:
case QChar::Mark_NonSpacing:
case QChar::Letter_Modifier:
case QChar::Other_Format:
break;
default:
qFatal("%x: joining type '%s' was met (category: %d); "
"the current implementation needs to be revised!",
codepoint, l[2].constData(), d.p.category);
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
}
Update UCD data to Unicode 12.1.0's Revision 24 Had to teach the update program to accept category Lm as for Joining_Transparent, for the sake of a new ArabicShaping.txt entry. Added three new Unicode versions, several new scripts and a new word-break class. Updated UCD's test data for tst_QTextBoundaryFinder. This left 57 tests failing; I have commented out the data rows for those tests, pending someone with more knowledge addressing this. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2019-10-23 15:17:49 +00:00
Q_FALLTHROUGH();
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
default:
d.p.joining = QChar::JoiningType(joining);
break;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
static void readDerivedAge()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading DerivedAge.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/DerivedAge.txt");
if (!f.exists())
qFatal("Couldn't find DerivedAge.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
line.replace(" ", "");
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
Q_ASSERT(l.size() == 2);
QByteArray codes = l[0];
codes.replace("..", ".");
QList<QByteArray> cl = codes.split('.');
bool ok;
int from = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int to = from;
if (cl.size() == 2) {
to = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
QChar::UnicodeVersion age = age_map.value(l[1].trimmed(), QChar::Unicode_Unassigned);
//qDebug() << hex << from << ".." << to << ba << age;
if (age == QChar::Unicode_Unassigned)
qFatal("unassigned or unhandled age value: %s", l[1].constData());
for (int codepoint = from; codepoint <= to; ++codepoint) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(codepoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
d.p.age = age;
}
}
}
static void readDerivedNormalizationProps()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading DerivedNormalizationProps.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/DerivedNormalizationProps.txt");
if (!f.exists())
qFatal("Couldn't find DerivedNormalizationProps.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
if (line.trimmed().isEmpty())
continue;
QList<QByteArray> l = line.split(';');
Q_ASSERT(l.size() >= 2);
QByteArray propName = l[1].trimmed();
[1/2] Implement Unicode Normalization Form Quick Check (NF QC) Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt into account and generate NF QC bits. \sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-10 12:41:52 +00:00
if (propName != "Full_Composition_Exclusion" &&
propName != "NFD_QC" && propName != "NFC_QC" &&
propName != "NFKD_QC" && propName != "NFKC_QC") {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// ###
continue;
[1/2] Implement Unicode Normalization Form Quick Check (NF QC) Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt into account and generate NF QC bits. \sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-10 12:41:52 +00:00
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QByteArray codes = l[0].trimmed();
codes.replace("..", ".");
QList<QByteArray> cl = codes.split('.');
bool ok;
int from = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int to = from;
if (cl.size() == 2) {
to = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
for (int codepoint = from; codepoint <= to; ++codepoint) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(codepoint);
[1/2] Implement Unicode Normalization Form Quick Check (NF QC) Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt into account and generate NF QC bits. \sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-10 12:41:52 +00:00
if (propName == "Full_Composition_Exclusion") {
d.excludedComposition = true;
} else {
Port Q_STATIC_ASSERT(_X) to static_assert There is no reason for keep using our macro now that we have C++17. The macro itself is left in for the moment being, as well as its detection logic, because it's needed for C code (not everything supports C11 yet). A few more cleanups will arrive in the next few patches. Note that this is a mere search/replace; some places were using double braces to work around the presence of commas in a macro, no attempt has been done to fix those. tst_qglobal had just some minor changes to keep testing the macro. Change-Id: I1c1c397d9f3e63db3338842bf350c9069ea57639 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2020-06-11 09:35:19 +00:00
static_assert(QString::NormalizationForm_D == 0);
static_assert(QString::NormalizationForm_C == 1);
static_assert(QString::NormalizationForm_KD == 2);
static_assert(QString::NormalizationForm_KC == 3);
[1/2] Implement Unicode Normalization Form Quick Check (NF QC) Make Unicode tables generator take QuickCheck data from DerivedNormalizationProps.txt into account and generate NF QC bits. \sa http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms Change-Id: Ib73bd42ddb8f99d0be0aff609711943c52dd9c24 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2013-08-10 12:41:52 +00:00
QString::NormalizationForm form;
if (propName == "NFD_QC")
form = QString::NormalizationForm_D;
else if (propName == "NFC_QC")
form = QString::NormalizationForm_C;
else if (propName == "NFKD_QC")
form = QString::NormalizationForm_KD;
else// if (propName == "NFKC_QC")
form = QString::NormalizationForm_KC;
Q_ASSERT(l.size() == 3);
l[2] = l[2].trimmed();
enum { NFQC_YES = 0, NFQC_NO = 1, NFQC_MAYBE = 3 };
uchar ynm = (l[2] == "N" ? NFQC_NO : l[2] == "M" ? NFQC_MAYBE : NFQC_YES);
if (ynm == NFQC_MAYBE) {
// if this changes, we need to revise the normalizationQuickCheckHelper() implementation
Q_ASSERT(form == QString::NormalizationForm_C || form == QString::NormalizationForm_KC);
}
d.p.nfQuickCheck |= (ynm << (form << 1)); // 2 bits per NF
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
QChar: add isSurrogate() and isNonCharacter() to the public API + QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro replace uses of hardcoded values with the new API; remove leftovers Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2012-05-15 17:48:20 +00:00
for (int codepoint = 0; codepoint <= QChar::LastValidCodePoint; ++codepoint) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(codepoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (!d.excludedComposition
&& d.decompositionType == QChar::Canonical
&& d.decomposition.size() > 1) {
Q_ASSERT(d.decomposition.size() == 2);
int part1 = d.decomposition.at(0);
int part2 = d.decomposition.at(1);
// all non-starters are listed in DerivedNormalizationProps.txt
// and already excluded from composition
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
Q_ASSERT(UnicodeData::valueRef(part1).p.combiningClass == 0);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
++numLigatures;
highestLigature = qMax(highestLigature, part1);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
Ligature l = { part1, part2, codepoint };
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
ligatureHashes[part2].append(l);
}
}
}
struct NormalizationCorrection {
uint codepoint;
uint mapped;
minor improvement for NormalizationCorrections let's don't hardcode the latests affected version value and simply use the one parsed from NormalizationCorrections.txt Change-Id: I37021e8238d77deada4c5ba7a2d160c87186b9dd Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-04 23:28:37 +00:00
int version;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
static QByteArray createNormalizationCorrections()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading NormalizationCorrections.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/NormalizationCorrections.txt");
if (!f.exists())
qFatal("Couldn't find NormalizationCorrections.txt");
f.open(QFile::ReadOnly);
QByteArray out;
out += "struct NormalizationCorrection {\n"
" uint ucs4;\n"
" uint old_mapping;\n"
" int version;\n"
"};\n\n"
"static const NormalizationCorrection uc_normalization_corrections[] = {\n";
minor improvement for NormalizationCorrections let's don't hardcode the latests affected version value and simply use the one parsed from NormalizationCorrections.txt Change-Id: I37021e8238d77deada4c5ba7a2d160c87186b9dd Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-04 23:28:37 +00:00
int maxVersion = 0;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int numCorrections = 0;
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
line.replace(" ", "");
if (line.isEmpty())
continue;
Q_ASSERT(!line.contains(".."));
QList<QByteArray> fields = line.split(';');
Q_ASSERT(fields.size() == 4);
NormalizationCorrection c = { 0, 0, 0 };
bool ok;
c.codepoint = fields.at(0).toInt(&ok, 16);
Q_ASSERT(ok);
c.mapped = fields.at(1).toInt(&ok, 16);
Q_ASSERT(ok);
if (fields.at(3) == "3.2.0")
c.version = QChar::Unicode_3_2;
else if (fields.at(3) == "4.0.0")
c.version = QChar::Unicode_4_0;
else
qFatal("unknown unicode version in NormalizationCorrection.txt");
out += " { 0x" + QByteArray::number(c.codepoint, 16) + ", 0x" + QByteArray::number(c.mapped, 16)
+ ", " + QString::number(c.version) + " },\n";
++numCorrections;
minor improvement for NormalizationCorrections let's don't hardcode the latests affected version value and simply use the one parsed from NormalizationCorrections.txt Change-Id: I37021e8238d77deada4c5ba7a2d160c87186b9dd Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-04 23:28:37 +00:00
maxVersion = qMax(c.version, maxVersion);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
if (out.endsWith(",\n"))
out.chop(2);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
out += "\n};\n\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
minor improvement for NormalizationCorrections let's don't hardcode the latests affected version value and simply use the one parsed from NormalizationCorrections.txt Change-Id: I37021e8238d77deada4c5ba7a2d160c87186b9dd Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-04 23:28:37 +00:00
"enum { NumNormalizationCorrections = " + QByteArray::number(numCorrections) + " };\n"
"enum { NormalizationCorrectionsVersionMax = " + QByteArray::number(maxVersion) + " };\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
return out;
}
static void readLineBreak()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading LineBreak.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/LineBreak.txt");
if (!f.exists())
qFatal("Couldn't find LineBreak.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
line.replace(" ", "");
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
Q_ASSERT(l.size() == 2);
QByteArray codes = l[0];
codes.replace("..", ".");
QList<QByteArray> cl = codes.split('.');
bool ok;
int from = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int to = from;
if (cl.size() == 2) {
to = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
LineBreakClass lb = line_break_map.value(l[1], LineBreak_Unassigned);
if (lb == LineBreak_Unassigned)
qFatal("unassigned line break class: %s", l[1].constData());
for (int codepoint = from; codepoint <= to; ++codepoint) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(codepoint);
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
d.p.lineBreakClass = lb;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
}
static void readSpecialCasing()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading SpecialCasing.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/SpecialCasing.txt");
if (!f.exists())
qFatal("Couldn't find SpecialCasing.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
QByteArray condition = l.size() < 5 ? QByteArray() : l[4].trimmed();
if (!condition.isEmpty())
// #####
continue;
bool ok;
int codepoint = l[0].trimmed().toInt(&ok, 16);
Q_ASSERT(ok);
// if the condition below doesn't hold anymore we need to modify our
// lower/upper/title casing code and case folding code
replace hardcoded values with a surrogate handling methods Change-Id: Iba079953c46a29404232d2dacbe0c90170097d51 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-08 07:18:45 +00:00
Q_ASSERT(!QChar::requiresSurrogates(codepoint));
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// qDebug() << "codepoint" << hex << codepoint;
// qDebug() << line;
QList<QByteArray> lower = l[1].trimmed().split(' ');
QList<int> lowerMap;
for (int i = 0; i < lower.size(); ++i) {
bool ok;
lowerMap.append(lower.at(i).toInt(&ok, 16));
Q_ASSERT(ok);
}
QList<QByteArray> title = l[2].trimmed().split(' ');
QList<int> titleMap;
for (int i = 0; i < title.size(); ++i) {
bool ok;
titleMap.append(title.at(i).toInt(&ok, 16));
Q_ASSERT(ok);
}
QList<QByteArray> upper = l[3].trimmed().split(' ');
QList<int> upperMap;
for (int i = 0; i < upper.size(); ++i) {
bool ok;
upperMap.append(upper.at(i).toInt(&ok, 16));
Q_ASSERT(ok);
}
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &ud = UnicodeData::valueRef(codepoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Q_ASSERT(lowerMap.size() > 1 || lowerMap.at(0) == codepoint + ud.p.lowerCaseDiff);
Q_ASSERT(titleMap.size() > 1 || titleMap.at(0) == codepoint + ud.p.titleCaseDiff);
Q_ASSERT(upperMap.size() > 1 || upperMap.at(0) == codepoint + ud.p.upperCaseDiff);
if (lowerMap.size() > 1) {
ud.p.lowerCaseSpecial = true;
ud.p.lowerCaseDiff = appendToSpecialCaseMap(lowerMap);
}
if (titleMap.size() > 1) {
ud.p.titleCaseSpecial = true;
ud.p.titleCaseDiff = appendToSpecialCaseMap(titleMap);
}
if (upperMap.size() > 1) {
ud.p.upperCaseSpecial = true;
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
ud.p.upperCaseDiff = appendToSpecialCaseMap(upperMap);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
}
static int maxCaseFoldDiff = 0;
static void readCaseFolding()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading CaseFolding.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/CaseFolding.txt");
if (!f.exists())
qFatal("Couldn't find CaseFolding.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
bool ok;
int codepoint = l[0].trimmed().toInt(&ok, 16);
Q_ASSERT(ok);
l[1] = l[1].trimmed();
if (l[1] == "F" || l[1] == "T")
continue;
// qDebug() << "codepoint" << hex << codepoint;
// qDebug() << line;
QList<QByteArray> fold = l[2].trimmed().split(' ');
QList<int> foldMap;
for (int i = 0; i < fold.size(); ++i) {
bool ok;
foldMap.append(fold.at(i).toInt(&ok, 16));
Q_ASSERT(ok);
}
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &ud = UnicodeData::valueRef(codepoint);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (foldMap.size() == 1) {
int caseFolded = foldMap.at(0);
int diff = caseFolded - codepoint;
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
// if the conditions below doesn't hold anymore we need to modify our case folding code
Q_ASSERT(QChar::requiresSurrogates(codepoint) == QChar::requiresSurrogates(caseFolded));
if (QChar::requiresSurrogates(codepoint)) {
Q_ASSERT(QChar::highSurrogate(codepoint) == QChar::highSurrogate(caseFolded));
Q_ASSERT(QChar::lowSurrogate(codepoint) + diff == QChar::lowSurrogate(caseFolded));
}
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
if (qAbs(diff) >= (1<<13)) {
workaround issue where casing diff overflows signed short there are two such codepoints were added in the Unicode 5.1: U+1D79 LATIN SMALL LETTER INSULAR G U+A77D LATIN CAPITAL LETTER INSULAR G two more of them were added in the Unicode 6.0: U+0265 LATIN SMALL LETTER TURNED H U+A78D LATIN CAPITAL LETTER TURNED H and two more were added in the Unicode 6.1: U+0266 LATIN SMALL LETTER H WITH HOOK U+A7AA LATIN CAPITAL LETTER H WITH HOOK we map them like special cases with length == 1 (note: all are in BMP which is checked explicitly in the generator) Change-Id: I8a34164eb3ee2e575b7799cc12d4b96ad5bcd9c6 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-23 15:11:57 +00:00
qWarning() << "caseFoldDiff exceeded (" << hex << codepoint << "->" << caseFolded << "); map it for special case";
ud.p.caseFoldSpecial = true;
ud.p.caseFoldDiff = appendToSpecialCaseMap(foldMap);
} else {
ud.p.caseFoldDiff = diff;
maxCaseFoldDiff = qMax(maxCaseFoldDiff, qAbs(diff));
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
} else {
qFatal("we currently don't support full case foldings");
// qDebug() << "special" << hex << foldMap;
ud.p.caseFoldSpecial = true;
ud.p.caseFoldDiff = appendToSpecialCaseMap(foldMap);
}
}
}
static void readGraphemeBreak()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading GraphemeBreakProperty.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/GraphemeBreakProperty.txt");
if (!f.exists())
qFatal("Couldn't find GraphemeBreakProperty.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
line.replace(" ", "");
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
Q_ASSERT(l.size() == 2);
QByteArray codes = l[0];
codes.replace("..", ".");
QList<QByteArray> cl = codes.split('.');
bool ok;
int from = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int to = from;
if (cl.size() == 2) {
to = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
GraphemeBreakClass brk = grapheme_break_map.value(l[1], GraphemeBreak_Unassigned);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (brk == GraphemeBreak_Unassigned)
qFatal("unassigned grapheme break class: %s", l[1].constData());
for (int codepoint = from; codepoint <= to; ++codepoint) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &ud = UnicodeData::valueRef(codepoint);
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
ud.p.graphemeBreakClass = brk;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
}
static void readWordBreak()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading WordBreakProperty.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/WordBreakProperty.txt");
if (!f.exists())
qFatal("Couldn't find WordBreakProperty.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
line.replace(" ", "");
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
Q_ASSERT(l.size() == 2);
QByteArray codes = l[0];
codes.replace("..", ".");
QList<QByteArray> cl = codes.split('.');
bool ok;
int from = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int to = from;
if (cl.size() == 2) {
to = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
WordBreakClass brk = word_break_map.value(l[1], WordBreak_Unassigned);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (brk == WordBreak_Unassigned)
qFatal("unassigned word break class: %s", l[1].constData());
for (int codepoint = from; codepoint <= to; ++codepoint) {
QTBF: Fix issue with no splitting the words at "." (FULL STOP) As of Unicode 5.1, some punctuation marks were mapped to MidLetter and MidNumLet for better URL and abbreviations handling which caused "hi.there" to be treated like if it were just a single word; until we have the Unicode Text Segmentation tailoring mechanism, retain the old behavior by remapping (some of) those characters back to their old values. Change-Id: I49dea6064f2ea40a82fc0b1bc3c4f0b4e803919f Reviewed-by: David Faure <david.faure@kdab.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-11-22 01:25:05 +00:00
// ### [
// as of Unicode 5.1, some punctuation marks were mapped to MidLetter and MidNumLet
// which caused "hi.there" to be treated like if it were just a single word;
// until we have a tailoring mechanism, retain the old behavior by remapping those characters here.
if (codepoint == 0x002E) // FULL STOP
brk = WordBreak_MidNum;
else if (codepoint == 0x003A) // COLON
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
brk = WordBreak_Any;
QTBF: Fix issue with no splitting the words at "." (FULL STOP) As of Unicode 5.1, some punctuation marks were mapped to MidLetter and MidNumLet for better URL and abbreviations handling which caused "hi.there" to be treated like if it were just a single word; until we have the Unicode Text Segmentation tailoring mechanism, retain the old behavior by remapping (some of) those characters back to their old values. Change-Id: I49dea6064f2ea40a82fc0b1bc3c4f0b4e803919f Reviewed-by: David Faure <david.faure@kdab.com> Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2012-11-22 01:25:05 +00:00
// ] ###
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &ud = UnicodeData::valueRef(codepoint);
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
ud.p.wordBreakClass = brk;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
}
static void readSentenceBreak()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading SentenceBreakProperty.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/SentenceBreakProperty.txt");
if (!f.exists())
qFatal("Couldn't find SentenceBreakProperty.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line;
line.resize(1024);
int len = f.readLine(line.data(), 1024);
line.resize(len-1);
int comment = line.indexOf('#');
if (comment >= 0)
line = line.left(comment);
line.replace(" ", "");
if (line.isEmpty())
continue;
QList<QByteArray> l = line.split(';');
Q_ASSERT(l.size() == 2);
QByteArray codes = l[0];
codes.replace("..", ".");
QList<QByteArray> cl = codes.split('.');
bool ok;
int from = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int to = from;
if (cl.size() == 2) {
to = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
SentenceBreakClass brk = sentence_break_map.value(l[1], SentenceBreak_Unassigned);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (brk == SentenceBreak_Unassigned)
qFatal("unassigned sentence break class: %s", l[1].constData());
for (int codepoint = from; codepoint <= to; ++codepoint) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &ud = UnicodeData::valueRef(codepoint);
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
ud.p.sentenceBreakClass = brk;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
}
#if 0
// this piece of code does full case folding and comparison. We currently
// don't use it, since this gives lots of issues with things as case insensitive
// search and replace.
static inline void foldCase(uint ch, ushort *out)
{
const QUnicodeTables::Properties *p = qGetProp(ch);
if (!p->caseFoldSpecial) {
*(out++) = ch + p->caseFoldDiff;
} else {
const ushort *folded = specialCaseMap + p->caseFoldDiff;
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
ushort length = *folded++;
while (length--)
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
*out++ = *folded++;
}
*out = 0;
}
static int ucstricmp(const ushort *a, const ushort *ae, const ushort *b, const ushort *be)
{
if (a == b)
return 0;
if (a == 0)
return 1;
if (b == 0)
return -1;
while (a != ae && b != be) {
const QUnicodeTables::Properties *pa = qGetProp(*a);
const QUnicodeTables::Properties *pb = qGetProp(*b);
if (pa->caseFoldSpecial | pb->caseFoldSpecial)
goto special;
int diff = (int)(*a + pa->caseFoldDiff) - (int)(*b + pb->caseFoldDiff);
if ((diff))
return diff;
++a;
++b;
}
}
if (a == ae) {
if (b == be)
return 0;
return -1;
}
return 1;
special:
ushort abuf[SPECIAL_CASE_MAX_LEN + 1];
ushort bbuf[SPECIAL_CASE_MAX_LEN + 1];
abuf[0] = bbuf[0] = 0;
ushort *ap = abuf;
ushort *bp = bbuf;
while (1) {
if (!*ap) {
if (a == ae) {
if (!*bp && b == be)
return 0;
return -1;
}
foldCase(*(a++), abuf);
ap = abuf;
}
if (!*bp) {
if (b == be)
return 1;
foldCase(*(b++), bbuf);
bp = bbuf;
}
if (*ap != *bp)
return (int)*ap - (int)*bp;
++ap;
++bp;
}
}
static int ucstricmp(const ushort *a, const ushort *ae, const uchar *b)
{
if (a == 0)
return 1;
if (b == 0)
return -1;
while (a != ae && *b) {
const QUnicodeTables::Properties *pa = qGetProp(*a);
const QUnicodeTables::Properties *pb = qGetProp((ushort)*b);
if (pa->caseFoldSpecial | pb->caseFoldSpecial)
goto special;
int diff = (int)(*a + pa->caseFoldDiff) - (int)(*b + pb->caseFoldDiff);
if ((diff))
return diff;
++a;
++b;
}
if (a == ae) {
if (!*b)
return 0;
return -1;
}
return 1;
special:
ushort abuf[SPECIAL_CASE_MAX_LEN + 1];
ushort bbuf[SPECIAL_CASE_MAX_LEN + 1];
abuf[0] = bbuf[0] = 0;
ushort *ap = abuf;
ushort *bp = bbuf;
while (1) {
if (!*ap) {
if (a == ae) {
if (!*bp && !*b)
return 0;
return -1;
}
foldCase(*(a++), abuf);
ap = abuf;
}
if (!*bp) {
if (!*b)
return 1;
foldCase(*(b++), bbuf);
bp = bbuf;
}
if (*ap != *bp)
return (int)*ap - (int)*bp;
++ap;
++bp;
}
}
#endif
#if 0
static QList<QByteArray> blockNames;
struct BlockInfo
{
int blockIndex;
int firstCodePoint;
int lastCodePoint;
};
static QList<BlockInfo> blockInfoList;
static void readBlocks()
{
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug("Reading Blocks.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QFile f("data/Blocks.txt");
if (!f.exists())
qFatal("Couldn't find Blocks.txt");
f.open(QFile::ReadOnly);
while (!f.atEnd()) {
QByteArray line = f.readLine();
line.resize(line.size() - 1);
int comment = line.indexOf("#");
if (comment >= 0)
line = line.left(comment);
line.replace(" ", "");
if (line.isEmpty())
continue;
int semicolon = line.indexOf(';');
Q_ASSERT(semicolon >= 0);
QByteArray codePoints = line.left(semicolon);
QByteArray blockName = line.mid(semicolon + 1);
int blockIndex = blockNames.indexOf(blockName);
if (blockIndex == -1) {
blockIndex = blockNames.size();
blockNames.append(blockName);
}
codePoints.replace("..", ".");
QList<QByteArray> cl = codePoints.split('.');
bool ok;
int first = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int last = first;
if (cl.size() == 2) {
last = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
BlockInfo blockInfo = { blockIndex, first, last };
blockInfoList.append(blockInfo);
}
}
#endif
static void readScripts()
{
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
qDebug("Reading Scripts.txt");
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
QFile f("data/Scripts.txt");
if (!f.exists())
qFatal("Couldn't find Scripts.txt");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
f.open(QFile::ReadOnly);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
while (!f.atEnd()) {
QByteArray line = f.readLine();
line.resize(line.size() - 1);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
int comment = line.indexOf("#");
if (comment >= 0)
line = line.left(comment);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
line.replace(" ", "");
line.replace("_", "");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
if (line.isEmpty())
continue;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
int semicolon = line.indexOf(';');
Q_ASSERT(semicolon >= 0);
QByteArray codePoints = line.left(semicolon);
QByteArray scriptName = line.mid(semicolon + 1);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
codePoints.replace("..", ".");
QList<QByteArray> cl = codePoints.split('.');
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
bool ok;
int first = cl[0].toInt(&ok, 16);
Q_ASSERT(ok);
int last = first;
if (cl.size() == 2) {
last = cl[1].toInt(&ok, 16);
Q_ASSERT(ok);
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Add QChar::Script enum ...where the values are not aliased to Common script. The old QUnicodeTables::Script enum was retained for compatibility reasons until Qt internals are updated to use QChar::script(). Using QChar::Script instead of QUnicodeTables::Script would improve both the text analysis (itemization, boundary finding) and the text shaping quality. This also a required step for switching to Hurfbuzz-NG. /* This adds 6668 more .rodata bytes */ Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-08 03:36:49 +00:00
if (!scriptMap.contains(scriptName))
qFatal("Unhandled script property value: %s", scriptName.constData());
QChar::Script script = scriptMap.value(scriptName, QChar::Script_Unknown);
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
for (int codepoint = first; codepoint <= last; ++codepoint) {
UnicodeData &ud = UnicodeData::valueRef(codepoint);
Add QChar::Script enum ...where the values are not aliased to Common script. The old QUnicodeTables::Script enum was retained for compatibility reasons until Qt internals are updated to use QChar::script(). Using QChar::Script instead of QUnicodeTables::Script would improve both the text analysis (itemization, boundary finding) and the text shaping quality. This also a required step for switching to Hurfbuzz-NG. /* This adds 6668 more .rodata bytes */ Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-08 03:36:49 +00:00
ud.p.script = script;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
}
#if 0
static void dump(int from, int to)
{
for (int i = from; i <= to; ++i) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(i);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qDebug("0x%04x: cat=%d combining=%d dir=%d case=%x mirror=%x joining=%d age=%d",
i, d.p.category, d.p.combiningClass, d.p.direction, d.otherCase, d.mirroredChar, d.p.joining, d.p.age);
if (d.decompositionType != QChar::NoDecomposition) {
qDebug(" decomposition: type=%d, length=%d, first=%x", d.decompositionType, d.decomposition.size(),
d.decomposition[0]);
}
}
qDebug(" ");
}
#endif
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
static QList<PropertyFlags> uniqueProperties;
static void computeUniqueProperties()
{
qDebug("computeUniqueProperties:");
for (int codepoint = 0; codepoint <= QChar::LastValidCodePoint; ++codepoint) {
UnicodeData &d = UnicodeData::valueRef(codepoint);
int index = uniqueProperties.indexOf(d.p);
if (index == -1) {
index = uniqueProperties.size();
uniqueProperties.append(d.p);
}
d.propertyIndex = index;
}
qDebug(" %d unique unicode properties found", uniqueProperties.size());
}
struct UniqueBlock {
inline UniqueBlock() : index(-1) {}
inline bool operator==(const UniqueBlock &other) const
{ return values == other.values; }
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int index;
Use QList instead of QVector in util Task-number: QTBUG-84469 Change-Id: I077fb5c32456d438a457c1f73852313ea2ea9ae5 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-07 10:04:21 +00:00
QList<int> values;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
};
static QByteArray createPropertyInfo()
{
qDebug("createPropertyInfo:");
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
// we reserve one bit more than in the assert below for the sign
Q_ASSERT(maxMirroredDiff < (1<<12));
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
Q_ASSERT(maxLowerCaseDiff < (1<<13));
Q_ASSERT(maxUpperCaseDiff < (1<<13));
Q_ASSERT(maxTitleCaseDiff < (1<<13));
Q_ASSERT(maxCaseFoldDiff < (1<<13));
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
const int BMP_BLOCKSIZE = 32;
const int BMP_SHIFT = 5;
const int BMP_END = 0x11000;
const int SMP_END = 0x110000;
const int SMP_BLOCKSIZE = 256;
const int SMP_SHIFT = 8;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
QList<UniqueBlock> uniqueBlocks;
Use QList instead of QVector in util Task-number: QTBUG-84469 Change-Id: I077fb5c32456d438a457c1f73852313ea2ea9ae5 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-07 10:04:21 +00:00
QList<int> blockMap;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int used = 0;
for (int block = 0; block < BMP_END/BMP_BLOCKSIZE; ++block) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
UniqueBlock b;
b.values.reserve(BMP_BLOCKSIZE);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < BMP_BLOCKSIZE; ++i) {
int uc = block*BMP_BLOCKSIZE + i;
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(uc);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(d.propertyIndex);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int index = uniqueBlocks.indexOf(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (index == -1) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
index = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
b.index = used;
used += BMP_BLOCKSIZE;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
uniqueBlocks.append(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
blockMap.append(uniqueBlocks.at(index).index);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int bmp_blocks = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int block = BMP_END/SMP_BLOCKSIZE; block < SMP_END/SMP_BLOCKSIZE; ++block) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
UniqueBlock b;
b.values.reserve(SMP_BLOCKSIZE);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < SMP_BLOCKSIZE; ++i) {
int uc = block*SMP_BLOCKSIZE + i;
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(uc);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(d.propertyIndex);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int index = uniqueBlocks.indexOf(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (index == -1) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
index = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
b.index = used;
used += SMP_BLOCKSIZE;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
uniqueBlocks.append(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
blockMap.append(uniqueBlocks.at(index).index);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int smp_blocks = uniqueBlocks.size() - bmp_blocks;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int bmp_block_data = bmp_blocks*BMP_BLOCKSIZE*sizeof(unsigned short);
int bmp_trie = BMP_END/BMP_BLOCKSIZE*sizeof(unsigned short);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int bmp_mem = bmp_block_data + bmp_trie;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug(" %d unique blocks in BMP.", bmp_blocks);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qDebug(" block data uses: %d bytes", bmp_block_data);
qDebug(" trie data uses : %d bytes", bmp_trie);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int smp_block_data = smp_blocks*SMP_BLOCKSIZE*sizeof(unsigned short);
int smp_trie = (SMP_END-BMP_END)/SMP_BLOCKSIZE*sizeof(unsigned short);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int smp_mem = smp_block_data + smp_trie;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug(" %d unique blocks in SMP.", smp_blocks);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qDebug(" block data uses: %d bytes", smp_block_data);
qDebug(" trie data uses : %d bytes", smp_trie);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int prop_data = uniqueProperties.size() * SizeOfPropertiesStruct;
qDebug("\n properties data uses : %d bytes", prop_data);
qDebug(" memory usage: %d bytes", bmp_mem + smp_mem + prop_data);
Q_ASSERT(blockMap.last() + blockMap.size() < (1<<(sizeof(unsigned short)*8)));
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QByteArray out;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "static const unsigned short uc_property_trie[] = {\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// first write the map
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
out += " // [0x0..0x" + QByteArray::number(BMP_END, 16) + ")";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < BMP_END/BMP_BLOCKSIZE; ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
if (!((i*BMP_BLOCKSIZE) % 0x1000))
out += "\n";
out += "\n ";
}
out += QByteArray::number(blockMap.at(i) + blockMap.size());
out += ", ";
}
if (out.endsWith(' '))
out.chop(1);
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
out += "\n\n // [0x" + QByteArray::number(BMP_END, 16) + "..0x" + QByteArray::number(SMP_END, 16) + ")\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = BMP_END/BMP_BLOCKSIZE; i < blockMap.size(); ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
if (!(i % (0x10000/SMP_BLOCKSIZE)))
out += "\n";
out += "\n ";
}
out += QByteArray::number(blockMap.at(i) + blockMap.size());
out += ", ";
}
if (out.endsWith(' '))
out.chop(1);
out += "\n";
// write the data
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
for (int i = 0; i < uniqueBlocks.size(); ++i) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (out.endsWith(' '))
out.chop(1);
out += "\n";
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
const UniqueBlock &b = uniqueBlocks.at(i);
for (int j = 0; j < b.values.size(); ++j) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (!(j % 8)) {
if (out.endsWith(' '))
out.chop(1);
out += "\n ";
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += QByteArray::number(b.values.at(j));
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += ", ";
}
}
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
if (out.endsWith(", "))
out.chop(2);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "\n};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "#define GET_PROP_INDEX(ucs4) \\\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
" (ucs4 < 0x" + QByteArray::number(BMP_END, 16) + " \\\n"
" ? (uc_property_trie[uc_property_trie[ucs4>>" + QByteArray::number(BMP_SHIFT) +
"] + (ucs4 & 0x" + QByteArray::number(BMP_BLOCKSIZE-1, 16)+ ")]) \\\n"
" : (uc_property_trie[uc_property_trie[((ucs4 - 0x" + QByteArray::number(BMP_END, 16) +
")>>" + QByteArray::number(SMP_SHIFT) + ") + 0x" + QByteArray::number(BMP_END/BMP_BLOCKSIZE, 16) + "]"
" + (ucs4 & 0x" + QByteArray::number(SMP_BLOCKSIZE-1, 16) + ")]))\n\n"
"#define GET_PROP_INDEX_UCS2(ucs2) \\\n"
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
" (uc_property_trie[uc_property_trie[ucs2>>" + QByteArray::number(BMP_SHIFT) +
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
"] + (ucs2 & 0x" + QByteArray::number(BMP_BLOCKSIZE-1, 16)+ ")])\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "static const Properties uc_properties[] = {";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// keep in sync with the property declaration
for (int i = 0; i < uniqueProperties.size(); ++i) {
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
const PropertyFlags &p = uniqueProperties.at(i);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out += "\n { ";
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
// " ushort category : 8; /* 5 used */\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += QByteArray::number( p.category );
out += ", ";
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
// " ushort direction : 8; /* 5 used */\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += QByteArray::number( p.direction );
out += ", ";
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
// " ushort combiningClass : 8;\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += QByteArray::number( p.combiningClass );
out += ", ";
Introduce QChar::JoiningType enum and QChar::joiningType() method This aimed to disctinct joining types "L", "T", and "U" from just "U". Unicode 6.3.0 has introduced a character with joining type "L" and Unicode 7.0 will add a few more characters of joining type "L", so we'll have to deal with it anyways. [ChangeLog][QtCore][QChar] Added JoiningType enum and joiningType() method that deprecates the old QChar::Joining enum and joining() method. Change-Id: I4be3a3f745d944e689feb9b62d4ca86d1cf371b0 Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2014-01-26 00:42:37 +00:00
// " ushort joining : 3;\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += QByteArray::number( p.joining );
out += ", ";
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
// " signed short digitValue : 5;\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += QByteArray::number( p.digitValue );
out += ", ";
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
// " signed short mirrorDiff : 16;\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += QByteArray::number( p.mirrorDiff );
out += ", ";
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
// " ushort unicodeVersion : 8; /* 5 used */\n"
out += QByteArray::number( p.age );
out += ", ";
// " ushort nfQuickCheck : 8;\n"
out += QByteArray::number( p.nfQuickCheck );
out += ", ";
// " struct {\n"
// " ushort special : 1;\n"
// " signed short diff : 15;\n"
// " } cases[NumCases];\n"
out += " { {";
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
out += QByteArray::number( p.lowerCaseSpecial );
out += ", ";
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
out += QByteArray::number( p.lowerCaseDiff );
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
out += "}, {";
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
out += QByteArray::number( p.upperCaseSpecial );
out += ", ";
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
out += QByteArray::number( p.upperCaseDiff );
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
out += "}, {";
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
out += QByteArray::number( p.titleCaseSpecial );
out += ", ";
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
out += QByteArray::number( p.titleCaseDiff );
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
out += "}, {";
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
out += QByteArray::number( p.caseFoldSpecial );
out += ", ";
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
out += QByteArray::number( p.caseFoldDiff );
QUnicodeTables: use array for case folding tables Instead of four pairs of :1 :15 bit fields, use an array of four :1, :15 structs. This allows to replace the case folding traits classes with a simple enum that indexes into said array. I don't know what the WASM #ifdef'ed code is supposed to effect (a :0 bit-field is only useful to separate adjacent bit-field into separate memory locations for multi-threading), but I thought it safer to leave it in, and that means the array must be a 64-bit block of its own, so I had to move two fields around. Saves ~4.5KiB in text size on optimized GCC 10 LTO Linux AMD64 builds. Change-Id: Ib52cd7706342d5227b50b57545d073829c45da9a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2019-09-03 18:53:31 +00:00
out += "} }, ";
Update Text segmentation and line break data to Unicode 10.0 Also adjusted the text segmentation and line break algorithms so that they can handle the new data, and pass the test suite. Change-Id: Ib727fd80003e34e96458d7a681996de3fa3691e7 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2017-12-12 09:14:28 +00:00
// " ushort graphemeBreakClass : 5; /* 5 used */\n"
// " ushort wordBreakClass : 5; /* 5 used */\n"
// " ushort lineBreakClass : 6; /* 6 used */\n"
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
out += QByteArray::number( p.graphemeBreakClass );
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += ", ";
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
out += QByteArray::number( p.wordBreakClass );
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += ", ";
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
out += QByteArray::number( p.lineBreakClass );
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
out += ", ";
QUnicodeTables: pack Properties struct GCC doesn't like the sequence : 5 : 5 : 8 : 6 : 8 and inserts a :6 padding between the :5 and the :8 and a :2 padding between the :6 and the :8, growing the bitfield by 8 bits of embedded padding and another byte to bring the struct back to sizeof % 2 == 0. Fix by reshuffling the elements and adding a static_assert for the next round. Saves ~5KiB in QtCore executable size. Change-Id: I4758a6f48ba389abc2aee92f60997d42ebb0e5b8 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-09-03 21:40:56 +00:00
// " ushort sentenceBreakClass : 8; /* 4 used */\n"
out += QByteArray::number( p.sentenceBreakClass );
out += ", ";
Update Unicode data up to v7.0 * Two newly adopted currency symbols: the Azerbaijan manat and the Russia ruble * Pictographic symbols (including many emoji), geometric symbols, arrows, and ornaments originating from the Wingdings and Webdings sets * Twenty-three new lesser-used and historic scripts extending support for written languages of North America, China, India, other Asian countries, and Africa * Letters used in Teuthonista and other transcriptional systems, and a new notational set, Duployan For more details, see http://www.unicode.org/versions/Unicode7.0.0/ The Properties struct's .*Diff members were narrowed down to signed 15 bits and the unicodeVersion has been expanded to 8 bits. [ChangeLog][QtCore] Unicode data updated to v.7.0 Change-Id: I93ab6f79fa3b05f61abc7279f1d046834c1c1a0b Reviewed-by: Lars Knoll <lars.knoll@digia.com>
2015-03-23 21:01:06 +00:00
// " ushort script : 8;\n"
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
out += QByteArray::number( p.script );
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out += " },";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
if (out.endsWith(','))
out.chop(1);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out += "\n};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
out += "Q_DECL_CONST_FUNCTION static inline const Properties *qGetProp(char32_t ucs4) noexcept\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"{\n"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" return uc_properties + GET_PROP_INDEX(ucs4);\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"}\n"
"\n"
QChar: port low-level functions from uint/ushort to char32/16_t Now that the standard gives us proper types for UTF-16 and UTF-32 characters, use them. Will eventually make the code much easier to read than today, where uint could be an index as well as a char32_t. It also ensures that the result of e.g. QChar::highSurrogate() can still be implicitly converted to a QChar now that the QChar(non-characater-integral-types) ctors are being made explicit. [ChangeLog][QtCore][QChar] All low-level functions (e.g. highSurrogate()) now take and return char16_t instead of ushort and char32_t instead of uint. Change-Id: I9cd8ebf6fb998fe1075dae96c7c4484a057f0b91 Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-22 10:50:55 +00:00
"Q_DECL_CONST_FUNCTION static inline const Properties *qGetProp(char16_t ucs2) noexcept\n"
"{\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
" return uc_properties + GET_PROP_INDEX_UCS2(ucs2);\n"
QChar: port low-level functions from uint/ushort to char32/16_t Now that the standard gives us proper types for UTF-16 and UTF-32 characters, use them. Will eventually make the code much easier to read than today, where uint could be an index as well as a char32_t. It also ensures that the result of e.g. QChar::highSurrogate() can still be implicitly converted to a QChar now that the QChar(non-characater-integral-types) ctors are being made explicit. [ChangeLog][QtCore][QChar] All low-level functions (e.g. highSurrogate()) now take and return char16_t instead of ushort and char32_t instead of uint. Change-Id: I9cd8ebf6fb998fe1075dae96c7c4484a057f0b91 Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-22 10:50:55 +00:00
"}\n"
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_DECL_CONST_FUNCTION Q_CORE_EXPORT const Properties * QT_FASTCALL properties(char32_t ucs4) noexcept\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"{\n"
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
" return qGetProp(ucs4);\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"}\n"
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_DECL_CONST_FUNCTION Q_CORE_EXPORT const Properties * QT_FASTCALL properties(char16_t ucs2) noexcept\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"{\n"
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
" return qGetProp(ucs2);\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"}\n\n";
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
out += "Q_CORE_EXPORT GraphemeBreakClass QT_FASTCALL graphemeBreakClass(char32_t ucs4) noexcept\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"{\n"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" return static_cast<GraphemeBreakClass>(qGetProp(ucs4)->graphemeBreakClass);\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"}\n"
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT WordBreakClass QT_FASTCALL wordBreakClass(char32_t ucs4) noexcept\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"{\n"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" return static_cast<WordBreakClass>(qGetProp(ucs4)->wordBreakClass);\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"}\n"
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT SentenceBreakClass QT_FASTCALL sentenceBreakClass(char32_t ucs4) noexcept\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"{\n"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" return static_cast<SentenceBreakClass>(qGetProp(ucs4)->sentenceBreakClass);\n"
add some useful methods to QUnicodeTables:: in order to reduce code duplication and prepare the ground for upcoming changes Change-Id: I980244149f65384c9484bbec4682de8b7b848b08 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-09 13:44:36 +00:00
"}\n"
"\n"
QUnicodeTables: port to charNN_t This makes existing calls passing uint or ushort ambiguous, so fix all the callers. There do not appear to be callers outside QtBase. In fact, the ...BreakClass() functions appear to be utterly unused. Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-23 19:10:48 +00:00
"Q_CORE_EXPORT LineBreakClass QT_FASTCALL lineBreakClass(char32_t ucs4) noexcept\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"{\n"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" return static_cast<LineBreakClass>(qGetProp(ucs4)->lineBreakClass);\n"
Make QUnicodeTables::script() support SMP code points Instead of expanding the scripts table with script values for the code points >= 0x10000, it has been merged with the properties table in order to increase perfomance of the script itemization code (not affected yet). (Stats: the properties table grew up in 97428-89800 = 7628 bytes; the old scripts table was of size 7680 bytes) The outdated ScriptsInitial.txt and ScriptsCorrections.txt file has been removed (they were just empty, the "corrigendum" script corrections should be applied to Scripts.txt directly, *no customization allowed*!). More script testcases has been added - at least one per supported script. Task-number: QTBUG-6530 Change-Id: I40a9e76f681e2dd552fd4c61af0808d043962e79 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-11 12:19:24 +00:00
"}\n"
Add QChar::Script enum ...where the values are not aliased to Common script. The old QUnicodeTables::Script enum was retained for compatibility reasons until Qt internals are updated to use QChar::script(). Using QChar::Script instead of QUnicodeTables::Script would improve both the text analysis (itemization, boundary finding) and the text shaping quality. This also a required step for switching to Hurfbuzz-NG. /* This adds 6668 more .rodata bytes */ Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-08 03:36:49 +00:00
"\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
return out;
}
static QByteArray createSpecialCaseMap()
{
qDebug("createSpecialCaseMap:");
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
QByteArray out;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
out += "static const unsigned short specialCaseMap[] = {\n"
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
" 0x0, // placeholder";
optimize QString::toLower()/toUpper() for special cases, step 2 from now, QUnicodeTables::specialCaseMap[] starts with a placeholder; so, if somethingCaseSpecial is true, then somethingCaseDiff is always greater than 0 Change-Id: Ibb1870512836eee71b1521564c0745096c05b2f9 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:21 +00:00
int i = 1;
QChar/QString: centralize case folding in qchar.cpp There are (at least) two implementations of the low-level case-folding algorithm, one of which (for QChar::toLower()) seems to be wrong (it doesn't deal with special cases which expand to more than one code point). The algoithm hidden in QString and entangled with the QString detaching code makes reusing the code much harder. At the same time, the dependency of the algorithm on the unicode tables makes exposing a non-allocating result type in the public API hard. std::u16string would be an alternative if we can assure that all implementations use SSO with at least four characters. So, for the time being, leave this as internal API for use in an upcoming QStringView::toLower() as well as case-insensitive hashing. Change-Id: Iabb2611846f6176776aa20e634f44d8464f3305c Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2020-04-24 10:54:20 +00:00
int maxN = 0;
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
while (i < specialCaseMap.size()) {
out += "\n ";
int n = specialCaseMap.at(i);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
for (int j = 0; j <= n; ++j) {
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
out += QByteArray(" 0x") + QByteArray::number(specialCaseMap.at(i+j), 16);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out += ",";
optimize QString::toLower()/toUpper() for special cases, step 1 reorganize QUnicodeTables::specialCaseMap as follows: specialCaseMap contains sequence entries in form { length, a, b, .. } Change-Id: Iea1f80bc2f4dc1f505428dad981cde26daaa52c7 Merge-request: 70 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Olivier Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
2011-10-18 17:12:20 +00:00
}
i += n + 1;
QChar/QString: centralize case folding in qchar.cpp There are (at least) two implementations of the low-level case-folding algorithm, one of which (for QChar::toLower()) seems to be wrong (it doesn't deal with special cases which expand to more than one code point). The algoithm hidden in QString and entangled with the QString detaching code makes reusing the code much harder. At the same time, the dependency of the algorithm on the unicode tables makes exposing a non-allocating result type in the public API hard. std::u16string would be an alternative if we can assure that all implementations use SSO with at least four characters. So, for the time being, leave this as internal API for use in an upcoming QStringView::toLower() as well as case-insensitive hashing. Change-Id: Iabb2611846f6176776aa20e634f44d8464f3305c Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2020-04-24 10:54:20 +00:00
maxN = std::max(maxN, n);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out.chop(1);
QChar/QString: centralize case folding in qchar.cpp There are (at least) two implementations of the low-level case-folding algorithm, one of which (for QChar::toLower()) seems to be wrong (it doesn't deal with special cases which expand to more than one code point). The algoithm hidden in QString and entangled with the QString detaching code makes reusing the code much harder. At the same time, the dependency of the algorithm on the unicode tables makes exposing a non-allocating result type in the public API hard. std::u16string would be an alternative if we can assure that all implementations use SSO with at least four characters. So, for the time being, leave this as internal API for use in an upcoming QStringView::toLower() as well as case-insensitive hashing. Change-Id: Iabb2611846f6176776aa20e634f44d8464f3305c Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2020-04-24 10:54:20 +00:00
out += "\n};\n\nconst unsigned int MaxSpecialCaseLength = ";
out += QByteArray::number(maxN);
out += ";\n\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Fix a bug in the generation of the Unicode joining property data The code that was supposed to initialize the joining property of characters to the correct defaults was actually applied after reading in the data from the Unicode file, and was in a couple of cases overwriting explicitly specified data in ArabicShaping.txt Task-number: QTBUG-63191 Change-Id: Ie35261039b2211a827322ca11afacd9555ccefc7 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2017-12-11 10:28:11 +00:00
qDebug(" memory usage: %ld bytes", specialCaseMap.size()*sizeof(unsigned short));
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
return out;
}
static QByteArray createCompositionInfo()
{
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
qDebug("createCompositionInfo: highestComposedCharacter=0x%x", highestComposedCharacter);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
const int BMP_BLOCKSIZE = 16;
const int BMP_SHIFT = 4;
const int BMP_END = 0x3400; // start of Han
const int SMP_END = 0x30000;
const int SMP_BLOCKSIZE = 256;
const int SMP_SHIFT = 8;
if (SMP_END <= highestComposedCharacter)
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
qFatal("end of table smaller than highest composed character 0x%x", highestComposedCharacter);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Use QList instead of QVector in util Task-number: QTBUG-84469 Change-Id: I077fb5c32456d438a457c1f73852313ea2ea9ae5 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-07 10:04:21 +00:00
QList<unsigned short> decompositions;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int tableIndex = 0;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
QList<UniqueBlock> uniqueBlocks;
Use QList instead of QVector in util Task-number: QTBUG-84469 Change-Id: I077fb5c32456d438a457c1f73852313ea2ea9ae5 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-07 10:04:21 +00:00
QList<int> blockMap;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int used = 0;
for (int block = 0; block < BMP_END/BMP_BLOCKSIZE; ++block) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
UniqueBlock b;
b.values.reserve(BMP_BLOCKSIZE);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < BMP_BLOCKSIZE; ++i) {
int uc = block*BMP_BLOCKSIZE + i;
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(uc);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (!d.decomposition.isEmpty()) {
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
int utf16Length = 0;
decompositions.append(0);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int j = 0; j < d.decomposition.size(); ++j) {
int code = d.decomposition.at(j);
replace hardcoded values with a surrogate handling methods Change-Id: Iba079953c46a29404232d2dacbe0c90170097d51 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-08 07:18:45 +00:00
if (QChar::requiresSurrogates(code)) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// save as surrogate pair
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
decompositions.append(QChar::highSurrogate(code));
decompositions.append(QChar::lowSurrogate(code));
utf16Length += 2;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
} else {
decompositions.append(code);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
utf16Length++;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
decompositions[tableIndex] = d.decompositionType + (utf16Length<<8);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(tableIndex);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
tableIndex += utf16Length + 1;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
} else {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(0xffff);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int index = uniqueBlocks.indexOf(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (index == -1) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
index = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
b.index = used;
used += BMP_BLOCKSIZE;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
uniqueBlocks.append(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
blockMap.append(uniqueBlocks.at(index).index);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int bmp_blocks = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int block = BMP_END/SMP_BLOCKSIZE; block < SMP_END/SMP_BLOCKSIZE; ++block) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
UniqueBlock b;
b.values.reserve(SMP_BLOCKSIZE);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < SMP_BLOCKSIZE; ++i) {
int uc = block*SMP_BLOCKSIZE + i;
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
UnicodeData &d = UnicodeData::valueRef(uc);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (!d.decomposition.isEmpty()) {
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
int utf16Length = 0;
decompositions.append(0);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int j = 0; j < d.decomposition.size(); ++j) {
int code = d.decomposition.at(j);
replace hardcoded values with a surrogate handling methods Change-Id: Iba079953c46a29404232d2dacbe0c90170097d51 Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-08 07:18:45 +00:00
if (QChar::requiresSurrogates(code)) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
// save as surrogate pair
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
decompositions.append(QChar::highSurrogate(code));
decompositions.append(QChar::lowSurrogate(code));
utf16Length += 2;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
} else {
decompositions.append(code);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
utf16Length++;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
decompositions[tableIndex] = d.decompositionType + (utf16Length<<8);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(tableIndex);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
tableIndex += utf16Length + 1;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
} else {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(0xffff);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int index = uniqueBlocks.indexOf(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (index == -1) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
index = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
b.index = used;
used += SMP_BLOCKSIZE;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
uniqueBlocks.append(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
blockMap.append(uniqueBlocks.at(index).index);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int smp_blocks = uniqueBlocks.size() - bmp_blocks;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
// if the condition below doesn't hold anymore we need to modify our decomposition code
Q_ASSERT(tableIndex < 0xffff);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int bmp_block_data = bmp_blocks*BMP_BLOCKSIZE*sizeof(unsigned short);
int bmp_trie = BMP_END/BMP_BLOCKSIZE*sizeof(unsigned short);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int bmp_mem = bmp_block_data + bmp_trie;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug(" %d unique blocks in BMP.", bmp_blocks);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qDebug(" block data uses: %d bytes", bmp_block_data);
qDebug(" trie data uses : %d bytes", bmp_trie);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int smp_block_data = smp_blocks*SMP_BLOCKSIZE*sizeof(unsigned short);
int smp_trie = (SMP_END-BMP_END)/SMP_BLOCKSIZE*sizeof(unsigned short);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int smp_mem = smp_block_data + smp_trie;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug(" %d unique blocks in SMP.", smp_blocks);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qDebug(" block data uses: %d bytes", smp_block_data);
qDebug(" trie data uses : %d bytes", smp_trie);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int decomposition_data = decompositions.size() * 2;
qDebug("\n decomposition data uses : %d bytes", decomposition_data);
qDebug(" memory usage: %d bytes", bmp_mem + smp_mem + decomposition_data);
Q_ASSERT(blockMap.last() + blockMap.size() < (1<<(sizeof(unsigned short)*8)));
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QByteArray out;
out += "static const unsigned short uc_decomposition_trie[] = {\n";
// first write the map
out += " // 0 - 0x" + QByteArray::number(BMP_END, 16);
for (int i = 0; i < BMP_END/BMP_BLOCKSIZE; ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
if (!((i*BMP_BLOCKSIZE) % 0x1000))
out += "\n";
out += "\n ";
}
out += QByteArray::number(blockMap.at(i) + blockMap.size());
out += ", ";
}
if (out.endsWith(' '))
out.chop(1);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out += "\n\n // 0x" + QByteArray::number(BMP_END, 16) + " - 0x" + QByteArray::number(SMP_END, 16) + "\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = BMP_END/BMP_BLOCKSIZE; i < blockMap.size(); ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
if (!(i % (0x10000/SMP_BLOCKSIZE)))
out += "\n";
out += "\n ";
}
out += QByteArray::number(blockMap.at(i) + blockMap.size());
out += ", ";
}
if (out.endsWith(' '))
out.chop(1);
out += "\n";
// write the data
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
for (int i = 0; i < uniqueBlocks.size(); ++i) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (out.endsWith(' '))
out.chop(1);
out += "\n";
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
const UniqueBlock &b = uniqueBlocks.at(i);
for (int j = 0; j < b.values.size(); ++j) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (!(j % 8)) {
if (out.endsWith(' '))
out.chop(1);
out += "\n ";
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "0x" + QByteArray::number(b.values.at(j), 16);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += ", ";
}
}
if (out.endsWith(' '))
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out.chop(2);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "\n};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "#define GET_DECOMPOSITION_INDEX(ucs4) \\\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
" (ucs4 < 0x" + QByteArray::number(BMP_END, 16) + " \\\n"
" ? (uc_decomposition_trie[uc_decomposition_trie[ucs4>>" + QByteArray::number(BMP_SHIFT) +
"] + (ucs4 & 0x" + QByteArray::number(BMP_BLOCKSIZE-1, 16)+ ")]) \\\n"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" : (ucs4 < 0x" + QByteArray::number(SMP_END, 16) + " \\\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
" ? uc_decomposition_trie[uc_decomposition_trie[((ucs4 - 0x" + QByteArray::number(BMP_END, 16) +
")>>" + QByteArray::number(SMP_SHIFT) + ") + 0x" + QByteArray::number(BMP_END/BMP_BLOCKSIZE, 16) + "]"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" + (ucs4 & 0x" + QByteArray::number(SMP_BLOCKSIZE-1, 16) + ")] \\\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
" : 0xffff))\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "static const unsigned short uc_decomposition_map[] = {";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < decompositions.size(); ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
out += "\n ";
}
out += "0x" + QByteArray::number(decompositions.at(i), 16);
out += ", ";
}
if (out.endsWith(' '))
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out.chop(2);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += "\n};\n\n";
return out;
}
static QByteArray createLigatureInfo()
{
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
qDebug("createLigatureInfo: numLigatures=%d, highestLigature=0x%x", numLigatures, highestLigature);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
significant unicodetables generator performance optimization since the entire range of a valid unicode code points is in use, QHash is suboptimal and could be replaced with QList; taking the value by ref and not inserting it back to the map + not calculating the default value over and over gains us up to 60% performance boost! Change-Id: I48c54a8e88472cf76c79c0aac44e65eeefa44861 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-05-08 01:43:16 +00:00
for (int i = 0; i < ligatureHashes.size(); ++i) {
const QList<Ligature> &l = ligatureHashes.value(i);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
for (int j = 0; j < l.size(); ++j) {
// if the condition below doesn't hold anymore we need to modify our ligatureHelper code
Q_ASSERT(QChar::requiresSurrogates(l.at(j).u2) == QChar::requiresSurrogates(l.at(j).ligature) &&
QChar::requiresSurrogates(l.at(j).u2) == QChar::requiresSurrogates(l.at(j).u1));
}
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
const int BMP_BLOCKSIZE = 32;
const int BMP_SHIFT = 5;
const int BMP_END = 0x3100;
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
const int SMP_END = 0x12000;
const int SMP_BLOCKSIZE = 256;
const int SMP_SHIFT = 8;
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
if (SMP_END <= highestLigature)
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
qFatal("end of table smaller than highest ligature character 0x%x", highestLigature);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
QList<unsigned short> ligatures;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int tableIndex = 0;
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
QList<UniqueBlock> uniqueBlocks;
Use QList instead of QVector in util Task-number: QTBUG-84469 Change-Id: I077fb5c32456d438a457c1f73852313ea2ea9ae5 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2020-07-07 10:04:21 +00:00
QList<int> blockMap;
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int used = 0;
for (int block = 0; block < BMP_END/BMP_BLOCKSIZE; ++block) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
UniqueBlock b;
b.values.reserve(BMP_BLOCKSIZE);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < BMP_BLOCKSIZE; ++i) {
int uc = block*BMP_BLOCKSIZE + i;
QList<Ligature> l = ligatureHashes.value(uc);
if (!l.isEmpty()) {
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
Q_ASSERT(!QChar::requiresSurrogates(uc));
Remove usages of deprecated APIs from QtAlgorithms Task-number: QTBUG-76491 Change-Id: I9dab736a0cbd2e86588919640c26e8ce6b3674d0 Reviewed-by: Alex Blasche <alexander.blasche@qt.io> Reviewed-by: Leena Miettinen <riitta-leena.miettinen@qt.io>
2019-06-20 15:40:45 +00:00
std::sort(l.begin(), l.end()); // needed for bsearch in ligatureHelper code
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
ligatures.append(l.size());
for (int j = 0; j < l.size(); ++j) {
ligatures.append(l.at(j).u1);
ligatures.append(l.at(j).ligature);
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(tableIndex);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
tableIndex += 2*l.size() + 1;
} else {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(0xffff);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int index = uniqueBlocks.indexOf(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (index == -1) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
index = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
b.index = used;
used += BMP_BLOCKSIZE;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
uniqueBlocks.append(b);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
blockMap.append(uniqueBlocks.at(index).index);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int bmp_blocks = uniqueBlocks.size();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
for (int block = BMP_END/SMP_BLOCKSIZE; block < SMP_END/SMP_BLOCKSIZE; ++block) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
UniqueBlock b;
b.values.reserve(SMP_BLOCKSIZE);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
for (int i = 0; i < SMP_BLOCKSIZE; ++i) {
int uc = block*SMP_BLOCKSIZE + i;
QList<Ligature> l = ligatureHashes.value(uc);
if (!l.isEmpty()) {
Q_ASSERT(QChar::requiresSurrogates(uc));
Remove usages of deprecated APIs from QtAlgorithms Task-number: QTBUG-76491 Change-Id: I9dab736a0cbd2e86588919640c26e8ce6b3674d0 Reviewed-by: Alex Blasche <alexander.blasche@qt.io> Reviewed-by: Leena Miettinen <riitta-leena.miettinen@qt.io>
2019-06-20 15:40:45 +00:00
std::sort(l.begin(), l.end()); // needed for bsearch in ligatureHelper code
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
ligatures.append(l.size());
for (int j = 0; j < l.size(); ++j) {
ligatures.append(QChar::highSurrogate(l.at(j).u1));
ligatures.append(QChar::lowSurrogate(l.at(j).u1));
ligatures.append(QChar::highSurrogate(l.at(j).ligature));
ligatures.append(QChar::lowSurrogate(l.at(j).ligature));
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(tableIndex);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
tableIndex += 4*l.size() + 1;
} else {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
b.values.append(0xffff);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
}
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int index = uniqueBlocks.indexOf(b);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
if (index == -1) {
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
index = uniqueBlocks.size();
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
b.index = used;
used += SMP_BLOCKSIZE;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
uniqueBlocks.append(b);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
blockMap.append(uniqueBlocks.at(index).index);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int smp_blocks = uniqueBlocks.size() - bmp_blocks;
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
// if the condition below doesn't hold anymore we need to modify our composition code
Q_ASSERT(tableIndex < 0xffff);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int bmp_block_data = bmp_blocks*BMP_BLOCKSIZE*sizeof(unsigned short);
int bmp_trie = BMP_END/BMP_BLOCKSIZE*sizeof(unsigned short);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
int bmp_mem = bmp_block_data + bmp_trie;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug(" %d unique blocks in BMP.", bmp_blocks);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
qDebug(" block data uses: %d bytes", bmp_block_data);
qDebug(" trie data uses : %d bytes", bmp_trie);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int smp_block_data = smp_blocks*SMP_BLOCKSIZE*sizeof(unsigned short);
int smp_trie = (SMP_END-BMP_END)/SMP_BLOCKSIZE*sizeof(unsigned short);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
int smp_mem = smp_block_data + smp_trie;
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
qDebug(" %d unique blocks in SMP.", smp_blocks);
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
qDebug(" block data uses: %d bytes", smp_block_data);
qDebug(" trie data uses : %d bytes", smp_trie);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
int ligature_data = ligatures.size() * 2;
qDebug("\n ligature data uses : %d bytes", ligature_data);
qDebug(" memory usage: %d bytes", bmp_mem + smp_mem + ligature_data);
Q_ASSERT(blockMap.last() + blockMap.size() < (1<<(sizeof(unsigned short)*8)));
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QByteArray out;
out += "static const unsigned short uc_ligature_trie[] = {\n";
// first write the map
out += " // 0 - 0x" + QByteArray::number(BMP_END, 16);
for (int i = 0; i < BMP_END/BMP_BLOCKSIZE; ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
if (!((i*BMP_BLOCKSIZE) % 0x1000))
out += "\n";
out += "\n ";
}
out += QByteArray::number(blockMap.at(i) + blockMap.size());
out += ", ";
}
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
if (out.endsWith(' '))
out.chop(1);
out += "\n\n // 0x" + QByteArray::number(BMP_END, 16) + " - 0x" + QByteArray::number(SMP_END, 16) + "\n";
for (int i = BMP_END/BMP_BLOCKSIZE; i < blockMap.size(); ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
if (!(i % (0x10000/SMP_BLOCKSIZE)))
out += "\n";
out += "\n ";
}
out += QByteArray::number(blockMap.at(i) + blockMap.size());
out += ", ";
}
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (out.endsWith(' '))
out.chop(1);
out += "\n";
// write the data
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
for (int i = 0; i < uniqueBlocks.size(); ++i) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (out.endsWith(' '))
out.chop(1);
out += "\n";
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
const UniqueBlock &b = uniqueBlocks.at(i);
for (int j = 0; j < b.values.size(); ++j) {
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
if (!(j % 8)) {
if (out.endsWith(' '))
out.chop(1);
out += "\n ";
}
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "0x" + QByteArray::number(b.values.at(j), 16);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += ", ";
}
}
if (out.endsWith(' '))
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out.chop(2);
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "\n};\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "#define GET_LIGATURE_INDEX(ucs4) \\\n"
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
" (ucs4 < 0x" + QByteArray::number(BMP_END, 16) + " \\\n"
" ? (uc_ligature_trie[uc_ligature_trie[ucs4>>" + QByteArray::number(BMP_SHIFT) +
"] + (ucs4 & 0x" + QByteArray::number(BMP_BLOCKSIZE-1, 16)+ ")]) \\\n"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" : (ucs4 < 0x" + QByteArray::number(SMP_END, 16) + " \\\n"
add support for non-BMP ligatures > http://www.unicode.org/versions/Unicode5.2.0/ D. Character Additions: There are three new characters in the newly-encoded Kaithi script that will require changes in implementations which make hard-coded assumptions about composition during normalization. Most new characters added to the standard with decompositions cannot be generated by the operations toNFC() or toNFKC), but these three can. Implementers should check their code carefully to ensure that it handles these three characters correctly. U+1109A KAITHI LETTER DDDHA U+1109C KAITHI LETTER RHA U+110AB KAITHI LETTER VA UCD 6.1 adds two more of them: U+1112E CHAKMA VOWEL SIGN O U+1112F CHAKMA VOWEL SIGN AU Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-04-26 16:29:08 +00:00
" ? uc_ligature_trie[uc_ligature_trie[((ucs4 - 0x" + QByteArray::number(BMP_END, 16) +
")>>" + QByteArray::number(SMP_SHIFT) + ") + 0x" + QByteArray::number(BMP_END/BMP_BLOCKSIZE, 16) + "]"
QUnicodeTables generator: Minor clean-up & stricter checks Use static_cast instead of C-style cast; ushort -> unsigned short; Avoid a lvalue in qGetProp(); Minor styling fixes. Change-Id: Ic6412e28ac9316a587d32d0e674d21f672967fd6 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2015-11-06 20:31:43 +00:00
" + (ucs4 & 0x" + QByteArray::number(SMP_BLOCKSIZE-1, 16) + ")] \\\n"
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
" : 0xffff))\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
Clean-up the Unicode tables generator code and the generated header This fixes the blocks and memory consumption reports, the whitespace issues and makes the code a bit cleaner. Since I'm the only one who does change this code, such a no-op commit could not hurt anyone or even git blame ;) Change-Id: Ib069f925a3791c82e16c368c8392bcffbfd68c53 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-06-17 01:20:59 +00:00
out += "static const unsigned short uc_ligature_map[] = {";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
for (int i = 0; i < ligatures.size(); ++i) {
if (!(i % 8)) {
if (out.endsWith(' '))
out.chop(1);
out += "\n ";
}
out += "0x" + QByteArray::number(ligatures.at(i), 16);
out += ", ";
}
if (out.endsWith(' '))
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
out.chop(2);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
out += "\n};\n\n";
return out;
}
QByteArray createCasingInfo()
{
QByteArray out;
out += "struct CasingInfo {\n"
" uint codePoint : 16;\n"
" uint flags : 8;\n"
" uint offset : 8;\n"
"};\n\n";
return out;
}
int main(int, char **)
{
initAgeMap();
initCategoryMap();
initDecompositionMap();
initDirectionMap();
initJoiningMap();
initGraphemeBreak();
initWordBreak();
initSentenceBreak();
initLineBreak();
Add QChar::Script enum ...where the values are not aliased to Common script. The old QUnicodeTables::Script enum was retained for compatibility reasons until Qt internals are updated to use QChar::script(). Using QChar::Script instead of QUnicodeTables::Script would improve both the text analysis (itemization, boundary finding) and the text shaping quality. This also a required step for switching to Hurfbuzz-NG. /* This adds 6668 more .rodata bytes */ Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5 Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
2012-12-08 03:36:49 +00:00
initScriptMap();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
readUnicodeData();
readBidiMirroring();
readArabicShaping();
readDerivedAge();
readDerivedNormalizationProps();
readSpecialCasing();
readCaseFolding();
// readBlocks();
readScripts();
readGraphemeBreak();
readWordBreak();
readSentenceBreak();
readLineBreak();
computeUniqueProperties();
QByteArray properties = createPropertyInfo();
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
QByteArray specialCases = createSpecialCaseMap();
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QByteArray compositions = createCompositionInfo();
QByteArray ligatures = createLigatureInfo();
QByteArray normalizationCorrections = createNormalizationCorrections();
QByteArray header =
"/****************************************************************************\n"
"**\n"
Update UCD to Revision 26 Include WordBreakTest.html, since a test uses sample strings from it, albeit without actually reading the file. Had to comment out more of the new tests, as at Revision 24, pending an update to harfbuzz and the text boundary detection code. Task-number: QTBUG-79631 Task-number: QTBUG-79418 Task-number: QTBUG-82747 Change-Id: I0082294b09d67ffdc6a9b5c15acf77ad3b86f65f Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2020-03-13 16:26:53 +00:00
"** Copyright (C) 2020 The Qt Company Ltd.\n"
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
"** Contact: https://www.qt.io/licensing/\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"**\n"
"** This file is part of the QtCore module of the Qt Toolkit.\n"
"**\n"
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
"** $QT_BEGIN_LICENSE:LGPL$\n"
Change copyrights from Nokia to Digia Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-19 12:28:29 +00:00
"** Commercial License Usage\n"
"** Licensees holding valid commercial Qt licenses may use this file in\n"
"** accordance with the commercial license agreement provided with the\n"
"** Software or, alternatively, in accordance with the terms contained in\n"
Update copyright headers Qt copyrights are now in The Qt Company, so we could update the source code headers accordingly. In the same go we should also fix the links to point to qt.io. Outdated header.LGPL removed (use header.LGPL21 instead) Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing combination. New header.LGPL-COMM taken in the use file which were using old header.LGPL3 (src/plugins/platforms/android/extract.cpp) Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license combination Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
2015-01-28 08:44:43 +00:00
"** a written agreement between you and The Qt Company. For licensing terms\n"
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
"** and conditions see https://www.qt.io/terms-conditions. For further\n"
"** information use the contact form at https://www.qt.io/contact-us.\n"
Change copyrights from Nokia to Digia Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-19 12:28:29 +00:00
"**\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"** GNU Lesser General Public License Usage\n"
Change copyrights from Nokia to Digia Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
2012-09-19 12:28:29 +00:00
"** Alternatively, this file may be used under the terms of the GNU Lesser\n"
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
"** General Public License version 3 as published by the Free Software\n"
"** Foundation and appearing in the file LICENSE.LGPL3 included in the\n"
"** packaging of this file. Please review the following information to\n"
"** ensure the GNU Lesser General Public License version 3 requirements\n"
"** will be met: https://www.gnu.org/licenses/lgpl-3.0.html.\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"**\n"
Updated license headers From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some exceptions, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one (in those files which will be under GPL 3 with exceptions) Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
2016-01-15 12:36:27 +00:00
"** GNU General Public License Usage\n"
"** Alternatively, this file may be used under the terms of the GNU\n"
"** General Public License version 2.0 or (at your option) the GNU General\n"
"** Public license version 3 or any later version approved by the KDE Free\n"
"** Qt Foundation. The licenses are as published by the Free Software\n"
"** Foundation and appearing in the file LICENSE.GPL2 and LICENSE.GPL3\n"
"** included in the packaging of this file. Please review the following\n"
"** information to ensure the GNU General Public License requirements will\n"
"** be met: https://www.gnu.org/licenses/gpl-2.0.html and\n"
"** https://www.gnu.org/licenses/gpl-3.0.html.\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"**\n"
"** $QT_END_LICENSE$\n"
"**\n"
"****************************************************************************/\n\n";
QByteArray note =
Fix a bug in the generation of the Unicode joining property data The code that was supposed to initialize the joining property of characters to the correct defaults was actually applied after reading in the data from the Unicode file, and was in a couple of cases overwriting explicitly specified data in ArabicShaping.txt Task-number: QTBUG-63191 Change-Id: Ie35261039b2211a827322ca11afacd9555ccefc7 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2017-12-11 10:28:11 +00:00
"/* This file is autogenerated from the Unicode " DATA_VERSION_S " database. Do not edit */\n\n";
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
QByteArray warning =
"//\n"
"// W A R N I N G\n"
"// -------------\n"
"//\n"
"// This file is not part of the Qt API. It exists for the convenience\n"
"// of internal files. This header file may change from version to version\n"
"// without notice, or even be removed.\n"
"//\n"
"// We mean it.\n"
"//\n\n";
Move text-related code out of corelib/tools/ to corelib/text/ This includes byte array, string, char, unicode, locale, collation and regular expressions. Change-Id: I8b125fa52c8c513eb57a0f1298b91910e5a0d786 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2019-05-27 17:13:54 +00:00
QFile f("../../src/corelib/text/qunicodetables.cpp");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.open(QFile::WriteOnly|QFile::Truncate);
f.write(header);
f.write(note);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
f.write("#include \"qunicodetables_p.h\"\n\n");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.write("QT_BEGIN_NAMESPACE\n\n");
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
f.write("namespace QUnicodeTables {\n\n");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.write(properties);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
f.write("\n");
f.write(specialCases);
f.write("\n");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.write(compositions);
f.write(ligatures);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
f.write("\n");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.write(normalizationCorrections);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
f.write("} // namespace QUnicodeTables\n\n");
f.write("using namespace QUnicodeTables;\n\n");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.write("QT_END_NAMESPACE\n");
f.close();
Move text-related code out of corelib/tools/ to corelib/text/ This includes byte array, string, char, unicode, locale, collation and regular expressions. Change-Id: I8b125fa52c8c513eb57a0f1298b91910e5a0d786 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2019-05-27 17:13:54 +00:00
f.setFileName("../../src/corelib/text/qunicodetables_p.h");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.open(QFile::WriteOnly | QFile::Truncate);
f.write(header);
f.write(note);
f.write(warning);
f.write("#ifndef QUNICODETABLES_P_H\n"
"#define QUNICODETABLES_P_H\n\n"
Fix a bug in the generation of the Unicode joining property data The code that was supposed to initialize the joining property of characters to the correct defaults was actually applied after reading in the data from the Unicode file, and was in a couple of cases overwriting explicitly specified data in ArabicShaping.txt Task-number: QTBUG-63191 Change-Id: Ie35261039b2211a827322ca11afacd9555ccefc7 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2017-12-11 10:28:11 +00:00
"#include <QtCore/private/qglobal_p.h>\n\n"
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
"#include <QtCore/qchar.h>\n\n"
"QT_BEGIN_NAMESPACE\n\n");
Fix a bug in the generation of the Unicode joining property data The code that was supposed to initialize the joining property of characters to the correct defaults was actually applied after reading in the data from the Unicode file, and was in a couple of cases overwriting explicitly specified data in ArabicShaping.txt Task-number: QTBUG-63191 Change-Id: Ie35261039b2211a827322ca11afacd9555ccefc7 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2017-12-11 10:28:11 +00:00
f.write("#define UNICODE_DATA_VERSION " DATA_VERSION_STR "\n\n");
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.write("namespace QUnicodeTables {\n\n");
f.write(property_string);
QUnicodeTables: pack Properties struct GCC doesn't like the sequence : 5 : 5 : 8 : 6 : 8 and inserts a :6 padding between the :5 and the :8 and a :2 padding between the :6 and the :8, growing the bitfield by 8 bits of embedded padding and another byte to bring the struct back to sizeof % 2 == 0. Fix by reshuffling the elements and adding a static_assert for the next round. Saves ~5KiB in QtCore executable size. Change-Id: I4758a6f48ba389abc2aee92f60997d42ebb0e5b8 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2019-09-03 21:40:56 +00:00
f.write(sizeOfPropertiesStructCheck);
QUnicodeTables: some internal API renamings enums GraphemeBreak, WordBreak, and SentenceBreak has been renamed to GraphemeBreakClass, WordBreakClass, and SentenceBreakClass respectively, their values has been renamed to contain a '_' as logical enum-value separator (just like many other nums in Qt, e.g. LineBreakClass); *BreakFormat has been replaced with *Break_Extend (some format characters are kind of subtype of the extender characters, not vice versa). Change-Id: I9ddbcf8848da87409736c2d6d1798a62fa28cab8 Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
2012-06-17 01:55:07 +00:00
f.write(grapheme_break_class_string);
f.write(word_break_class_string);
f.write(sentence_break_class_string);
qunicodetables generator: improve the output and the generated code better memory usage report; an additional asserts with conditions the implementation is depends on; a namespace for the internal static data; styling fixes Change-Id: Id4048ff6104c56b5f590f9ac6fbf7c0bce79ec47 Reviewed-by: Lars Knoll <lars.knoll@nokia.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2012-04-23 03:00:16 +00:00
f.write(line_break_class_string);
Initial import from the monolithic Qt. This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
2011-04-27 10:05:43 +00:00
f.write(methods);
f.write("} // namespace QUnicodeTables\n\n"
"QT_END_NAMESPACE\n\n"
"#endif // QUNICODETABLES_P_H\n");
f.close();
qDebug() << "maxMirroredDiff = " << hex << maxMirroredDiff;
qDebug() << "maxLowerCaseDiff = " << hex << maxLowerCaseDiff;
qDebug() << "maxUpperCaseDiff = " << hex << maxUpperCaseDiff;
qDebug() << "maxTitleCaseDiff = " << hex << maxTitleCaseDiff;
qDebug() << "maxCaseFoldDiff = " << hex << maxCaseFoldDiff;
#if 0
// dump(0, 0x7f);
// dump(0x620, 0x640);
// dump(0x10000, 0x10020);
// dump(0x10800, 0x10820);
qDebug("decompositionLength used:");
int totalcompositions = 0;
int sum = 0;
for (int i = 1; i < 20; ++i) {
qDebug(" length %d used %d times", i, decompositionLength.value(i, 0));
totalcompositions += i*decompositionLength.value(i, 0);
sum += decompositionLength.value(i, 0);
}
qDebug(" len decomposition map %d, average length %f, num composed chars %d",
totalcompositions, (float)totalcompositions/(float)sum, sum);
qDebug("highest composed character %x", highestComposedCharacter);
qDebug("num ligatures = %d highest=%x, maxLength=%d", numLigatures, highestLigature, longestLigature);
qBubbleSort(ligatures);
for (int i = 0; i < ligatures.size(); ++i)
qDebug("%s", ligatures.at(i).data());
// qDebug("combiningClass usage:");
// int numClasses = 0;
// for (int i = 0; i < 255; ++i) {
// int num = combiningClassUsage.value(i, 0);
// if (num) {
// ++numClasses;
// qDebug(" combiningClass %d used %d times", i, num);
// }
// }
// qDebug("total of %d combining classes used", numClasses);
#endif
}
Reference in New Issue Copy Permalink