Pack some of the arrays that contain locale data more tightly. The
AlphaCode struct is a char[4] but always holds only [a-z]{,3} which
could be fit into 16 bits, halving the size of an AlphaCode struct.
With the new constructor the initialization of the AlphaCode struct
also changes - modify qlocalexml2cpp.py to reflect this change and
regenerate the languageCodeList.
Fixes: QTBUG-105050
Change-Id: I2b1e93ab7cc3f2d667bf67b45769b74a15211931
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Revise a comment in ldml.py about Suzhou "digits", since it turns out
they aren't the same as hanidec, which is far from contiguous.
Change-Id: Ia3947dbc5a927772026e55fe197c8ebce2540da2
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
New languages (and one local for each) added with v42
- Haryanvi
- Moksha
- Northern Frisian
- Obolo
- Pijin
- Rajasthani
- Toki Pona
It also appears that Canada has changed its date format. Modify the
relevant test case to reflect this change.
Task-number: QTBUG-110333
Pick-to: 6.5
Change-Id: Ia8975c2866cd54c9e565543d05bacd52f4987909
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
We can easily enough obtain the root of the present source tree using
the value of __file__, so might as well do so.
Change-Id: If14773ac1127278b6018a090c0b376437b9c6eec
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.
Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
This commit extends functionality for QLocale::codeToLanguage()
and QLocale::languageToCode() by adding an additional argument
that allows selection of the ISO 639 code-set to consider for
those operations.
The following ISO 639 codes are supported:
* Part 1
* Part 2 bibliographic
* Part 2 terminological
* Part 3
As a result of this change the codeToLanguage() overload without
the additional argument now returns a Language value if it matches
any know code. Previously a valid language was returned only if
the function argument matched the first code defined for that
language from the above list.
[ChangeLog][QtCore][QLocale] Added overloads for codeToLanguage()
and languageToCode() that support specifying which ISO 639 codes
to consider.
Fixes: QTBUG-98129
Change-Id: I4da8a89e2e68a673cf63a621359cded609873fa2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Update the locale generation script to support Kaingang and
Nheengatu languages. These are new in CLDR v40. Regenerate
the locale data.
Task-number: QTBUG-94358
Change-Id: I5195d5161d8c4d9f17129bbcfde39dfd3fcf1cd5
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
FTP is insecure and is not supported by modern browsers anymore.
See also: https://mywiki.wooledge.org/FtpMustDie
Change-Id: Iad65d29912e79a4f3fadb9317bb5d9c5fe9b68d7
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
They are not needed. Iterations over the table track their sizes.
The size-of-table constants just needed their -1s removed.
Incidentally use std::size() rather than sizeof(array)/sizeof(element).
Change-Id: Ie20eef9f6f5786d93c10b830a87e006d3c5bcc1a
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Use context manager interface (with statement) to atomically update source
files. This ensures that all files are properly closed and the temporary
file is removed even in case of errors.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I18cd96f1d03e467dea6212f6576a41e65f414ce1
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Using NamedTemporaryFile instead mkstemp + fdopen simplifies the code.
It also makes it easier to switch to using context managers for handling
source file modification.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Ibeae840ac6dde3d0b49cd7f985cfa6cd775b7f47
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
pathlib's API is more modern and easier to use than os.path. It
also allows to distinguish between paths and other strings in type
annotations.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Ie6d9b4e35596f7f6befa4c9635f4a65ea3b20025
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
arparse is the standard way to parse command line arguments in Python.
It provides help and usage information for free and is easier to extend
than a custom argument parser.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I1e4c9cd914449e083d01932bc871ef10d26f0bc2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Replace most uses of str.format() and string arithmetic by f-strings.
This results in more compact code and the code is easier to read
when using an appropriate editor.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I3409f745b5d0324985cbd5690f5eda8d09b869ca
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The schema is in RelaxNG Compact syntax. It can be used to validate
files produced by the cldr2qlocalexml.py script and also gives an
overview of the file format.
Change-Id: I344978f2201c5e67e236ab580a12ad33262f33cb
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This is the standard way to call base class methods in Python 3 and
it is shorter than the custom one used now.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Ifaff591a46e92148fbf514856109ff794a50c9f7
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Instead of implementing all the intricacies of a cmp for the python
sort-function, support for which is due to be dropped at Python 3 in
any case, implement a much simpler key function that achieves the same
result.
In the process, eliminate the ugly kludge of setting an attribute on a
function to, in effect, communicate with it via a global. Instead,
instantiate a class, that wraps the value previously given to the
attribute and whose instance provides the key-function.
Thanks to Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io> for
pointing out that a key function is the way of the future - and
sorted() is a nicer way to sort.
Pick-to: 6.2
Change-Id: Icf1ed5597fedf420d054fbc860e3e7fc6615875c
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
The ordering function used to sort the locale data generated for
QLocale attempted to sort the default territory for a given language
and script before other territories, but was too tangled for it to be
obvious this is what it was doing. The result turned out to be
non-transitive. Replace with code that implements the same preference
but only applies it where the result is compatible with transitivity.
This leads to a shuffling of the order of the Serbian-language
locales, which sorts the Cyrillic ones before the Latin ones. This is
consistent with my reading of the CLDR data, which fills in Cyrillic
and Serbia for Serbian; Serbian/Cyrillic/Serbia did previously sort
before all other Serbian variants.
Thanks to Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io> for
discovering the non-transitivity.
Pick-to: 6.2
Change-Id: I0ce9f78e620e714f980f32b85b7100ed0f92ad74
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
The behavior of StopIteration in generators was changed in Python 3
(see https://www.python.org/dev/peps/pep-0479/). Not raising that
exception makes it easier to port the code to Python 3.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Iac6e3f6f1e1e8ef3a1a0d89b19d2ac2d186434f5
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
IOError does not have property 'message' in Python 3. Instead of
attempting to access it, just use the string representation of
the exception object. This produces the error message possibly combined
with additional arguments in both Python 2 and Python 3.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Icb198a409e7f80b832e474d8390b770fdeacc6c2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Name 'stem' is undefined inside CalendarDataWriter.write(). The error
was repoted by flake8.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Ib816b40d0bde2afd3112da76deee0ce39985693a
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This way the output is easier to compare between versions.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: If4053c574c4ad200a179b06276bd889f2cb9e1c6
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Output of cldr2qlocalexml.py looks weird without the final new line.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I5d675e475c57cdc8101887c39052007ba0a19857
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This is not a script that can be run independently.
Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I82a93b9ab37ae759b789058d48e94298ecd29b6f
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
We should at least know when members of QLocale's enums aren't adding
any value, and it may make sense to deprecate the unused ones.
Change-Id: Icf202f81d2a35904c13ccdc202d41985bcb3f2e6
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Change the nomenclature used in the scripts and the QLocaleXML data
format to use "territory" and "territories" in place of "country" and
"countries". Does not change the generated source files.
Change-Id: I4b208d8d01ad2bfc70d289fa6551f7e0355df5ef
Reviewed-by: JiDe Zhang <zhangjide@uniontech.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
These variables provide mappings, not lists, so name them non-deceptively.
Change-Id: Idf15e78ad73790bc86dd8b9d4f248d1c4f73993c
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
The only reason cldr.py imported enumdata was so as to pass what it
imported to writer.enumData(); that method might as well do the import
itself.
Change-Id: Ie77dcd29058f926b8cca4deef35837f30505859f
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
It's now a data-only module. The callers of its code-to-ID functions
have, for some time now, been rearranging its mappings to get at data
efficiently.
Change-Id: Ia16dcaa767203cdf3b81a96bd51793491ad41563
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
The use of "Country" is misleading as some entries in the enumeration
are not countries (eg, HongKong), for all that most are. The Unicode
Consortium's Common Locale Data Repository (CLDR, from which QLocale's
data is taken) calls these territories, so introduce territory-based
names and prepare to deprecate the country-based ones in due course.
[ChangeLog][QtCore][QLocale] QLocale now has Territory as an alias for
its Country enumeration, and associated territory-based names to match
its country-named methods, to better match the usage in relevant
standards. The country-based names shall in due course be deprecated
in favor of the territory-based names.
Fixes: QTBUG-91686
Change-Id: Ia1ae1ad7323867016186fb775c9600cd5113aa42
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
No change to QLocale's data, one addition to the Windows time-zone
data. What was formerly "Us Mountain Standard time / Canada" is now
Yukon Standard Time.
Fixes: QTBUG-89784
Pick-to: 6.0 5.15
Change-Id: I4c9a23620e74ea379be8a4c5ba0896d35fe9b594
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The comments in enumdata.py indicating macrolanguages meant nothing to
me, until I stumbled on a reference that lead me to ISO 639's usage of
the term. Add a minimal explanation to save such confusion for others.
Change-Id: Ia1d849d93a1d94c04c8c461debdecf879e9a7db5
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
We were extracting several candidate display names from CLDR for each
currency, joining them with semicolons, storing in a table, then using
only the first entry from the list - where we should probably have
used the first non-empty entry in any case.
So instead extract the first non-empty candidate name from CLDR and
store that simply, saving the need for semicolon-joining or parsing
out the first entry from the thus-joined list. This significantly
reduces the size of the currency name data table.
Change-Id: I201d0528348d5fcb9eceb5df86211b9c77de3485
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Follow through on a comment from 2012: sort the likely subtag array
(in the CLDR update script) and use bsearch to find entries in it.
This simplifies QLocaleXmlReader.likelyMap() slightly, moving the
detection of last entry to LocaleDataWriter.likelySubtags(), but
requires collecting all likely sub-tag mapping pairs (rather than just
passing them through from read to write via generators) in order to
sort them.
Change-Id: Ieb6875ccde1ddbd475ae68c0766a666ec32b7005
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Binary-incompatible change: change the numeric values of QLocale's
Language, Script and Country enums, as encouraged by a comment in the
generator script enumdata.py and clarify documentation around that.
In the process (since I was changing almost every line anyway),
convert the dictionary values from (mutable) lists of length two to
tuples, since they are (and should be) immutable data.
Change-Id: I26222bce45b9f5074b1d81ed70015a75ac34adcd
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Our enumdata.py namings of countries had fallen somewhat out of sync
with CLDR's names. In the process, support including hyphenation in
the unsquashed name, along with spacing. Distinguish, in comments,
between older renamings and those first seen in Qt6.
Change-Id: I91ec444bf35222ab6a9332e389ace19cca0e4fdf
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Move the repeated List suffix to the __enumTable() helper, where half
the parameter's uses were having to snip it off anyway.
Change-Id: Ia396e87e59ceeb81fc4b0890a86934dc67da10cb
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Compare the code->name mappings we're using to the ones CLDR's
common/main/en.xml provides; report discrepancies. Tolerate tags
missing from en.xml if they're known to the locale-inheritance
machinery.
Change-Id: Ibe96c18bf55984a35de3b3644f3586a9f30720b2
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Requires subsequent re-numbering of the enum tables to eliminate gaps,
before locale data can be regenerated. However, it will work with the
present locale data, since it merely loses the means to use some names
for which the available data was just the name and code. This implies
a transient issue of recognising some codes for which there is no
actual enum member; but relevant code will work as before, finding
nothing but the code and its name. This shall be resolved by a coming
BiC change to resort the language, country and script codes, changing
the numbering (almost) completely.
[ChangeLog][QtCore][QLocale] Various obsolete language and country
codes have been removed. Some lacked locale data, others were obsolete
aliases. All have been deprecated in 5.15.
Task-number: QTBUG-84669
Change-Id: I45fc76a5f2f6c3b0ea3c1bb61e917da984183783
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Routine update by running scripts, ignoring clang-format's extensive
grumbles. Added notes to util/locale_database/'s README, on the need
for that, and enumdata.py, on when to add entries. As usual, several
new locales are also added, for existing languages, territories and
scripts.
[ChangeLog][QtCore][QLocale] Updated to new version of CLDR (the
Unicode Consortium's Common Locale Data Repository) v37.
Fixes: QTBUG-84669
Pick-to: 5.15
Change-Id: Ib76848bf4bd1219180faf46820077e8d8049a4e3
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The code pervasively presumes their values can be held in a ushort, so
make sure the compiler knows we expect that to work (and doesn't
complain about narrowing when we do convert them to ushort).
Change-Id: Idde7be6cceee8a6dae333c5b1d5a0120fec32e4a
Reviewed-by: Andrei Golubev <andrei.golubev@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The script told me the wrong path to pass as first argument, so
correct that; and the README didn't mention the need to run it.
CLDR v37 makes no change to the actual generated data, though.
Tweaked wording of a comment in the script.
Task-number: QTBUG-84669
Change-Id: I56b510c666f414d9719cef650aeec6192c4fde6e
Reviewed-by: Andrei Golubev <andrei.golubev@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
This time based on grepping to also include documentation, tests and
examples previously missed by the automatic tool.
Change-Id: Ied1703f4bcc470fbc275f759ed5b7c588a5c4e9f
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
This only arises when the system locale tells us to use its zero as
our zero digit, since no CLDR locale uses it by default. Adapt an
MS-specific QLocale::system() test to use Suzhou numbering, so as to
test this.
While updating the locale-restoration code to also restore the digits
being set in that test, add restore code for the long time format,
where previously only the short time format was restored. Add a
comment to make it less likely one of those shall be missed in future.
Fixes: QTBUG-85409
Change-Id: I343324bb563ee0e455dfe77d4825bf8c3082ca30
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Read three more values from CLDR and add a byte to the bit-fields at
the end of QLocaleData, indicating the three group sizes. This adds
three new parameters to various low-level formatting functions. At the
same time, rename ThousandsGroup to GroupDigits, more faithfully
expressing what this (internal) option means.
This replaces commit 27d1391280 with a
fuller implementation that handles digit-grouping in any of the ways
that CLDR supports. The formerly "Indian" formatting now also applies
to at least some locales for Bangladesh, Bhutan and Sri Lanka.
Fixed Costa Rica currency formatting test that wrongly put a separator
after the first digit; the locale (in common with several Spanish
locales) requires at least two digits before the first separator.
[ChangeLog][QtCore][Important Behavior Changes] Some locales require
more than one digit before the first grouping separator; others use
group sizes other than three. The latter was partially supported (only
for India) at 5.15 but is now systematically supported; the former is
now also supported.
Task-number: QTBUG-24301
Fixes: QTBUG-81050
Change-Id: I4ea4e331f3254d1f34801cddf51f3c65d3815573
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
