Commit Graph

851 Commits

Author SHA1 Message Date
Mårten Nordheim
6ffe9b1c42 pro2cmake.py: Generate .cmake.conf files for versioning
And create one for QtBase at the same time.

Fixes: QTBUG-83835
Change-Id: Icc6b022165a57bd4e22c23bdb0016522b99a5b80
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-30 17:44:03 +00:00
Tor Arne Vestbø
ebc3350be7 pro2cmake: Translate (QT|QMAKE_USE)_FOR_PRIVATE into PRIVATE_MODULE_INTERFACE
Fixes: QTBUG-83063
Change-Id: I5b02cd236ef0fd3a9ed389294c05f8e3024463cd
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-30 14:20:26 +02:00
Tor Arne Vestbø
4618fa6f8d Fix pro2cmake help text
Change-Id: I84980dda981c50da159e4bb81b08190b882d6978
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-30 14:20:26 +02:00
Fabian Kosmale
14c713e759 pro2cmake: convert QMLTYPES_FILENAME to QT_QMLTYPES_FILENAME
Change-Id: I9afc6d4c0c9b3fd87ef28f15da0a984d778b83ac
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-30 10:13:56 +02:00
Joerg Bornemann
062b50abff CMake: Namespace all our IMPORTED targets
CMake IMPORTED targets should be namespaced so that CMake knows that
the name refers to a target and not a file.

Use the existing WrapXXX naming scheme where applicable.

Fixes: QTBUG-83773
Change-Id: I5b0b722c811200c56c260c69e76940a625228769
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-29 23:08:32 +02:00
Marc Mutz
7b04e0012b QUnicodeTables: port to charNN_t
This makes existing calls passing uint or ushort ambiguous, so
fix all the callers. There do not appear to be callers outside
QtBase. In fact, the ...BreakClass() functions appear to be
utterly unused.

Change-Id: I1c2251920beba48d4909650bc1d501375c6a3ecf
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-27 13:08:41 +02:00
Marc Mutz
20cdf807b1 QChar: port low-level functions from uint/ushort to char32/16_t
Now that the standard gives us proper types for UTF-16 and UTF-32
characters, use them. Will eventually make the code much easier to
read than today, where uint could be an index as well as a char32_t.

It also ensures that the result of e.g. QChar::highSurrogate() can
still be implicitly converted to a QChar now that the
QChar(non-characater-integral-types) ctors are being made explicit.

[ChangeLog][QtCore][QChar] All low-level functions
(e.g. highSurrogate()) now take and return char16_t instead of ushort
and char32_t instead of uint.

Change-Id: I9cd8ebf6fb998fe1075dae96c7c4484a057f0b91
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
2020-04-24 12:45:53 +02:00
Joerg Bornemann
4545eadd3e CMake: Port the 'static_runtime' feature
This feature is Windows-only and must be turned on manually.
For MSVC it sets the MSVC_RUNTIME_LIBRARY target property.
For MinGW it adds the -static linker flag.

Change-Id: I9da3b88d545b34bc34a3a80301b2dd1b5986fa88
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-23 10:09:15 +02:00
Leander Beernaert
6fbeef4c6b CMake: Add qt6_add_plugin public API
This patch adds a publicly callable qt6_add_plugin() API to create
plugins. This API is meant to cover cases such as the plugandpaint
example.

This patch also renames qt_add_plugin to qt_internal_add_plugin in order
to avoid clashes with the public API. To avoid breaking the existing
projects, a compatibility wrapper function is enabled by default unless
QT_DISABLE_QT_ADD_PLUGIN_COMPATIBILITY is specified.

Fixes: QTBUG-82961
Change-Id: If5b564a8406c90434f1bdad0b8df76d3e6626b5f
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-22 09:51:12 +02:00
Joerg Bornemann
2a0585e7cb CMake: Port the 'pkg-config' feature
Change-Id: If571208aaaba8c1d0af834e8ae11869872d42135
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
2020-04-21 09:01:39 +02:00
Joerg Bornemann
3eed6d76b7 CMake: Port the 'ccache' feature
This maps to the CMake variable QT_USE_CCACHE.

Change-Id: I3258027301284d907f6ecde6c65d2c0dde8f0a11
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-17 23:10:11 +02:00
Joerg Bornemann
8a5ffc4f73 CMake: Port the 'ltcg' feature
This maps to the CMake variable CMAKE_INTERPROCEDURAL_OPTIMIZATION.

Change-Id: Id0ce48f176b95c27e74ab80276e89503b1660f79
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-17 23:10:09 +02:00
Joerg Bornemann
9b45ca7411 CMake: Port the 'cross_compile' feature
Change-Id: Iccd1d55e95797740a4a8689462ce9ab1e49a62c1
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-17 20:42:42 +02:00
Joerg Bornemann
6f5c91e985 configurejson2cmake: Remove mention of the system-xcb feature
This feature was removed in commit 60588e1a.

Change-Id: I061410dfab13a2210474014892d1bc828a5b21cf
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-17 20:41:44 +02:00
Alexandru Croitor
67ee92f4d8 CMake: Handle automatic rpath embedding correctly
Instead of using CMAKE_INSTALL_RPATH to embed an absolute path
to prefix/libdir into all targets, use the more sophisticated aproach
that qmake does.

For certain targets (modules, plugins, tools) use relative rpaths.
Otherwise embed absolute paths (examples, regular binaries).
Installed tests currently have no rpaths.

On certain platforms rpaths are not used (Windows, Android,
iOS / uikit).

Frameworks, app bundles and shallow bundles should also be handled
correctly.

Additional rpaths can be provided via QT_EXTRA_RPATHS variable
(similar to the -R option that configure takes).

Automatic embedding can be disabled either via QT_FEATURE_rpath=OFF
or QT_DISABLE_RPATH=ON.

Note that installed examples are not relocatable at the moment (due
to always having an absolute path rpath), so this is a missing feature
compared to qmake. This is due to missing information on where
examples will be installed, so a relative rpath can not be computed.

By default a Qt installation is relocatable, so there is no need to
pass -DQT_EXTRA_RPATHS=. like Coin used to do with qmake e.g. -R .

Relative rpaths will have the appropriate 'relative base' prefixed
to them (e.g $ORIGIN on linux and @loader_path on darwin platforms).
There is currently no support for other platforms that might have a
different 'relative base' than the ones mentioned above.

Any extra rpaths are saved to BuildInternalsExtra which are re-used
when building other repositories.

configurejson2cmake modified to include correct conditions for the
rpath feature.

It's very likely that we will need a new qt_add_internal_app()
function for gui apps that are to be installed to prefix/bin.
For example for Assistant from qttools. Currently such apps
use qt_add_executable().
The distinction is necessary to make sure that relative rpaths are
embedded into apps, but not executables (which tests are part of).

Amends e835a6853b

Task-number: QTBUG-83497
Change-Id: I3510f63c0a59489741116cc8ec3ef6a0a7704f25
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2020-04-17 15:59:25 +02:00
Eskil Abrahamsen Blomfeldt
ad5aee2e34 Get rid of QT_OPENGL_ES*
The QT_OPENGL_ES* macros are leftovers from an earlier,
ad hoc configuration system, which has since been
replaced by QT_CONFIG. To clean things up in Qt 6,
we use the new way instead.

Task-number: QTBUG-83467
Change-Id: I578dc7695bff9d5ee303b22e44f60fee22fe0c28
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
2020-04-17 11:50:24 +02:00
Leander Beernaert
30f69f7ff6 CMake: Add library mapping for xcb-fixes
Change-Id: Ida893674fa0782446acdac4ee5dfec4849d4605f
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-15 16:25:19 +02:00
Leander Beernaert
812cc76b02 CMake: Update pro2cmake and friend to convert QtVirtualkeyboard
Expand $$MODULE_BASE_DIR to ${QT_BUILD_DIR}.

Add library mappings for Hunspell.

Add test mappings for QtVirtualKeyboard regarding 3rdparty Hunspell and
T9Write.

Change-Id: Ief007456d0471cbcf9a03d819291edec5f63680f
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-15 14:15:52 +00:00
Leander Beernaert
a77cd07b24 CMake: pro2cmake expand variables for qt_add_resource()
Change-Id: Ida1196645dabcb0c3aee210d9bc5ee5632a9ccc5
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
2020-04-14 13:19:23 +02:00
Mikhail Svetkin
ae57e4fba5 cmake: Add library mapping for QtHttpServer
Change-Id: I3e12f164f7f6891c60c0d9da700b888850021e53
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-09 20:11:31 +02:00
Lars Knoll
c6cdf38e75 Change qHash() to work with size_t instead of uint
This is required, so that QHash and QSet can hold more
than 2^32 items on 64 bit platforms.

The actual hashing functions for strings are still 32bit, this will
be changed in a follow-up commit.

Change-Id: I4372125252486075ff3a0b45ecfa818359fe103b
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2020-04-09 20:03:25 +02:00
Alexandru Croitor
066e8a4557 CMake: Handle missing packages in project compile tests
If a find_package() in a try_compile project doesn't find a package,
and we then link against a non-existent target, the configuration
failure of the compile test also fails the configuration of the
project.

To avoid that, separate library targets from non-targets, and make sure
to only link against the targets if they exist.

pro2cmake now outputs modified compile test project code which iterates
over targets and non-target libraries, and links against them when
needed.

Change-Id: Ib0f4b5f07af13929c42d01a661df2cabdf9b926b
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
2020-04-09 11:26:42 +02:00
Qt Forward Merge Bot
c937ed8af4 Merge "Merge remote-tracking branch 'origin/5.15' into dev" 2020-04-08 22:04:23 +02:00
Alexandru Croitor
e0346df1b2 CMake: Handle finding of OpenSSL headers correctly
In Coin when provisioning for Android, we download and configure
the OpenSSL package, but don't actually build it. This means that
find_package(OpenSSL) can find the headers, but not the library,
and thus the package is marked as not found.

Previously the openssl_headers feature used the result of finding
the OpenSSL package, which led to it being disabled in the above
described Android case.

Introduce 2 new find scripts FindWrapOpenSSL and
FindWrapOpenSSLHeaders. FindWrapOpenSSLHeaders wraps FindOpenSSL,
and checks if the headers were found, regardless of the OpenSSL_FOUND
value, which can be used for implementing the openssl_headers feature.

FindWrapOpenSSL uses FindWrapOpenSSLHeaders, and simply wraps the
OpenSSL target if available.

The find scripts also have to set CMAKE_FIND_ROOT_PATH for Android.
Otherwise when someone passes in an OPENSSL_ROOT_DIR, its value will
always be prepended to the Android sysroot, causing the package not
to be found.

Adjust the mapping in helper.py to use the targets created by these
find scripts. This also replaces the openssl/nolink target.

Adjust the projects and tests to use the new target names.

Adjust the compile tests for dtls and oscp to use the
WrapOpenSSLHeaders target, so that the features can be enabled even
if the library is dlopen-ed (like on Android).

Task-number: QTBUG-83371
Change-Id: I738600e5aafef47a57e1db070be40116ca8ab995
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
2020-04-08 22:03:24 +02:00
Alexandru Croitor
0a13c3a3f0 CMake: pro2cmake: Handle $$PWD in included .pri files correct-ish
Previously the $$PWD was evaluated within the including .pro file,
which generated incorrect relative paths to source files.

Now we use a horrible hack to evaluate keys ending with SOURCES
and HEADERS. If such is a case, use a map_file transformer which
will use the included scope, thus creating correct relative paths.

Fixes projects in qtdeclarative like src/qmltypregistrar and qmllint.

Checked that it doesn't break projects in qtdeclarative and qtbase.

Change-Id: I21f1e4c638c2cf8d0f67e94e1a583ebc54c175a2
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
2020-04-08 21:58:41 +02:00
Qt Forward Merge Bot
8823bb8d30 Merge remote-tracking branch 'origin/5.15' into dev
Conflicts:
	examples/opengl/doc/src/cube.qdoc
	src/corelib/global/qlibraryinfo.cpp
	src/corelib/text/qbytearray_p.h
	src/corelib/text/qlocale_data_p.h
	src/corelib/time/qhijricalendar_data_p.h
	src/corelib/time/qjalalicalendar_data_p.h
	src/corelib/time/qromancalendar_data_p.h
	src/network/ssl/qsslcertificate.h
	src/widgets/doc/src/graphicsview.qdoc
	src/widgets/widgets/qcombobox.cpp
	src/widgets/widgets/qcombobox.h
	tests/auto/corelib/tools/qscopeguard/tst_qscopeguard.cpp
	tests/auto/widgets/widgets/qcombobox/tst_qcombobox.cpp
	tests/benchmarks/corelib/io/qdiriterator/qdiriterator.pro
	tests/manual/diaglib/debugproxystyle.cpp
	tests/manual/diaglib/qwidgetdump.cpp
	tests/manual/diaglib/qwindowdump.cpp
	tests/manual/diaglib/textdump.cpp
	util/locale_database/cldr2qlocalexml.py
	util/locale_database/qlocalexml.py
	util/locale_database/qlocalexml2cpp.py

Resolution of util/locale_database/ are based on:
https://codereview.qt-project.org/c/qt/qtbase/+/294250
and src/corelib/{text,time}/*_data_p.h were then regenerated by
running those scripts.

Updated CMakeLists.txt in each of
	tests/auto/corelib/serialization/qcborstreamreader/
	tests/auto/corelib/serialization/qcborvalue/
	tests/auto/gui/kernel/
and generated new ones in each of
	tests/auto/gui/kernel/qaddpostroutine/
	tests/auto/gui/kernel/qhighdpiscaling/
	tests/libfuzzer/corelib/text/qregularexpression/optimize/
	tests/libfuzzer/gui/painting/qcolorspace/fromiccprofile/
	tests/libfuzzer/gui/text/qtextdocument/sethtml/
	tests/libfuzzer/gui/text/qtextdocument/setmarkdown/
	tests/libfuzzer/gui/text/qtextlayout/beginlayout/
by running util/cmake/pro2cmake.py on their changed .pro files.

Changed target name in
	tests/auto/gui/kernel/qaction/qaction.pro
	tests/auto/gui/kernel/qaction/qactiongroup.pro
	tests/auto/gui/kernel/qshortcut/qshortcut.pro
to ensure unique target names for CMake

Changed tst_QComboBox::currentIndex to not test the
currentIndexChanged(QString), as that one does not exist in Qt 6
anymore.

Change-Id: I9a85705484855ae1dc874a81f49d27a50b0dcff7
2020-04-08 20:11:39 +02:00
Alexandru Croitor
0bfbe10ff2 CMake: Implement proper exclusion of tools including installing
The previous approach didn't work for prefix builds. While a target
might be excluded from building via EXCLUDE_FROM_ALL property, when
calling make install it would still try to install the target and
thus fail.

It's not possible to modify an install() command in a post-processing
step, so we switch the semantics around.

pro2cmake will now write a
qt_exclude_tool_directories_from_default_target() call before adding
subdirectories. This will set an internal variable with a list
of the given subdirectories, which is checked by qt_add_executable.

If the current source dir matches one of the given subdirectories,
the EXCLUDE_FROM_ALL property is set both for the target and the
qt_install() command.

This should fix the failing Android prefix builds of qttools.

Amends 622894f96e

Change-Id: Ia19323a2ef72a3fb9cb752ad2d4f2742269d11c4
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2020-04-07 23:22:33 +02:00
Joerg Bornemann
067ae2df45 CMake: Port the 'static' feature
Now that we have the 'shared' feature implemented we can easily port
the 'static' feature.

Change-Id: Ia9b54b68d532d73c3d62d12a86c9e8b83e7909c8
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-06 21:42:53 +02:00
Edward Welbourne
727afdf344 Fix parameter order in cldr2qlocalexml.py's usage()
Callers and definition were out of sync.

Change-Id: Icda26887cb64c61c7e373766f25559b0d450d112
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-06 14:29:32 +02:00
Alexandru Croitor
9b21dbbd79 CMake: Reformat python scripts
Change-Id: I1dfac318cdbbc4b4b7c76b113edca7db8f52f56b
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-06 11:00:53 +02:00
Alexandru Croitor
d64a7c9f5a CMake: pro2cmake: Allow specifiying custom output file
Change-Id: If984d2bbc3e4b655a5eb58c68b282e3d13d51218
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
2020-04-06 11:00:53 +02:00
Alexandru Croitor
34c7344e2a CMake: configurejson2cmake: Port precompile_header feature
We don't need the test, we can just check the value of BUILD_WITH_PCH
to know whether the feature is enabled.

Regenerate configure.cmake files.

Change-Id: I5691a22af2913bc398f99825e0c41cf2daf5a587
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-06 11:00:53 +02:00
Sona Kurazyan
6a6482cb8d Add Qt Core5Compat to the library mapping
Change-Id: Iad613c75705b09f7ae043cff417b0fcf3f5dd946
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-03 17:45:43 +00:00
Alexandru Croitor
8a0a31ed68 CMake: pro2cmake: Skip conversion of doc snippets
Most of them are hand-written.

Change-Id: Ia3d83cdc9e279420c9b4700993b428e10cf8fb22
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-03 16:05:26 +02:00
Alexandru Croitor
61d9428d8c CMake: Make sure that the library config.test is used for assimp
Change-Id: Ia5b2a2ffca2dda460a323fd3f564c548be84c0aa
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-03 16:05:22 +02:00
Alexandru Croitor
d2931a2626 CMake: Handle standalone config.tests in configure libraries section
Some library entries in configure.json have a test entry.
An example is assimp in qtquick3d.

qmake tries to find the library via the sources section, and then tries
to compile the test found in config.tests/assimp/assimp.pro while
automagically passing it the include and link flags it found for assimp.

We didn't handle that in CMake, and now we kind of do.

configurejson2cmake will now create a corresponding
qt_config_compile_test call where it will pass a list of packages and
libraries to find and link against.

pro2cmake will in turn generate new code for the standalone
config.test project. This code will iterate over packages that need to
be found (like WrapAssimp) and then link against a list of passed-in
targets.

In this way the config.test/assimp/main.cpp file can successfully
use assimp code (due to propagated include headers).

qt_config_compile_test is augmented to take a new PACKAGES argument,
with an example as follows

PACKAGES PACKAGE Foo 6 COMPONENTS Bar
         PACKAGE Baz REQUIRED

The arguments will be parsed and passed to the try_compile project,
to call find_package() on them.

We also need to pass the C/C++ standard values to the try_compile
project, as well as other try_compile specific flags, like the
toolchain, as given by qt_get_platform_try_compile_vars().

Change-Id: I4a3f76c75309c70c78e580b80114b33870b2cf79
Reviewed-by: Leander Beernaert <leander.beernaert@qt.io>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2020-04-03 16:05:19 +02:00
Edward Welbourne
cabd8f860b Ensure we use UTF-8 for the emitted QLocaleXML data file
Python helpfully uses a sensible locale when stdout is a tty but uses
the system (not the filesystem) default encoding, which may be ascii
and unable to encode some of the data we need to save. So brute force
kludge it to ensure emit.encoding is UTF-8 when writing the output
we'll read as UTF-8 anyway.

(This matches dev's commit 0ef79d94f6
for the reworked version of the script.)

Task-number: QTBUG-79902
Change-Id: I60ddc896a308c06e01fa87e8e18e112faa17d601
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:44:06 +01:00
Edward Welbourne
67c0e28789 Purge a stray space from calendar locale data
It was causing all lines after the first, in each calendar's
locale_data[], to be over-indented. This only changes spacing.

Change-Id: Ibfc4986548eecbfdba2902cc18f44a2af669bc6d
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2020-04-02 19:44:01 +01:00
Edward Welbourne
3dfdc9b97a Convert the qlocale2cpp's last few %-formats to modern format() style
I've taken care of all the others in the course of other changes
already ...

Task-number: QTBUG-81344
Change-Id: I44e40a0d1c9f1e1a540a5f4cd252369fdc9b2698
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:43:50 +01:00
Edward Welbourne
963931550d Check all matches for each XPath when searching
Previously, if we found one element with required attributes, we would
search into it and ignore any later elements also with those required
attributes. This meant that, if the first didn't contain the child
elements we were looking for, we'd fail to find what we sought, if it
was in a later matching element (e.g. with some ignored attributes).
We would then go on to look for a match in a later file, where there
might have been a match we should have found in the earlier file.

Check all matches, rather than only the first match in each file.  Do
the search in each file "in parallel" to save reparsing the XPath.
This clears the search code of rather hard-to-follow break/else
handling in loops; and currently makes no change to the generated
data.

Change-Id: I86b010e65b9a1fc1b79e5fdd45a5aeff1ed5d5d5
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:43:43 +01:00
Edward Welbourne
89bd12b9ad Change QLocale to use CLDR's accounting formats for currencies
In particular, this changed the US currency formats for negative
amounts to be parenthesised versions of the positive amount forms,
rather than having a minus sign after the $ sign. Test updated.

[ChangeLog][QtCore][QLocale] Currency formats are now based on CLDR's
accounting formats, where they were previously mostly based (more or
less by accident) on standard formats. In particular, this now means
negative currency formats are specified, where available, where they
(mostly) were not previously.

Task-number: QTBUG-79902
Change-Id: Ie0c07515ece8bd518a74a6956bf97ca85e9894eb
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 20:43:34 +02:00
Edward Welbourne
81cf23c7a7 Take CLDR's distinguished attributes into account
When doing XPATH searches, child nodes that have distinguished
attributes that were not asked for should be skipped. This is part of
the LDML spec and matters when resolving locale inheritance. Scan the
LDML DTD (previously only scanned for the CLDR version) to find which
attributes of which tags are ignorable - all others are distinguished
- and take the result into account when performing XPATH searches.

The XPath we were using for currency formats wasn't excluding
currencyFormatLength elements with type="short" and patterns specific
to thousands (and larger multiples); this is fixed by taking
distinguished attributes into account. However, the XPATH also wasn't
specifying the always distinguished attribute type="standard" that
was, in practice, used for nearly all locales that weren't (wrongly)
using short-forms for thousands; so type="standard" is now made
explicit, so as to minimize the diff.

This leaves only twenty-one locales with a negative currency formats.
A later commit shall switch to using accounting by default (it falls
back via an alias to standard, in any case), thereby restoring the two
mentioned below that were using it by accident, but the present change
gives the minimal diff here.

Thousands-specific formats replaced with sensible ones:
* zh_Hant_{HK,MO} (Traditional Mandarin, Hong Kong and Macau)
* eo_001 (Esperanto)
* fr_CA (Canadian French)
* ha_* (Hausa, when not written in Arabic)
* es_{GT,MX,US} (Spanish - Guatemala, Mexico, USA)
* sw_KE (Swahili, Kenya)
* yi_001 (Yiddish)
* mfe_MU (Morisyen, Mauritius)
* lag_TZ (Langi, Tanzania)
* mgh_MZ (Makhuwa Meetto, Mozambique)
* wae_CH (Walser, Switzerland)
* kkj_CM (Kako, Cameroon)
* lkt_US (Lakota, USA)
* pa_Arab_PK (Punjabi, in Arabic script, as used in Pakistan; uses
  arabext number system, whose currency falls back to latn's, for
  which pa_Arab over-rides the thousands-format).

Format changed from an over-ridden type="accounting" to standard (so
these lost a negative-specific form) in:
* en_SI (English, Slovenia)
* es_DO (Spanish, Dominican Republic; same)

For some locales we were picking up over-rides of narrow or short list
formats, or formats for or-lists or unit-lists rather than and-lists,
in place of the standard list format, that these locales don't
over-ride, provided by a parent locale. This changed list formats for:
* en_CA, en_IN (dropped "Oxford" comma before "and")
* qu_* (Quechua; dropped "utaq", presumably meaning "and")
* ur_IN (Urdu, India; was using unit-list formats)

[ChangeLog][QtCore][QLocale] Data used for currency formats in several
locales and list patterns in some locales have changed due to now
parsing the CLDR data more faithfully.

Fixes: QTBUG-81344
Change-Id: I6b95c6c37db92df167153767c1b103becfb0ac98
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:43:28 +01:00
Edward Welbourne
e5eb0aa428 Take number system into account in currency format look-up
CLDR's currency formats do have number system variation, so take it
into account. (The old xpathlite code clearly intended to do this, but
failed at it due to looking for the wrong component of an XPATH to
fix.) This changes the currency formats in use for
* all Dutch locales (because nl.xml lists a currency format for arab
  before the one for latn, and they differ),
* Punjabi, Urdu - specifically pa_Guru_IN, ur_Arab_PK (both like
  Dutch, arabext before latn; which is correct for pa_Arab_PK and
  ur_Arab_IN),
* Sindi (whose over-ride of latn currency format we were using, where
  we should be using arab's format, supplied by root's default),
* Tatar (which specifies a generic currency format, which we were
  using, before one specific to latn, which we now use),
* Tongan (same as Dutch),
* Konkani (like Dutch, deva before latn) and
* several North African Arabic locales (whose default number system is
  latn, rather than arab, but previously used arab's formats).

Task-number: QTBUG-79902
Change-Id: I18d8ec16bfd3a516d1bcd2f63bc7f7f15179a3f4
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:43:23 +01:00
Edward Welbourne
be3dfd7a71 Rework cldr2qlocalexml.py's reading of CLDR data
Move the code out to a CldrReader class in cldr.py, expand CldrAccess
with facilities that needs, expand ldml.py to include support for more
features, finally making xpathlite.py redundant. This initial commit
aims, though, to be bug-for-bug compatible with xpathlite in its
reading of the CLDR data.

It turns out we've been using draftier data than we were aware of
(which might not be a bad thing). The xpathlite code appeared to check
for draft attributes, but these only appear on leaf nodes and most
data were fetched by finding a parent and then scanning its children
without the draft check; only am/pm data was actually being excluded
based on draft values.  (We allowed contributed, for am/pm, in
addition to approved, which is all the xpathlite code allows
otherwise.) There are also some less equivocal bugs; I'll deal with
these in later commits.

Simplified number-system data look-ups; the old get_number_in_system()
was taking care of old LDML versions' placement of the number system
attribute; this is no longer needed. (It was also being used for a
currency value to which it was not appropriate, which is now handled
separately; this is one of the bugs mentioned above.) Ditched a
fall-back to nativeZeroDigit, which no longer exists in CLDR.

Change the command-line to take the root of the CLDR data tree, rather
than its common/main/ sub-directory. Support naming the file to which
to write output, as a second command-line argument, instead of always
writing to stdout (which remains the default) and leaving whoever runs
the script to redirect stdout.

Support (internally for now, while adding TODOs to give main() more
command-line options) separating the stderr output into its more and
less interesting parts; for now, continue producing both, but suppress
the least interesting entirely.

Task-number: QTBUG-81344
Change-Id: Ie611b47403a9452b51feaeeaaa0fbc8f7e84dc71
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:43:18 +01:00
Edward Welbourne
c834dbc6fb Move cldr2qtimezone.py's CLDR-reading to a CldrAccess class
This begins the process of replacing xpathlite.py, adding low-level
DOM-access classes to ldml.py and the CldrAccess class to cldr.py

Moved a format comment from cldr2qtimezone.py's doc-string to the
method of CldrAccess that does the actual reading.

Task-number: QTBUG-81344
Change-Id: I46ae3f402f8207ced6d30a1de5cedaeef47b2bcf
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:43:13 +01:00
Edward Welbourne
9fab53a513 Rework qlocalexml2cpp.py to use writers based on Transcriber
This saves repetition of temporary-file manipulation code. In the
process, ensure that we tidy away temporary files on failure.

Moved a comment in qlocale.h to *outside* the re-written portion, to
save having to rewrite it every time. Added blank lines to separate
script data from country data in the generated output. Changed 0s in
one comment to zeros, to match another comment.

Isolated use of sys to the __main__ block.
Isolated use of enumdata to the new LocaleHeaderWriter class.
Modernised all the string-formatting I touched.

Task-number: QTBUG-81344
Change-Id: I5768e45d9a8ea23facc303b3dd8af8b3ccbf7ff2
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:42:56 +01:00
Edward Welbourne
5b1c33cc78 Rework cldr2qtimezone.py into more maintainable form
Broke out the updating of a source file to a ZoneIdWriter helper
class, which enables tidying away the temporary file if we fail.
Collected up the rest of the script into a main() that's now
called from a __name__ == '__main__' block.
Rationalized the imports.

Eliminated an inefficient lookup function by constructing a suitable
dict() before entering the loop that needed it.

Separated the "data you might need to update" tables from the code
that does the work, to make it easier for those adding support for new
zones to see what they're doing.

Removed the spurious $Revision$ from the output and reworded the
premable of the generated file. (It would seem CLDR no longer uses an
RCS-based version-control system.) Generated output is otherwise
unchanged.

Task-number: QTBUG-81344
Change-Id: I7d9de8357ebcb599d154de9f862e25f7ade00390
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:42:50 +01:00
Edward Welbourne
bb4242341b Add tools to localetools to facilitate source file recreation
For now unused; later commits shall put them to use.
Transcriber -- base, takes care of tempfile and renaming.
SourceFileEditor -- handles copying parts before and after a common delimiter.

Task-number: QTBUG-81344
Change-Id: I28cf977d0a08825fbb873fb330da6823b88ad3ed
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:42:45 +01:00
Edward Welbourne
c3dea1ffca Move some shared code to a localetools module
The time-zone script was importing two functions from the locale data
generation script. Move them to a separate module, to which I'll
shortly add some more shared utilities. Cleaned up some imports in the
process.

Combined qlocalexml2cpp's and xpathlit's error classes into a new
Error class in the new module and made it a bit more like a proper
python error class.

Task-number: QTBUG-81344
Change-Id: Idbe0139ba9aaa2f823b8f7216dee1d2539c18b75
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:42:40 +01:00
Edward Welbourne
4d9f1a87de Move qlocalexml2cpp.py's XML-reading to QLocaleXmlReader
This new class mirrors the existing QLocaleXmlWriter and places the
two side-by-side in qlocalexml.py, rather than having the writing and
reading in separate places.

Made judicious use of transformed versions of mappings to save
repeated iteration of a mapping's entries to do lookups on fist
entries of pair-values; several (id, name, code) data-sets are
sometimes indexed by id, sometimes by name.

Reworked the default_map, that the complicated compareLocaleKeys()
used in sorting locale keys, to map IDs instead of names; the function
also needed the locale_map so that it could convert IDs to names,
which we can skip by going directly with IDs.

Task-number: QTBUG-81344
Change-Id: Iff6a97f7f0755b56dda70d8a6796ec074c558910
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2020-04-02 19:42:34 +01:00