Commit Graph

1089 Commits

Author SHA1 Message Date
Thiago Macieira
b2298b7e94 qsimd_p.h: let the generator script generate the ARCH target strings
__attribute__((target("arch=xxxx"))) does not work because the compilers
(GCC at least) don't test the CPU features that they are targeting, so
we keep getting "inline failed" compiler errors.

GCC bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90129

Upstream patch: https://github.com/opendcdiag/opendcdiag/pull/59

Change-Id: I6fcda969a9e9427198bffffd16cea09fda4406d2
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Allan Sandfeld Jensen <allan.jensen@qt.io>
2022-02-19 13:40:21 -08:00
Thiago Macieira
b852584556 qsimd: update the generator script from OpenDCDiag
I'd been making changes to that and improving it for the past 2 years
without bringing it back into Qt.

The list of features is mostly the same, except:
- removed TSX features
- removed features specific to Xeon Phi processors
- added CET and AVX512FP16 features
- added the bit for hybrid CPU detection

See matching update at https://github.com/opendcdiag/opendcdiag/pull/49

Change-Id: I6fcda969a9e9427198bffffd16ce860b5a38aece
Reviewed-by: Allan Sandfeld Jensen <allan.jensen@qt.io>
2022-02-18 15:46:41 -08:00
Dimitrios Apostolou
c37d0e84bc Clean up QNX platform configs and improve testrunner scripts
QNX requires its own TESTRUNNER script in order to SSH to the QEMU guest
and run the tests. Simplify it a lot and fix the way it passes arguments
so that they retain spaces and quotation.

Also wrap it with the generic TESTRUNNER script for CI, which is
qt-testrunner.py, so that tests can re-run in case of flakiness.

Delete prefix.sh as the environment variables are now properly set in
the Coin platform config files.

Avoid executing extra chmod commands to make scripts executable,
but use the right Coin instruction for that. For reference, we
use 493 for file mode which equals to 755 in octal, as Coin expects
the file mode in decimal.

Change-Id: Ife4d1caef606f48b92ba1da1cfb14ec0dea11ef2
Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
2022-02-14 12:56:06 +01:00
Jani Heikkinen
994fd04b2b Bump version to 6.4.0
Change-Id: Ie0e2133d6c9125b901364c979c60b6efd585f026
Reviewed-by: Qt Submodule Update Bot <qt_submodule_update_bot@qt-project.org>
2022-01-31 17:20:06 +02:00
Dimitrios Apostolou
b013f94086 Write XML test log files even for the flaky test re-runs
Previously the qt-testrunner script was avoiding to write XML log files
for the flaky test re-runs, since parsing them always provides limited
information about what went wrong (for example unparseable files) and
might miss issues completely (for example a crashed test might not even
write a log file), or even mislead if multiple different data points
about a single testcase end up into the database..

But currently Coin is the system that uploads test information to the
testresults database and expects to find such log files in an attempt to
send as much information as possible regarding failures.

Re-enabling these log files will allow Coin to deduce as much
information as possible about the test runs.

This is a temporary step at least, until the test-uploading logic is
separated from Coin. Then it will be easier to find the full information
about what happened with flaky or crashed tests by special logs from
qt-testrunner.py, which is the test re-running orchestrator and
therefore has more knowledge about what went wrong.

Fixes: QTQAINFRA-4754
Change-Id: I69e4469cbe6c0a6ca5b2473eaf8d034632564342
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Toni Saario <toni.saario@qt.io>
Reviewed-by: Simo Fält <simo.falt@qt.io>
2022-01-30 01:00:35 +01:00
Dimitrios Apostolou
b4d9d5c89c qt-testrunner: do not try to re-run initTestCase and others
Some function names are special for qtestlib, in the sense that they can
not be specified as a command line argument to run individually.

In such cases qt-testrunner treats the failure specially and tries once
to re-run the full test executable.

Fixes: QTBUG-89011
Change-Id: I0cc25f91c57374e5ac65ade10e2e223fe969f211
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-30 01:00:35 +01:00
Joerg Bornemann
eced23b6a8 Don't set CMAKE_AUTORCC in pro2cmake.py
To tool generates calls to qt6_add_resources instead, and we don't want
to promote the usage of CMAKE_AUTORCC.

Task-number: QTBUG-87643
Change-Id: I58458416514949a6ffc9b93c7eb6d24bc2ed01ca
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2022-01-21 19:54:13 +01:00
Joerg Bornemann
dc2c590b1a pro2cmake: Fix exception with newer pyparsing module
The pyparsing module's debug action properties have been renamed.
Check the existence of the attributes before assigning.

Change-Id: I8fff652304a0af215c56f54b63d613a1f6a5207a
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2022-01-21 19:54:13 +01:00
Dimitrios Apostolou
5180e70dbe qt-testrunner: be more clear that it was the test that crashed
Task-number: QTBUG-99970
Change-Id: Id2d01b6ab7d428356d9dfc953107014791393d35
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
2022-01-18 19:43:54 +01:00
Edward Welbourne
755b0aa681 Turn a condition into a lambda to save needless evaluation
The public suffix list scanner's utf8encode()'s main loop always
worked out whether a character is a hex digit, even when it didn't
need to know. Package the computation in a lambda and only test it
when it is needed. Also assert non-empty input generates non-empty
output.

Change-Id: Iaf48aad382624e421cea9c9cdb8bba5fc47b1596
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2022-01-18 16:23:56 +01:00
Dimitrios Apostolou
3acfd39653 qt-testrunner: disable JUnit XML test logs
It causes crashes because of interleaved execution of Quick tests.

Task-number: QTBUG-98350
Change-Id: I61e26f5da5617e06810db8eb7991f17eecdd3acb
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-01-12 16:14:18 +01:00
Dimitrios Apostolou
98f1034f96 Properly wait for the Android emulator to fully boot
It had been noticed that sometimes the Android emulator in our CI took
several minutes to fully boot. It turns out that this behavior can be
reproduced locally by removing the image files under $HOME/.android
especially the "userdata-qemu.img.qcow2" file.

Then the emulator goes through several reboots until fully booted with a
full set of packages installed. We discovered that the property that
signifies it is finished is dev.bootcomplete=1. So we now check for this
flag and remove the other heuristics we had.

We also disable the debug output to avoid hundreds lines of
logs. Instead we selectively print the values that the emulator returns,
every second until full boot is detected.

We increase the Coin timeout for the shell script, from 5min to 10min,
since it has been measured that it takes about 2.5min on a good day, and
the script itself retries several times to restart the emulator in case
of failure.

Finally we adjust the coding style a bit to be more consistent.

Pick-to: 6.3 6.2
Fixes: QTQAINFRA-4681
Change-Id: I77062dceb91477e957696c89bfacb4ebabc34c1f
Reviewed-by: Toni Saario <toni.saario@qt.io>
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
2022-01-07 19:34:50 +01:00
Dimitrios Apostolou
c8e0a742fb Wait for boot to be fully complete with all packages installed
It happens that the android emulator supposedly "finishes" booting and
getprop shows bootanim=stopped and boot_completed=1. But sometimes not
all packages have been installed (`pm list packages` shows only 16
packages installed).

After around half a minute the boot animation starts spinning
(bootanim=running) again despite boot_completed=1 all the time. After
some minutes the boot animation stops again and the list of packages
contains 80 packages, among which is also the "development" package.
Only then is the device ready for `adb install` of custom packages.

This should be the last time we see the message:

Error: Could not access the Package Manager.  Is the system running?

Just for completeness, we also properly disown the emulator process and
its file descriptors, since the parent shell dies but the process stays.

Pick-to: 6.3 6.2
Fixes: QTQAINFRA-4681
Change-Id: I2b9d4bdc3dac0f10d09d4f09c83b4028ebc31f48
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-06 17:14:58 +01:00
Dimitrios Apostolou
8bf1beb9bd Do not hide android emulator output
It might print useful messages, even though they might come mixed with
other test messages.

However we can't leave stdout/stderr as is, because Coin agent hangs at
the end of the shell script, waiting for them to close. So we just
redirect them to a file.

Pick-to: 6.3 6.2
Change-Id: I797af78786b7df31131b3c3261e1c1fc00e5d460
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-06 14:34:15 +01:00
Dimitrios Apostolou
0aaf514042 Wait for android emulator to be up and running properly
Pick-to: 6.3 6.2
Fixes: QTQAINFRA-4681
Change-Id: Icfa75b6982964970172726379e5a2a2bb640f8bf
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-06 14:34:09 +01:00
Dimitrios Apostolou
24572fccae Enable verbose logging in the last re-run of a failing test
Fixes: COIN-728
Change-Id: I08802e377e26e5dd7f7d1c44b5efe4280b09f957
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-03 17:40:35 +01:00
Dimitrios Apostolou
63ffae3fa3 Introduce qt-testrunner.py
Script that wraps Qt test execution in order to iron-out flakiness.

Task-number: QTBUG-96353
Change-Id: Ie8ee780e3f4f8d086c080a7784f9f68fd1867be2
Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
2022-01-03 16:32:56 +01:00
Ievgenii Meshcheriakov
4f53c703e4 QLocale: Extend support for language codes
This commit extends functionality for QLocale::codeToLanguage()
and QLocale::languageToCode() by adding an additional argument
that allows selection of the ISO 639 code-set to consider for
those operations.

The following ISO 639 codes are supported:
    * Part 1
    * Part 2 bibliographic
    * Part 2 terminological
    * Part 3

As a result of this change the codeToLanguage() overload without
the additional argument now returns a Language value if it matches
any know code. Previously a valid language was returned only if
the function argument matched the first code defined for that
language from the above list.

[ChangeLog][QtCore][QLocale] Added overloads for codeToLanguage()
and languageToCode() that support specifying which ISO 639 codes
to consider.

Fixes: QTBUG-98129
Change-Id: I4da8a89e2e68a673cf63a621359cded609873fa2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-12-09 03:45:08 +01:00
Morten Johan Sørvig
d3646d2b05 Say hello to qtwasmserver
Development web server for web applications. Supports
http and https. Sets COOP and COEP headers.

The web server script supports certificate generation using
mkcert (github.com/FiloSottile/mkcert). Briefly, mkcert
supports generating server certificates for the current
local ip address(es), using a certificate authority
which can be installed on devices where the app should
run.

The COOP and COEP headers are required to enable the
SharedArrayBuffer API, which is required for Emscripten’s
pthreads implementation.

The server serves the current directory on localhost
by default. Use the “-a” argument to bind to an additional
address, or "--all” to bind to all IPv4 addresses found on
local network interfaces. Change the directory by passing
it as a positional argument.

Task-number: QTBUG-79087
Change-Id: Id0cba649e42af53ed8106e336a77e78398bcf901
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: David Skoland <david.skoland@qt.io>
2021-12-06 16:36:09 +01:00
Ievgenii Meshcheriakov
aee0021fc8 QLocale: Add support for Kaingang and Nheengatu languages
Update the locale generation script to support Kaingang and
Nheengatu languages. These are new in CLDR v40. Regenerate
the locale data.

Task-number: QTBUG-94358
Change-Id: I5195d5161d8c4d9f17129bbcfde39dfd3fcf1cd5
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-11-10 00:36:12 +01:00
Ievgenii Meshcheriakov
2c505eea01 locale_database: Add entry for 'South Sudan Standard Time'
This timezone is new in CLDR 40, Olson database calls it Africa/Juba.
The offset is UTC+2.

Reference: https://techcommunity.microsoft.com/t5/daylight-saving-time-time-zone/2021-time-zone-updates-for-republic-of-south-sudan-now-available/ba-p/2234981

Task-number: QTBUG-94358
Pick-to: 6.2
Change-Id: Ib70dbd9b472eb9cf8cb62a0eb5e241199148c077
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-11-10 00:36:11 +01:00
Ievgenii Meshcheriakov
7d7663cfc4 Use a HTTPS URL for the CLDR download link
FTP is insecure and is not supported by modern browsers anymore.

See also: https://mywiki.wooledge.org/FtpMustDie

Change-Id: Iad65d29912e79a4f3fadb9317bb5d9c5fe9b68d7
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-11-05 19:32:23 +01:00
Joerg Bornemann
6ab16d52d2 configure: Remove unused facility to define variable assignments
The function qt_commandline_assignment and everything related to it is
removed from configure.  It was only used in qtbase, and all usage has
been removed.

More general variable assignments will be added in a subsequent commit.

Task-number: QTBUG-88210
Change-Id: I7cfa782e89914f2b0dc0277c46e425c8a825557e
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2021-11-01 22:32:01 +01:00
Assam Boudjelthia
a81f24ada5 Rename deprecated ANDROID_SDK_HOME to ANDROID_SDK_ROOT
Pick-to: 6.2
Task-number: QTBUG-97002
Change-Id: I777491f542e5388e04ed4c1b7ff5d9db75f9e778
Reviewed-by: Heikki Halmet <heikki.halmet@qt.io>
2021-10-22 17:21:10 +02:00
Ievgenii Meshcheriakov
826fc8c9bd Update UCD to Revision 28
This corresponds to Unicode version 14.0.0.

Added the following scripts:

    * CyproMinoan
    * OldUyghur
    * Tangsa
    * Toto
    * Vithkuqi

Full support of these scripts requires harfbuzz version 3.0.0,
this version adds support for Unicode 14.0:

    https://github.com/harfbuzz/harfbuzz/releases/tag/3.0.0

With this release 10 test cases in tst_qurluts46 were fixed, one
additional test case is failing in tst_qtextboundaryfinder and
is commented out. In total 62 line break test cases and 44 word
break test cases are failing.

A comment in src/corelib/text/qt_attribution.json was updated to
include the URL of the page containing UCD version number.

Fixes: QTBUG-94359
Change-Id: Iefc9ff13f3df279f91cbdb1246d56f75b20ecb35
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-10-18 16:45:10 +00:00
Joerg Bornemann
f19668ce65 Don't use libs in /usr/local for configure tests by default on macOS
Since commit f3c7d22dd0 we do not use
libraries from /usr/local and other non-system locations on macOS.  But
our configure tests still did.  This led to discrepancies between
find_package calls in configure tests and the Qt project itself.

Mentioned commit removed /usr/local and friends from
CMAKE_SYSTEM_PREFIX_PATH.  But we can't pass this variable to the
configure tests, because CMake sets it up and overwrites our value.

Pass CMAKE_SYSTEM_PREFIX_PATH and CMAKE_SYSTEM_FRAMEWORK_PATH as
QT_CONFIGURE_TEST_CMAKE_SYSTEM_{PREFIX|FRAMEWORK}_PATH variables to
tests.

Tests with separate project files that call find_package() must add code
like this after the project() command:

if(DEFINED QT_CONFIGURE_TEST_CMAKE_SYSTEM_PREFIX_PATH)
    set(CMAKE_SYSTEM_PREFIX_PATH
        "${QT_CONFIGURE_TEST_CMAKE_SYSTEM_PREFIX_PATH}")
endif()
if(DEFINED QT_CONFIGURE_TEST_CMAKE_SYSTEM_FRAMEWORK_PATH)
    set(CMAKE_SYSTEM_FRAMEWORK_PATH
        "${QT_CONFIGURE_TEST_CMAKE_SYSTEM_FRAMEWORK_PATH}")
endif()

Adjust pro2cmake accordingly.

Task-number: QTBUG-97076
Change-Id: Iac1622768d1200e6ea63be569eef12b7eada6c76
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2021-10-08 23:15:31 +02:00
Ievgenii Meshcheriakov
906cb8d4ef Remove util/harfbuzz/update-harfbuzz
This is not the correct script to update harfbuzz. The correct one is
src/3rdparty/harfbuzz-ng/import_from_tarball.sh

Change-Id: Ic9ed43cae591cd6957effb10f557166c43498ad9
Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
2021-10-05 21:53:10 +02:00
Ievgenii Meshcheriakov
394f8a9c4b Unicode: Add script to facilitate UCD update
Add a script that downloads UCD data for a given Unicode version,
unpacks it, and copies the used files to appropriate locations inside
the Qt source code.

Also update the README and use an HTTPS link for the UCD data file.
FTP links are no longer supported by some browsers.

Task-number: QTBUG-94359
Change-Id: I2aa70a588f675e411fa6b3ce5b4444a7c07ed707
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-10-05 20:38:02 +02:00
Alexandru Croitor
9665929706 pro2cmake: Generate public CMake API code for projects by default
Previously pro2cmake assumed that all .pro were internal Qt projects
like qtbase/src/corelib.pro and generated private api calls like
qt_internal_add_module().

Public CMake API calls (like qt_add_executable) were only generated if
an --is-example command line flag was passed to the script or if the
.pro file was located under the examples subfolder of a qt repo source
directory (indicated by the presence of a .qmake.conf file).

Change the logic to always generate public CMake API code by default,
unless a .cmake.conf/.qmake.conf file is encountered with a valid repo
module version inside, in which case apply the old heuristic of
checking for the ./examples subfolder.

The intention is to have a sensible default so that Qt users can use
the script to more easily migrate their qmake projects without having
to explicitly specify the --is-example flag.

Pick-to: 6.2
Task-number: QTBUG-96799
Change-Id: I38f4f1b349a5b9688cf7bc5914d4fde72e660a98
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2021-09-27 21:36:36 +02:00
Alexandru Croitor
124f7c681f pro2cmake: Augment searching of .qmake.conf with .cmake.conf
Adapt to fb656c036d which removes
.qmake.conf files in repos by searching for either .qmake.conf or
.cmake.conf.

The locations of those files are only used to compute relative paths
based on the directory where the conf file is.

For now, remove the warning if a conf file is not found, because we
intend to change the script to generate user project code by default,
instead of Qt internal project code and the warning looks scary.

Pick-to: 6.2
Task-number: QTBUG-96799
Change-Id: I301e93d2ce5c05a52877eb6ed9eb9cee54fc5b15
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2021-09-27 21:36:33 +02:00
Joerg Bornemann
a0e56294c1 Skip unnecessary commands when cross-building tools
Introduce a new macro qt_internal_return_unless_building_tools which
simply calls return() if tools are not built.  This macro is supposed to
be called after qt_internal_add_tool().

Using this macro avoids having to special-case code for when
qt_internal_add_tool() creates imported targets in cross-builds.

Adjust pro2cmake accordingly.

Task-number: QTBUG-85084
Change-Id: I9e1c455c29535dd8c318efa890ebd739c42effc1
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2021-09-27 16:34:23 +02:00
Alexandru Croitor
0da123d67b CMake: Bump almost all cmake_minimum_required calls to 3.16
Needed for subsequent change that will check and error out if the
version is lower than 3.16. We do that to ensure all policies
introduced by CMake up to version 3.16 have their behavior set to
NEW.

Pick-to: 6.2
Task-number: QTBUG-95018
Change-Id: Ieaf82c10987dd797d86a3fd4a986a67e72de486a
Reviewed-by: Kai Koehne <kai.koehne@qt.io>
2021-09-22 19:36:49 +02:00
Edward Welbourne
7a4b586f4b Remove conditioning on Android embedded
It is no longer handled separately from Android.
This effectively reverts commit 6d50f746fe

Change-Id: Ic2d75b8c5a09895810913311ab2fe3355d4d2983
Reviewed-by: Assam Boudjelthia <assam.boudjelthia@qt.io>
2021-09-17 17:30:14 +02:00
Ievgenii Meshcheriakov
8be862687d unicode: Fix typo s/supersting/superstring/
Thanks to Konstantin Ritt for spotting it.

Change-Id: Ia3b5c4103b315cdb690fcd8b42239f000acdbef0
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-09-15 21:58:12 +02:00
Edward Welbourne
ce868ffb7c Remove trailing zero rows from qtimezoneprivate_data_p.h's tables
They are not needed. Iterations over the table track their sizes.
The size-of-table constants just needed their -1s removed.
Incidentally use std::size() rather than sizeof(array)/sizeof(element).

Change-Id: Ie20eef9f6f5786d93c10b830a87e006d3c5bcc1a
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-09-10 19:33:06 +02:00
Alexandru Croitor
8a66e3cfe5 pro2cmake: Fix reported make flake8 issues
pro2cmake.py:458:5: F841 local variable 'is_parent_path' is assigned
to but never used

Pick-to: 6.2
Change-Id: I1c61efe5cf568fa0c99b07eed28b93d3444a5fac
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2021-09-09 10:21:27 +02:00
Alexandru Croitor
21a281d962 pro2cmake: run make format to fix styling issues
Pick-to: 6.2
Change-Id: I718e34d628576bb709096c0672da3a69b71ae8df
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2021-09-09 10:21:27 +02:00
Oliver Eftevaag
8337493301 pro2cmake.py: generate examples with properties in qt_add_executable()
This patch will slightly change the output of CMakeLists.txt files that
are generated for examples.

* set_target_properties() will no longer be added to the top-level
  scope. If the WIN32 and MACOSX_BUNDLE properties should be added,
  they will instead be added to qt_add_executable().

* The version in cmake_minimum_required() will now be 3.16, rather than
  3.14.

Pick-to: 6.2
Change-Id: I79e1865dace5538d2b7ff264da02f9e28a655ae9
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2021-09-08 15:43:16 +02:00
Ievgenii Meshcheriakov
5f3f073126 unicode: Build IDNA map superstring greedily
This slightly reduces memory required for mapping tables:

    uncompressed size: 1146 characters
    consolidated size: 904 characters
    memory usage: 47856 bytes

Task-number: QTBUG-85323
Change-Id: Ic960789e433e80acf1a4e36791533a1c55a735c8
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-09-03 14:43:16 +02:00
Ievgenii Meshcheriakov
2d78b71fc6 unicode: Pack 2 QChar's into one entry for IDNA mapping
Store up to 2 QChar's for mapping values inside the mapping
table itself. This reduces the size of the superstring for
other mapping values.

results:
    uncompressed size: 1146 characters
    consolidated size: 1001 characters
    memory usage: 48050 bytes

Task-number: QTBUG-85323
Change-Id: I922a6d2037551d0532ddae1a032ec1a9890f40a7
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-09-03 14:43:16 +02:00
Ievgenii Meshcheriakov
ca1eeb23fa unicode: More compact IDNA mapping tables implementation
This implementation stores mapping that are 1 QChar long
inside the mapping tables, the longer mappings are stored as
an offset and length pairs pointing into the common superstring
of all such mapping values.

Size comparison with the existing implementation follows.

old:
    max mapping length: 6
    memory usage: 103608 bytes

new:
    uncompressed size: 3250 characters
    consolidated size: 2367 characters
    memory usage: 50782 bytes

Task-number: QTBUG-85323
Change-Id: I9f2e32438dd463457e0fcd783136bb17145e27a8
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-09-03 14:43:16 +02:00
Ievgenii Meshcheriakov
4e460aa3f7 tests: Add test for UTS #46 implementation using Unicode data
The new test is called tst_qurluts46. It verifies QUrl::{to,from}Ace()
functionality using the data from IdnaTestV2.txt supplied by Unicode.

The file was downloaded from
https://www.unicode.org/Public/idna/13.0.0/IdnaTestV2.txt

Task-Id: QTBUG-85371
Change-Id: I4c6a4942ef6018dafc90cb84ef73f6b2614566d7
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-09-01 14:15:45 +02:00
Ievgenii Meshcheriakov
2afe1a3c19 unicode: Generate tables for IDNA/UTS #46
Update the Unicode data processing tool to generate properties
and mapping tables needed to implement UTS #46
(https://unicode.org/reports/tr46/). The implementation extends
the standard to allow usage of underscores in URLs. This is done
for compatibility with DNS-SD and SMB protocols.

The data file needed to generate the new properties was taken from
https://www.unicode.org/Public/idna/13.0.0/IdnaMappingTable.txt

Task-number: QTBUG-85323
Change-Id: I2c303bf8a08aefb18a7491fb9b55385563bfa219
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-08-26 16:55:05 +02:00
Alexandru Croitor
c13b98d9bc CMake: Bump project versions
Fixes: QTBUG-95454
Change-Id: I2467d3ae27b54424e59a7a4ab00d364eaec517d5
Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
2021-08-02 12:42:29 +02:00
Ievgenii Meshcheriakov
e9519d207e locale_database: Use context manager interface to update source files
Use context manager interface (with statement) to atomically update source
files. This ensures that all files are properly closed and the temporary
file is removed even in case of errors.

Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I18cd96f1d03e467dea6212f6576a41e65f414ce1
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2021-07-20 16:51:51 +02:00
Ievgenii Meshcheriakov
8f06d3f938 locale_database: Use NamedTemporaryFile for temporary files
Using NamedTemporaryFile instead mkstemp + fdopen simplifies the code.
It also makes it easier to switch to using context managers for handling
source file modification.

Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Ibeae840ac6dde3d0b49cd7f985cfa6cd775b7f47
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-07-19 22:05:54 +02:00
Ievgenii Meshcheriakov
2d0c2c9f1c locale_database: Use pathlib to manipulate paths in Python code
pathlib's API is more modern and easier to use than os.path. It
also allows to distinguish between paths and other strings in type
annotations.

Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: Ie6d9b4e35596f7f6befa4c9635f4a65ea3b20025
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-07-19 22:05:54 +02:00
Ievgenii Meshcheriakov
5ef5dce53b locale_database: Use argparse module to parse command line arguments
arparse is the standard way to parse command line arguments in Python.
It provides help and usage information for free and is easier to extend
than a custom argument parser.

Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I1e4c9cd914449e083d01932bc871ef10d26f0bc2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-07-16 19:04:20 +02:00
Ievgenii Meshcheriakov
41458fafa0 locale_database: Use f-strings in Python code
Replace most uses of str.format() and string arithmetic by f-strings.
This results in more compact code and the code is easier to read
when using an appropriate editor.

Task-number: QTBUG-83488
Pick-to: 6.2
Change-Id: I3409f745b5d0324985cbd5690f5eda8d09b869ca
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-07-16 19:04:20 +02:00
Ievgenii Meshcheriakov
65a0e04072 locale_database: Add schema for intermediate locale data files
The schema is in RelaxNG Compact syntax. It can be used to validate
files produced by the cldr2qlocalexml.py script and also gives an
overview of the file format.

Change-Id: I344978f2201c5e67e236ab580a12ad33262f33cb
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2021-07-16 18:27:28 +02:00