Move pre/and post condition handling out of the main loop
to make that one as fast as possible.
Remove special handling of a corner case when the input length
is zero, where the utf8 decoder did something else than all
other decoders.
Change-Id: I94992767ea15405b38f7953adadaa6ff98b20b6f
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
IgnoreHeader was a rather badly defined enum, in addition the
utf8 and utf16 codecs where handling BOMs somewhat different
for stateless decoding.
Fix this by introducing explicit flags for writing a bom when
encoding and not skipping the initial bom when decoding.
Source compatibility for QTextCodec is done with a couple of
static constexpr variables.
Change-Id: I0b2d94f84c937cec1e0494c16ef448c00382691d
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The new QStringEncoder and QStringDecoder classes
(with a common QStringConverter base class) are
there to replace QTextCodec in Qt 6.
It currently uses a trivial wrapper around the utf
encoding functionality.
Added some autotests, mostly copied from the text codec
tests.
Change-Id: Ib6eeee55fba918b9424be244cbda9dfd5096f7eb
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The fromUtf16(ushort*) and fromUcs4(uint*) overloads are going
to be deprecated. Use the newer fromUtf16(char16_t*) and
fromUcs4(char32_t*) overloads.
As a drive-by, use std::end()/std::size() where applicable.
Change-Id: I5a93e38cae4a2e33d49c90d06c5f14f7cb7ce90c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
And remove a test failure when compiling with asan enabled.
Change-Id: I2b8e676665572adcbbac6a910983d5b209bf6d23
Reviewed-by: Alex Blasche <alexander.blasche@qt.io>
Now that all our supported compilers know char16_t, we no longer need
QStringViewLiteral, whose only purpose in life was to turn u"" into
L"" for MSVC < 2015.
Change-Id: I25a094fe7992d9d5dbeb4a524d9e99e043dcb8ce
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
We were handling this properly, but not testing them. I guess we weren't
testing because the condition is a valid intermediate state, so
hasFailure() is correct it returning false.
Testing inspired by the bug reported in
https://github.com/intel/tinycbor/issues/137
Change-Id: Ib47c56818178458a88b4fffd1554ecfdd0af637e
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
The helper application is only needed for Unix platforms, so there is no
need to build it for other configurations.
Change-Id: I4ebb896c66d3ded016f72fdbe631ec2f1276db22
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@qt.io>
Remaining uses of Q_DECL_OVERRIDE are in:
src/corelib/global/qcompilerdetection.h
src/corelib/global/qglobal.cpp
doc/global/qt-cpp-defines.qdocconf
(definition and documentation of Q_DECL_OVERRIDE)
tests/manual/qcursor/qcursorhighdpi/main.cpp
(a test executable compilable both under Qt4 and Qt5)
Change-Id: Ib9b05d829add69e98a86238274b6a1fcb19b49ba
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Ville Voutilainen <ville.voutilainen@qt.io>
Reviewed-by: Olivier Goffart (Woboq GmbH) <ogoffart@woboq.com>
C0 to DF take one continuation byte; E0 to EF take two.
It's invalid UTF-8 anyway, but at least this is what the test row meant:
overlong sequence with 3 bytes of what should have been two.
This updates the comment to match the character that we were actually
testing.
Change-Id: I85a8bd6da2c44f52b4e3fffd14a75df2600487aa
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Replace all QT_NO_PROCESS with QT_CONFIG(process), define it in
qconfig-bootstrapped.h, add QT_REQUIRE_CONFIG(process) to the qprocess
headers, exclude the sources from compilation when switched off, guard
header inclusions in places where compilation without QProcess seems
supported, drop some unused includes, and fix some tests that were
apparently designed to work with QT_NO_PROCESS but failed to.
Change-Id: Ieceea2504dea6fdf43b81c7c6b65c547b01b9714
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@qt.io>
The population of data rows was factored into a separate file, qutf8data.cpp, in
commit e20c4730. Merge commit 9bd03235 failed to track a conflicting change into
the new file, and brought the code back into the tst_utf8.cpp, where it has been
duplicating the utf8data data ever since.
Change-Id: I4282685b882448f927289468bd7ab340a21ea0b3
Reviewed-by: David Faure <david.faure@kdab.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
- port Q_FOREACH to C++11 range-for
- port uses of inefficient QLists to QVector
Fixes errors pointed out by my tree's static checks.
Change-Id: Ica50f44d862f635df06cb8f09ce506b9d30fdfc5
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
There's no sharing, and the use of QSharedPointer(T*)
triggers my tree's static analyzer.
Easiest fix is to port to QScopedPointer, which is the
correct smart pointer to begin with.
Change-Id: I105c1a334c3d6712a475600c8394b0bebc420677
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
I wrote a script to help find the files, but I reviewed the
contributions manually to be sure I wasn't claiming copyright for search
& replace, adding Q_DECL_NOTHROW or adding "We mean it" headers.
Change-Id: I7a9e11d7b64a4cc78e24ffff142b506368fc8842
Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
From Qt 5.7 -> tools & applications are lisenced under GPL v3 with some
exceptions, see
http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/
Updated license headers to use new GPL-EXCEPT header instead of LGPL21 one
(in those files which will be under GPL 3 with exceptions)
Change-Id: I42a473ddc97101492a60b9287d90979d9eb35ae1
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
When the byte sequence for a BOM occurs in the middle of a utf8 stream,
it is a ZWNBSP.
When a ZWNBSP occurs in the middle of a utf8 character sequence, and the
SIMD conversion does some work (meaning: the length is at least 16
characters long), it would not recognize the fact some charactes were
already decoded. So the conversion would then strip the ZWNBSP out,
thinking it's a BOM.
The non-SIMD conversion did not have this problem: the very first
character conversion would already set the headerdone flag.
Change-Id: I39aacf607e2e068107106254021a8042d164f628
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Use character literals where applicable.
Change-Id: I1a026c320079ee5ca6f70be835d5a541deee2dd1
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
The keyword no longer has a meaning for the new CI.
Change-Id: Ibcea4c7a82fb7f982cf4569fdff19f82066543d1
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
- Replace Q[TRY]_VERIFY(pointer == 0) by Q[TRY]_VERIFY(!pointer).
- Replace Q[TRY]_VERIFY(smartPointer == 0) by
Q[TRY]_VERIFY(smartPointer.isNull()).
- Replace Q[TRY]_VERIFY(a == b) by Q[TRY]_COMPARE(a, b) and
add casts where necessary. The values will then be logged
should a test fail.
Tests from corelib/tools were omitted in this change.
Change-Id: I4c8786d33fcf429d11b2b624c7cd89c28cadb518
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
ICU doesn't support iso8859-16, so we need to fall back to
the Qt codec for this encoding.
Task-number: QTBUG-45053
Change-Id: I9754cf098c906fe8a75363a3d090029543cd0e35
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Properly QSKIP tests that use disabled QProcess and symlink
features instead of excluding them silently by #ifdef.
Other reason is that moc doesn't respect QT_NO_* defines
in class definition which causes build issues on some
platforms.
Change-Id: I041020f7452f7d36c7ec8a5866a4ba5eb23d1f94
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@theqtcompany.com>
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Qt copyrights are now in The Qt Company, so we could update the source
code headers accordingly. In the same go we should also fix the links to
point to qt.io.
Outdated header.LGPL removed (use header.LGPL21 instead)
Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing
combination. New header.LGPL-COMM taken in the use file which were
using old header.LGPL3 (src/plugins/platforms/android/extract.cpp)
Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license
combination
Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe
Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
ICU would return a utf-16 (endian dependent) codec for unicode
which is very rarely what people want. In most cases, unicode is
encoded in utf8 these days, so return a utf8 codec for it.
Task-number: QTBUG-41998
Change-Id: I51ee758d520702b263a8b2011787eb1f3455ed96
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Replace "Apple Roman" by "Macintosh" which is registered by IANA:
http://tools.ietf.org/html/rfc1345.
Replace unsupported "GB18030-0" by "GB18030".
Remove "JIS X 0201" and "JIS X 0208" as they are supported as parts of
other Japanese encodings but not directly.
Add "HP-ROMAN8" which is supported by both Qt and ICU.
Also clean the codecs test.
Change-Id: Iaf8e8ff1900d3f92ea0e0df75c60fe1534de23ac
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Substitute this encoding by "macintosh" when Qt is built with ICU.
It is for compatibility with Qt 4.x.
Change-Id: I70c51cba7d473ac81e25862736cb71a2f6894055
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
When a UTF-8 sequences is too short, QUtf8Functions::fromUtf8 returns
EndOfString. If the decoder is stateful, we must save the state and then
restart it when more data is supplied.
The new stateful decoder (8dd47e34b9)
mishandled the Error case by advancing the src pointer by a negative
number, thus causing a buffer overflow (the issue of the task).
And it also did not handle the len == 0 case properly, though neither
did the older decoder.
Task-number: QTBUG-38939
Change-Id: Ie03d7c55a04e51ee838ccdb3a01e5b989d8e67aa
Reviewed-by: Kai Koehne <kai.koehne@digia.com>
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
The SSE2-based conversion only kicks in with strings with 8 characters
or more in length.
Change-Id: Ic1daad0845571be03547553cc001d83550f0c89c
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Commit 125bb81bef wasn't enough. Let's
just make it simpler and use a regular function.
Change-Id: I9627dedaa87873b4b138224afd182e4fd9a55265
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
That commit made QString::toXxx (8-bit) functions use C++11 ref
qualifiers, so we need to match it here.
Change-Id: I45b50464d36f858d012b12e0cb511aae347ddb6f
Reviewed-by: Marc Mutz <marc.mutz@kdab.com>
Reviewed-by: Giuseppe D'Angelo <giuseppe.dangelo@kdab.com>
Like before, this is taken from the existing QUrl code and is optimized for
ASCII handling (for the same reasons). And like previously, make
QString::fromUtf8 use a stateless version of the codec, which is faster.
There's a small change in behavior in the decoding: we insert a U+FFFD for
each byte that cannot be decoded properly. Previously, it would "eat" all bad
high-bit bytes and replace them all with one single U+FFFD. Either behavior is
allowed by the UTF-8 specifications, even though this new behavior will cause
misalignment in the Bradley Kuhn sample UTF-8 text.
Change-Id: Ib1b1f0b4291293bab345acaf376e00204ed87565
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
This should be using qputenv; putenv is deprecated on MSVC.
Change-Id: I7c27cf5f7955624fa3553b7a34ab11c6fae462b8
Reviewed-by: Oliver Wolff <oliver.wolff@digia.com>
Changed the processing of non-character code handling in the UTF8 codec.
Non-character codes are now accepted in QStrings, QUrls and QJson strings.
Unit tests were adapted accordingly.
For more info about non-character codes,
see: http://www.unicode.org/versions/corrigendum9.html
[ChangeLog][QtCore][QUtf8]
UTF-8 now accepts non-character unicode points; these are not replaced
by the replacement character anymore
[ChangeLog][QtCore][QUrl]
QUrl now fully accepts non-character unicode points; they are encoded as
percent characters; they can also be pretty decoded
[ChangeLog][QtCore][QJson]
The Writer and the Parser now fully accept non-character unicode points.
Change-Id: I77cf4f0e6210741eac8082912a0b6118eced4f77
Task-number: QTBUG-33229
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
the function is a trivial wrapper for the QTextCodec one, so there is
little point in explicitly testing it.
Change-Id: I0c4950e5a54b7ffff9ba73a001cedb517497a596
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Codecs registered by creating new QTextCodec instances should be listed
there.
Task-number: QTBUG-32500
Change-Id: I56c00e0d6bbfef55a6cbd571bcf9aa2cf333ef3a
Reviewed-by: David Faure <david.faure@kdab.com>
Return the codec if one was found by QTextCodec::codecForUtfText,
instead of returning the default (UTF-8).
Task-number: QTBUG-31293
Change-Id: I95e3260376c00537006b7fbfdc3df5850e1ba657
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
QTextCodec::codecForHtml currently fails to detect the charset for this
HTML:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=9,chrome=1">
<title>Test</title>
</head>
This patch makes the detection of charsets more flexible, allowing for
the use of the HTML 5 charset attribute as well more terminator characters
("'", and ">").
I also added a *_data function for the unit tests.
Task-number: QTBUG-5451
Change-Id: I69fe4a04582f0d845cbbe9140a86a950fb7dc861
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Reviewed-by: Denis Dzyubenko <denis@ddenis.info>
