QColor: Fix UB (left shift of negative number)

If hex2int(const char*) is called with invalid input, it is expected to
return a negative value. However, it didn't check the return value of
h2i() before attempting a left-shift on it, leading to UB when the first
digit was already invalid.

UBSan agrees:

  qcolor_p.cpp:55:23: runtime error: left shift of negative value -1

This is particularly worrisome as the function can be called with
unsanitized input.

Fix by checking each value for non-negativity, returning -1 early
when errors are detected.

Also port to QtMiscUtils::fromHex() and add some docs.

Change-Id: I33dbc157ffb4fbfba27113a0a008eef35c1055f7
Reviewed-by: Olivier Goffart (Woboq GmbH) <ogoffart@woboq.com>

src/gui/painting/qcolor_p.cpp

@@ -34,31 +34,37 @@
 #include "qglobal.h"
 #include "qrgb.h"
 #include "qstringlist.h"
+#include "private/qtools_p.h"
 
 #include <algorithm>
 
 QT_BEGIN_NAMESPACE
 
-static inline int h2i(char hex)
+/*!
-{
+    \internal
-    if (hex >= '0' && hex <= '9')
+    If s[0..1] is a valid hex number, returns its integer value,
-        return hex - '0';
+    otherwise returns -1.
-    if (hex >= 'a' && hex <= 'f')
+ */
-        return hex - 'a' + 10;
-    if (hex >= 'A' && hex <= 'F')
-        return hex - 'A' + 10;
-    return -1;
-}
-
 static inline int hex2int(const char *s)
 {
-    return (h2i(s[0]) << 4) | h2i(s[1]);
+    const int hi = QtMiscUtils::fromHex(s[0]);
+    if (hi < 0)
+        return -1;
+    const int lo = QtMiscUtils::fromHex(s[1]);
+    if (lo < 0)
+        return -1;
+    return (hi << 4) | lo;
 }
 
+/*!
+    \internal
+    If s is a valid hex digit, returns its integer value,
+    multiplied by 0x11, otherwise returns -1.
+ */
 static inline int hex2int(char s)
 {
-    int h = h2i(s);
+    const int h = QtMiscUtils::fromHex(s);
-    return (h << 4) | h;
+    return h < 0 ? h : (h << 4) | h;
 }
 
 bool qt_get_hex_rgb(const char *name, QRgb *rgb)