QLibrary: fix deadlock caused by fix to QTBUG-39642

Commit ae6f73e856 inserted a mutex around
the entire load_sys(). We had reasoed that deadlocks would only occur if
the object creation in instance() recursed into its own instance(),
which was already a bug. But we had forgotten that dlopen()/
LoadLibrary() executes initialization code from the module being loaded,
which could cause a recursion back into the same QPluginLoader or
QLibrary object. This recursion is benign because the module *is* loaded
and dlopen()/LoadLibrary() returns the same handle.

[ChangeLog][QtCore][QLibrary and QPluginLoader] Fixed a deadlock that
would happen if the plugin or library being loaded has load-time
initialization code (C++ global variables) that recursed back into the
same QLibrary or QPluginLoader object.

PS: QLibraryPrivate::loadPlugin() updates pluginState outside a mutex
lock, so pluginState should be made an atomic variable. Once that is
done, we'll only need locking the mutex to update errorString (no
locking before loading).

Fixes: QTBUG-83207
Task-number: QTBUG-39642
Change-Id: Ibdc95e9af7bd456a94ecfffd160209304e5ab2eb
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: David Faure <david.faure@kdab.com>

src/corelib/plugin/qlibrary.cpp

@@ -576,9 +576,7 @@ bool QLibraryPrivate::load()
 
     Q_TRACE(QLibraryPrivate_load_entry, fileName);
 
-    mutex.lock();
     bool ret = load_sys();
-    mutex.unlock();
     if (qt_debug_component()) {
         if (ret) {
             qDebug() << "loaded library" << fileName;

src/corelib/plugin/qlibrary_unix.cpp

@@ -123,6 +123,7 @@ QStringList QLibraryPrivate::prefixes_sys()
 
 bool QLibraryPrivate::load_sys()
 {
+    QMutexLocker locker(&mutex);
     QString attempt;
     QFileSystemEntry fsEntry(fileName);
 
@@ -213,6 +214,7 @@ bool QLibraryPrivate::load_sys()
     }
 #endif
 
+    locker.unlock();
     bool retry = true;
     Handle hnd = nullptr;
     for (int prefix = 0; retry && !hnd && prefix < prefixes.size(); prefix++) {
@@ -273,6 +275,8 @@ bool QLibraryPrivate::load_sys()
         }
     }
 #endif
+
+    locker.relock();
     if (!hnd) {
         errorString = QLibrary::tr("Cannot load library %1: %2").arg(fileName, qdlerror());
     }

src/corelib/plugin/qlibrary_win.cpp

@@ -78,6 +78,7 @@ bool QLibraryPrivate::load_sys()
     //     fileName
     //
     // NB If it's a plugin we do not ever try the ".dll" extension
+    QMutexLocker locker(&mutex);
     QStringList attempts;
 
     if (pluginState != IsAPlugin)
@@ -95,6 +96,7 @@ bool QLibraryPrivate::load_sys()
         attempts.prepend(QDir::rootPath() + fileName);
 #endif
 
+    locker.unlock();
     Handle hnd = nullptr;
     for (const QString &attempt : qAsConst(attempts)) {
 #ifndef Q_OS_WINRT
@@ -115,6 +117,7 @@ bool QLibraryPrivate::load_sys()
 #ifndef Q_OS_WINRT
     SetErrorMode(oldmode);
 #endif
+    locker.relock();
     if (!hnd) {
         errorString = QLibrary::tr("Cannot load library %1: %2").arg(
                     QDir::toNativeSeparators(fileName), qt_error_string());