macOS: Compute NSWindow background color without checking styleMask

The check for styleMask == NSWindowStyleMaskBorderless to decide whether
to clear the NSWindow background was broken, as NSWindowStyleMaskBorderless
has the value 0, but is only supposed to be compared to its companion
NSWindowStyleMaskTitled (with value 1). A window can perfectly well be
NSWindowStyleMaskBorderless and NSWindowStyleMaskMiniaturizable e.g.,
so by comparing directly to NSWindowStyleMaskBorderless instead of
masking to the first bit first we ended up making miniaturizable
windows non-translucent.

We now check the Qt::FramelessWindowHint directly, and also whether
the window is opaque. Ideally we'd have QWindow flags that could
plumb WA_NoSystemBackground from Qt Widgets, as well as a background
color property on QWindow to control the system background, but
in the meantime we'll have to use the FramelessWindowHint heuristic.

The QWidget docs have been updated to reflect this.

Task-number: QTBUG-95042
Pick-to: 6.2
Change-Id: I0d40eecace60883c205ebb8c76cef1092cdf1144
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>

src/plugins/platforms/cocoa/qcocoawindow.mm

@@ -529,9 +529,7 @@ NSUInteger QCocoaWindow::windowStyleMask(Qt::WindowFlags flags)
     if (frameless) {
         // Frameless windows do not display the traffic lights buttons for
         // e.g. minimize, however StyleMaskMiniaturizable is required to allow
-        // programatic minimize. However, for framless tool windows (e.g. dock windows)
-        // we don't want that, as it breaks translucency.
-        if (type != Qt::Tool)
+        // programmatic minimize.
         styleMask |= NSWindowStyleMaskMiniaturizable;
     } else if (flags & Qt::CustomizeWindowHint) {
         if (flags & Qt::WindowTitleHint)

src/plugins/platforms/cocoa/qnswindow.mm

@@ -324,8 +324,18 @@ OSStatus CGSClearWindowTags(const CGSConnectionID, const CGSWindowID, int *, int
 
 - (NSColor *)backgroundColor
 {
-    return self.styleMask == NSWindowStyleMaskBorderless ?
-        [NSColor clearColor] : [super backgroundColor];
+    // FIXME: Plumb to a WA_NoSystemBackground-like window flag,
+    // or a QWindow::backgroundColor() property. In the meantime
+    // we assume that if you have translucent content, without a
+    // frame then you intend to do all background drawing yourself.
+    const QWindow *window = m_platformWindow ? m_platformWindow->window() : nullptr;
+    if (!self.opaque && window && window->flags().testFlag(Qt::FramelessWindowHint))
+        return [NSColor clearColor];
+
+    // This still allows you to have translucent content with a frame,
+    // where the system background (or color set via NSWindow) will
+    // shine through.
+    return [super backgroundColor];
 }
 
 - (void)sendEvent:(NSEvent*)theEvent

src/widgets/kernel/qwidget.cpp

@@ -792,6 +792,8 @@ void QWidget::setAutoFillBackground(bool enabled)
     and a compositing window manager.
     \li Windows: The widget needs to have the Qt::FramelessWindowHint window flag set
     for the translucency to work.
+    \li \macos: The widget needs to have the Qt::FramelessWindowHint window flag set
+    for the translucency to work.
     \endlist