winpty/misc/console-handles.md

Console Handles
===============

This document will attempt to explain how console handles work and how they
interact with process creation and console attachment and detachment.  It is
based on experiments that I ran against various versions of Windows.

Traditional semantics
---------------------

In releases prior to Windows 8, console handles are not true NT handles.
Rather, the values are always multiples of four minus one (i.e. 0x3, 0x7,
0xb, 0xf, ...), and the functions in `kernel32.dll` detect the special handles
and perform RPCs to `csrss.exe` and/or `conhost.exe`.

Whenever a new console is created, Windows replaces the attached process'
set of open console handles (*ConsoleHandleSet*) with three inheritable handles
(0x3, 0x7, 0xb) and sets `STDIN/STDOUT/STDERR` to these handles.  This
behavior applies in these cases:

 - at process startup if `CreateProcess` was called with `CREATE_NEW_CONSOLE` set
 - at process startup if `CreateProcess` was called with `CREATE_NO_WINDOW` set
 - when `AllocConsole` is called


XXX: `AllocConsole` seemed to leave `STDIN/STDOUT/STDERR` alone on the BSOD Vista
test case, when launched from Cygwin.  In that case, the standard handles were already
pointing at the Cygwin pty pipes.  Do more testing here...  Also test AttachConsole
with pipe stanard handles.


Whenever a process inherits or attaches to an existing console, its
*ConsoleHandleSet* is completely replaced by the set of inheritable open
handles from the originating process.  Additionally, `AttachConsole` resets
the `STDIN/STDOUT/STDERR` handles to (0x3, 0x7, 0xb), even if those handles
are not open.  This behavior applies in these cases:

 - at process startup (all other cases)
 - when `AttachConsole` is called

After calling `FreeConsole`, no console APIs work, and all previous console
handles are apparently closed -- even `GetHandleInformation` fails on the
handles.  `FreeConsole` has no effect on the `STDIN/STDOUT/STDERR` values.

A new console's initial console handles are always inheritable, but
non-inheritable handles can also be created.  The inheritability can usually
be changed, except on Windows 7 (see notes below).  The `bInheritHandles`
parameter to `CreateProcess` has no effect on console handles, which are
always inherited if they are marked inheritable.  (As such, the
`PROC_THREAD_ATTRIBUTE_HANDLE_LIST` attribute should be irrelevant to console
handles; they would already be inherited.)  (XXX: However, verify that the
attribute does not suppress inheritance.)

Traditional console handles cannot be duplicated to other processes.

Windows 8 semantics
-------------------

### Console handles

Starting with Windows 8, console handles are true NT kernel handles that
reference NT kernel objects.

If a process is attached to a console, then it will have two handles open
to `\Device\ConDrv` that Windows uses internally.  These handles are never
observable by the user program.  (To view them, use `handle.exe` from
sysinternals, i.e. `handle.exe -a -p <pid>`.)  A process with no attached
console never has these two handles open.

Ordinary I/O console handles are also associated with `\Device\ConDrv`.  The
underlying console objects can be classified in two ways:

 - *Input* vs *Output*
 - *Bound* vs *Unbound*

A *Bound* *Input* object is tied to a particular console, and a *Bound*
*Output* object is tied to a particular console screen buffer.  These
objects are usable only if the process is attached to the correct
console.  *Bound* objects are created through these methods only:

 - `CreateConsoleScreenBuffer`
 - opening `CONIN$` or `CONOUT$`

Most console objects are *Unbound*, which are created during console
initialization.  For any given console API call, an *Unbound* *Input* object
refers to the currently attached console's input queue, and an *Unbound*
*Output* object refers to the screen buffer that was active during the calling
process' console initialization.  These objects are usable as long as the
calling process has any console attached.

### Console initialization

When a process' console state is initialized, Windows may open new handles.
This happens in these instances:

 - at process startup if `CreateProcess` was called with `CREATE_NEW_CONSOLE` set
 - at process startup if `CreateProcess` was called with `CREATE_NO_WINDOW` set
 - at process startup if `CreateProcess` was called with `bInheritHandles=FALSE`
 - when `AttachConsole` is called
 - when `AllocConsole` is called

When it opens handles, Windows sets `STDIN/STDOUT/STDERR` to three newly opened
handles to two *Unbound* console objects.

Regardless of whether new handles were opened, Windows increments a refcount
on the active screen buffer, which decrements only when the process detaches
from the console.

As in previous Windows releases, `FreeConsole` in Windows 8 does not change
the `STDIN/STDOUT/STDERR` values.  If Windows opened new handles for
`STDIN/STDOUT/STDERR` when it initialized the process' console state, then
`FreeConsole` will close those handles.  Otherwise, `FreeConsole` will only
close the two internal handles.

### Interesting Consequences

 * `FreeConsole` can close a non-console handle.  This happens if:

     1. Windows had opened handles during console initialization.
     2. The program closes its standard handles and opens new non-console
        handles with the same values.
     3. The program calls `FreeConsole`.

   (Perhaps programs are not expected to close their standard handles.)

 * Console handles--*Bound* or *Unbound*--**can** be duplicated to other
   processes.  The duplicated handles are sometimes usable, especially
   if *Unbound*.  The same *Unbound* *Output* object can be open in two
   different processes and refer to different screen buffers in the same
   console or in different consoles.

 * Even without duplicating console handles, it is possible to have open
   console handles that are not usable, even with a console attached.

 * Dangling *Bound* handles are not allowed, so it is possible to have
   consoles with no attached processes.  The console cannot be directly
   modified (or attached to), but its visible content can be changed by
   closing *Bound* *Output* handles to activate other screen buffers.

 * A program that repeatedly reinvoked itself with `CREATE_NEW_CONSOLE` and
   `bInheritHandles=TRUE` would accumulate console handles.  Each child
   would inherit all of the previous child's console handles, then allocate
   three more for itself.  All of the handles would be usable if the
   program kept track of them somehow.

Other notes
-----------

### SetActiveConsoleScreenBuffer

Screen buffers are referenced counted.  Changing the active screen buffer
with `SetActiveConsoleScreenBuffer` does not increment a refcount on the
buffer.  If the active buffer's refcount hits zero, then Windows chooses
another buffer and activates it.

### `CREATE_NO_WINDOW` process creation flag

The documentation for `CREATE_NO_WINDOW` is confusing:

>  The process is a console application that is being run without a
>  console window. Therefore, the console handle for the application is
>  not set.
>
>  This flag is ignored if the application is not a console application,
>  or if it is used with either `CREATE_NEW_CONSOLE` or `DETACHED_PROCESS`.

Here's what's evident from examining the OS behavior:

 * Specifying both `CREATE_NEW_CONSOLE` and `DETACHED_PROCESS` causes the
   `CreateProcess` call to fail.

 * If `CREATE_NO_WINDOW` is specified together with `CREATE_NEW_CONSOLE` or
   `DETACHED_PROCESS`, it is quietly ignored, just as documented.

 * Otherwise, `CreateProcess` behaves the same way with `CREATE_NO_WINDOW` as
   it does with `CREATE_NEW_CONSOLE`, except that the new console either has
   a hidden window (before Windows 7) or has no window at all (Windows 7
   and later).  These situations can be distinguished using the
   `GetConsoleWindow` and `IsWindowVisible` calls.  `GetConsoleWindow` returns
   `NULL` starting with Windows 7.

### Windows Vista BSOD

It is easy to cause a BSOD on Vista and Server 2008 by (1) closing all handles
to the last screen buffer, then (2) creating a new screen buffer:

    #include <windows.h>
    int main() {
        FreeConsole();
        AllocConsole();
        CloseHandle((HANDLE)0x7);
        CloseHandle((HANDLE)0xb);
        CreateConsoleScreenBuffer(
            GENERIC_READ | GENERIC_WRITE,
            FILE_SHARE_READ | FILE_SHARE_WRITE,
            NULL,
            CONSOLE_TEXTMODE_BUFFER,
            NULL);
        return 0;
    }

### Windows 7 inheritability

 * Calling `DuplicateHandle(h, FALSE)` on an inheritable console handle
   produces an inheritable handle.  According to documentation and previous
   releases, it should be non-inheritable.

 * Calling `SetHandleInformation` fails on console handles.

### Windows 7 `conhost.exe` crash with `CONOUT$`

XXX: Document this.  It's a problem...