213 lines
8.7 KiB
Markdown
213 lines
8.7 KiB
Markdown
Console Handles
|
|
===============
|
|
|
|
This document will attempt to explain how console handles work and how they
|
|
interact with process creation and console attachment and detachment. It is
|
|
based on experiments that I ran against various versions of Windows.
|
|
|
|
Traditional semantics
|
|
---------------------
|
|
|
|
In releases prior to Windows 8, console handles are not true NT handles.
|
|
Rather, the values are always multiples of four minus one (i.e. 0x3, 0x7,
|
|
0xb, 0xf, ...), and the functions in `kernel32.dll` detect the special handles
|
|
and perform RPCs to `csrss.exe` and/or `conhost.exe`.
|
|
|
|
Whenever a new console is created, Windows replaces the attached process'
|
|
set of open console handles (*ConsoleHandleSet*) with three inheritable handles
|
|
(0x3, 0x7, 0xb) and sets `STDIN/STDOUT/STDERR` to these handles. This
|
|
behavior applies in these cases:
|
|
|
|
- at process startup if `CreateProcess` was called with `CREATE_NEW_CONSOLE` set
|
|
- at process startup if `CreateProcess` was called with `CREATE_NO_WINDOW` set
|
|
- when `AllocConsole` is called
|
|
|
|
|
|
XXX: `AllocConsole` seemed to leave `STDIN/STDOUT/STDERR` alone on the BSOD Vista
|
|
test case, when launched from Cygwin. In that case, the standard handles were already
|
|
pointing at the Cygwin pty pipes. Do more testing here... Also test AttachConsole
|
|
with pipe stanard handles.
|
|
|
|
|
|
Whenever a process inherits or attaches to an existing console, its
|
|
*ConsoleHandleSet* is completely replaced by the set of inheritable open
|
|
handles from the originating process. Additionally, `AttachConsole` resets
|
|
the `STDIN/STDOUT/STDERR` handles to (0x3, 0x7, 0xb), even if those handles
|
|
are not open. This behavior applies in these cases:
|
|
|
|
- at process startup (all other cases)
|
|
- when `AttachConsole` is called
|
|
|
|
After calling `FreeConsole`, no console APIs work, and all previous console
|
|
handles are apparently closed -- even `GetHandleInformation` fails on the
|
|
handles. `FreeConsole` has no effect on the `STDIN/STDOUT/STDERR` values.
|
|
|
|
A new console's initial console handles are always inheritable, but
|
|
non-inheritable handles can also be created. The inheritability can usually
|
|
be changed, except on Windows 7 (see notes below). The `bInheritHandles`
|
|
parameter to `CreateProcess` has no effect on console handles, which are
|
|
always inherited if they are marked inheritable. (As such, the
|
|
`PROC_THREAD_ATTRIBUTE_HANDLE_LIST` attribute should be irrelevant to console
|
|
handles; they would already be inherited.) (XXX: However, verify that the
|
|
attribute does not suppress inheritance.)
|
|
|
|
Traditional console handles cannot be duplicated to other processes.
|
|
|
|
Windows 8 semantics
|
|
-------------------
|
|
|
|
### Console handles
|
|
|
|
Starting with Windows 8, console handles are true NT kernel handles that
|
|
reference NT kernel objects.
|
|
|
|
If a process is attached to a console, then it will have two handles open
|
|
to `\Device\ConDrv` that Windows uses internally. These handles are never
|
|
observable by the user program. (To view them, use `handle.exe` from
|
|
sysinternals, i.e. `handle.exe -a -p <pid>`.) A process with no attached
|
|
console never has these two handles open.
|
|
|
|
Ordinary I/O console handles are also associated with `\Device\ConDrv`. The
|
|
underlying console objects can be classified in two ways:
|
|
|
|
- *Input* vs *Output*
|
|
- *Bound* vs *Unbound*
|
|
|
|
A *Bound* *Input* object is tied to a particular console, and a *Bound*
|
|
*Output* object is tied to a particular console screen buffer. These
|
|
objects are usable only if the process is attached to the correct
|
|
console. *Bound* objects are created through these methods only:
|
|
|
|
- `CreateConsoleScreenBuffer`
|
|
- opening `CONIN$` or `CONOUT$`
|
|
|
|
Most console objects are *Unbound*, which are created during console
|
|
initialization. For any given console API call, an *Unbound* *Input* object
|
|
refers to the currently attached console's input queue, and an *Unbound*
|
|
*Output* object refers to the screen buffer that was active during the calling
|
|
process' console initialization. These objects are usable as long as the
|
|
calling process has any console attached.
|
|
|
|
### Console initialization
|
|
|
|
When a process' console state is initialized, Windows may open new handles.
|
|
This happens in these instances:
|
|
|
|
- at process startup if `CreateProcess` was called with `CREATE_NEW_CONSOLE` set
|
|
- at process startup if `CreateProcess` was called with `CREATE_NO_WINDOW` set
|
|
- at process startup if `CreateProcess` was called with `bInheritHandles=FALSE`
|
|
- when `AttachConsole` is called
|
|
- when `AllocConsole` is called
|
|
|
|
When it opens handles, Windows sets `STDIN/STDOUT/STDERR` to three newly opened
|
|
handles to two *Unbound* console objects.
|
|
|
|
Regardless of whether new handles were opened, Windows increments a refcount
|
|
on the active screen buffer, which decrements only when the process detaches
|
|
from the console.
|
|
|
|
As in previous Windows releases, `FreeConsole` in Windows 8 does not change
|
|
the `STDIN/STDOUT/STDERR` values. If Windows opened new handles for
|
|
`STDIN/STDOUT/STDERR` when it initialized the process' console state, then
|
|
`FreeConsole` will close those handles. Otherwise, `FreeConsole` will only
|
|
close the two internal handles.
|
|
|
|
### Interesting Consequences
|
|
|
|
* `FreeConsole` can close a non-console handle. This happens if:
|
|
|
|
1. Windows had opened handles during console initialization.
|
|
2. The program closes its standard handles and opens new non-console
|
|
handles with the same values.
|
|
3. The program calls `FreeConsole`.
|
|
|
|
(Perhaps programs are not expected to close their standard handles.)
|
|
|
|
* Console handles--*Bound* or *Unbound*--**can** be duplicated to other
|
|
processes. The duplicated handles are sometimes usable, especially
|
|
if *Unbound*. The same *Unbound* *Output* object can be open in two
|
|
different processes and refer to different screen buffers in the same
|
|
console or in different consoles.
|
|
|
|
* Even without duplicating console handles, it is possible to have open
|
|
console handles that are not usable, even with a console attached.
|
|
|
|
* Dangling *Bound* handles are not allowed, so it is possible to have
|
|
consoles with no attached processes. The console cannot be directly
|
|
modified (or attached to), but its visible content can be changed by
|
|
closing *Bound* *Output* handles to activate other screen buffers.
|
|
|
|
* A program that repeatedly reinvoked itself with `CREATE_NEW_CONSOLE` and
|
|
`bInheritHandles=TRUE` would accumulate console handles. Each child
|
|
would inherit all of the previous child's console handles, then allocate
|
|
three more for itself. All of the handles would be usable if the
|
|
program kept track of them somehow.
|
|
|
|
Other notes
|
|
-----------
|
|
|
|
### SetActiveConsoleScreenBuffer
|
|
|
|
Screen buffers are referenced counted. Changing the active screen buffer
|
|
with `SetActiveConsoleScreenBuffer` does not increment a refcount on the
|
|
buffer. If the active buffer's refcount hits zero, then Windows chooses
|
|
another buffer and activates it.
|
|
|
|
### `CREATE_NO_WINDOW` process creation flag
|
|
|
|
The documentation for `CREATE_NO_WINDOW` is confusing:
|
|
|
|
> The process is a console application that is being run without a
|
|
> console window. Therefore, the console handle for the application is
|
|
> not set.
|
|
>
|
|
> This flag is ignored if the application is not a console application,
|
|
> or if it is used with either `CREATE_NEW_CONSOLE` or `DETACHED_PROCESS`.
|
|
|
|
Here's what's evident from examining the OS behavior:
|
|
|
|
* Specifying both `CREATE_NEW_CONSOLE` and `DETACHED_PROCESS` causes the
|
|
`CreateProcess` call to fail.
|
|
|
|
* If `CREATE_NO_WINDOW` is specified together with `CREATE_NEW_CONSOLE` or
|
|
`DETACHED_PROCESS`, it is quietly ignored, just as documented.
|
|
|
|
* Otherwise, `CreateProcess` behaves the same way with `CREATE_NO_WINDOW` as
|
|
it does with `CREATE_NEW_CONSOLE`, except that the new console either has
|
|
a hidden window (before Windows 7) or has no window at all (Windows 7
|
|
and later). These situations can be distinguished using the
|
|
`GetConsoleWindow` and `IsWindowVisible` calls. `GetConsoleWindow` returns
|
|
`NULL` starting with Windows 7.
|
|
|
|
### Windows Vista BSOD
|
|
|
|
It is easy to cause a BSOD on Vista and Server 2008 by (1) closing all handles
|
|
to the last screen buffer, then (2) creating a new screen buffer:
|
|
|
|
#include <windows.h>
|
|
int main() {
|
|
FreeConsole();
|
|
AllocConsole();
|
|
CloseHandle((HANDLE)0x7);
|
|
CloseHandle((HANDLE)0xb);
|
|
CreateConsoleScreenBuffer(
|
|
GENERIC_READ | GENERIC_WRITE,
|
|
FILE_SHARE_READ | FILE_SHARE_WRITE,
|
|
NULL,
|
|
CONSOLE_TEXTMODE_BUFFER,
|
|
NULL);
|
|
return 0;
|
|
}
|
|
|
|
### Windows 7 inheritability
|
|
|
|
* Calling `DuplicateHandle(h, FALSE)` on an inheritable console handle
|
|
produces an inheritable handle. According to documentation and previous
|
|
releases, it should be non-inheritable.
|
|
|
|
* Calling `SetHandleInformation` fails on console handles.
|
|
|
|
### Windows 7 `conhost.exe` crash with `CONOUT$`
|
|
|
|
XXX: Document this. It's a problem...
|