2000-08-18 00:26:52 +00:00
|
|
|
<html lang="en">
|
1999-12-03 21:29:57 +00:00
|
|
|
<head>
|
2000-08-18 00:26:52 +00:00
|
|
|
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
|
|
|
|
|
<title>ICU Codepage Conversion</title>
|
1999-12-03 21:29:57 +00:00
|
|
|
</head>
|
|
|
|
|
|
2000-08-18 00:26:52 +00:00
|
|
|
<body>
|
2000-01-15 02:00:06 +00:00
|
|
|
|
2000-05-05 20:31:10 +00:00
|
|
|
<h1>International Components for Unicode</h1>
|
1999-12-03 21:29:57 +00:00
|
|
|
|
2000-08-18 00:26:52 +00:00
|
|
|
<h2>ICU Codepage Conversion</h2>
|
|
|
|
|
|
|
|
|
|
<p>The ICU conversion API is a set of C functions used to convert to and from
|
|
|
|
|
Unicode and various character sets (codepages, encodings, character encoding schemes).</p>
|
|
|
|
|
|
|
|
|
|
<h3>Conversion-related files:</h3>
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
<li>API: The API header files are in icu/source/common/unicode:<br>
|
|
|
|
|
For C, the API is defined in ucnv.h;
|
|
|
|
|
advanced functionality is also defined in ucnv_err.h (callbacks) and
|
|
|
|
|
in ucnv_cb.h (output functions for custom callbacks).<br>
|
|
|
|
|
For C++ the API is defined in convert.h (the C++ class is a wrapper around the C implementation).</li>
|
|
|
|
|
<li>Implementation: The converter implementation files are in icu/source/common;
|
|
|
|
|
all such files begin with "ucnv". The C++ wrapper implementation is in convert.cpp.</li>
|
|
|
|
|
<li>Conversion table generation tool: The makeconv tool that generates binary conversion files
|
|
|
|
|
from text files is in icu/source/tools/makeconv.
|
|
|
|
|
It reads .ucm text files with a format that is close to what the AIX tool uconvdef uses.
|
|
|
|
|
makeconv writes one binary, memory-mappable .cnv file per .ucm file.</li>
|
|
|
|
|
<li>Conversion data: The .ucm text files with the conversion table data are all in
|
|
|
|
|
the icu/data folder. During the build process, makeconv generates binary .cnv files from
|
|
|
|
|
each of them, and the pkgdata tool includes them into the common data file.<br>
|
|
|
|
|
In addition, the file icu/data/convrtrs.txt contains information about "aliases", i.e.,
|
|
|
|
|
alternative names for converters. It is read by gencnval (in icu/source/tools/gencnval)
|
|
|
|
|
which writes the binary file cnvalias.dat that also gets packaged into the common data file.</li>
|
1999-12-03 21:29:57 +00:00
|
|
|
</ul>
|
|
|
|
|
|
2000-08-18 00:26:52 +00:00
|
|
|
<h2>Converter types</h2>
|
|
|
|
|
|
|
|
|
|
<p>In order to handle many kinds of character encoding schemes, ICU has a number of
|
|
|
|
|
converter implementations, one per type. Some of these types are for purely algorithmic
|
|
|
|
|
conversions that do not need to load data. For example, the UTF converters calculate
|
|
|
|
|
Unicode code points from the input bytes, and vice versa. Also, the ISO_2022 converter
|
|
|
|
|
starts without any specific conversion data table until it needs to - handling escape
|
|
|
|
|
sequences and the general structure of ISO 2022 is done with static data.</p>
|
|
|
|
|
|
|
|
|
|
<p>Many other encodings share common characteristics and need by definition tables
|
|
|
|
|
to convert text between them and Unicode. A converter object for such an encoding
|
|
|
|
|
is instantiated by loading a (.cnv) data file (typically from the single, common
|
|
|
|
|
ICU data file) and associating it with a converter type implementation depending
|
|
|
|
|
on the type information in the data.</p>
|
|
|
|
|
|
|
|
|
|
<p>The following describes specifics about each converter type:</p>
|
|
|
|
|
|
|
|
|
|
<h3>MBCS</h3>
|
|
|
|
|
|
|
|
|
|
<p>The MBCS converter is a data-based converter for Multi-Byte Characater Sets.
|
|
|
|
|
It has been reimplemented for ICU 1.6 to handle a wider range of such encodings.
|
|
|
|
|
Its current capabilities and limitations are:
|
|
|
|
|
<ul>
|
|
|
|
|
<li>Support for variable-length, byte-based encodings with 1 to 4 bytes per character.</li>
|
|
|
|
|
<li>Support for all Unicode characters (code points 0..0x10ffff).
|
|
|
|
|
Since ICU uses UTF-16 as its Unicode encoding form, this means that surrogate
|
|
|
|
|
pairs are fully supported.</li>
|
|
|
|
|
<li>Efficient distinction of unassigned vs. illegal byte sequences.</li>
|
|
|
|
|
<li>It would possible in fromUnicode() to directly deal with simple
|
|
|
|
|
stateful encodings. (This is currently not used.)</li>
|
|
|
|
|
<li>It is possible to convert Unicode code points other than U+0000
|
|
|
|
|
to a single zero byte (but not as a fallback).</li>
|
|
|
|
|
<li>It is not otherwise possible to convert from Unicode to byte sequences
|
|
|
|
|
with leading zero bytes.</li>
|
1999-12-03 21:29:57 +00:00
|
|
|
</ul>
|
2000-08-18 00:26:52 +00:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>The conversion to Unicode uses a state machine to achieve the above capabilities with
|
|
|
|
|
reasonable data file sizes. The state machine information itself is loaded with the
|
|
|
|
|
conversion data and defines the structure of the codepage, including which byte sequences
|
|
|
|
|
are valid, unassigned, and illegal. This data cannot (or not easily) be computed from
|
|
|
|
|
the pure mapping data. Instead, the .ucm files for MBCS encodings have additional entries
|
|
|
|
|
that are specific to ICU's makeconv and this converter type. They are additional header lines
|
|
|
|
|
that start with <code><icu:state></code>. Each such line defines one state of the state machine.
|
|
|
|
|
The state machine uses a table of as many rows as there are states (= as many as there are
|
|
|
|
|
<code><icu:state></code> lines). Each row has 256 entries, one for each possible byte value.</p>
|
|
|
|
|
|
|
|
|
|
<p>The state table lines in the .ucm header follow the following EBNF-like grammar
|
|
|
|
|
(whitespace is allowed between all tokens):
|
|
|
|
|
<pre>
|
|
|
|
|
row=[firstentry ','] entry (',' entry)*
|
|
|
|
|
firstentry="initial" | "surrogates"
|
|
|
|
|
(initial state (default for state 0), output is all surrogate pairs)
|
|
|
|
|
</pre>
|
|
|
|
|
Each state table row description (that follows the <code><icu:state></code>)
|
|
|
|
|
begins with an optional <code>initial</code> or <code>surrogates</code> keyword
|
|
|
|
|
and is followed by one or more column entries.
|
|
|
|
|
For the purpose of MBCS state tables, the states=rows in the table are numbered
|
|
|
|
|
beginning with 0 at the first such line in the .ucm file header.
|
|
|
|
|
The numbers are assigned implicitly by makeconv in order of the <code><icu:state></code>
|
|
|
|
|
lines.
|
|
|
|
|
<pre>
|
|
|
|
|
entry=range [':' nextstate] ['.' [action]]
|
|
|
|
|
range=number ['-' number]
|
|
|
|
|
nextstate=number
|
|
|
|
|
(0..7f)
|
|
|
|
|
action='u' | 's' | 'p' | 'i'
|
|
|
|
|
(unassigned, state change only, surrogate pair, illegal)
|
|
|
|
|
number=(1- or 2-digit hexadecimal number)
|
|
|
|
|
</pre>
|
|
|
|
|
Each column entry consists at least of a hexadecimal byte value or value range
|
|
|
|
|
and is separated by the following column entry by a comma.
|
|
|
|
|
The column entry specifies how to interpret an input byte in the row's state.
|
|
|
|
|
If neither a next state nor an action is explicitly specified - only the byte
|
|
|
|
|
value (range) is given - then the byte value terminates the byte sequence,
|
|
|
|
|
results in a valid mapping to a Unicode BMP character, and the state number is
|
|
|
|
|
reset to 0.</p>
|
|
|
|
|
|
|
|
|
|
<p>The next state can be explicitly specified with a separating
|
|
|
|
|
colon (<code>:</code>) followed by the number of the state (=number/index of the row,
|
|
|
|
|
starting at 0). This is mostly used for intermediate byte values, i.e., for
|
|
|
|
|
bytes that are not the last ones in a sequence. The state machine needs to
|
|
|
|
|
proceed to the next state and read another byte. In this case, no other action
|
|
|
|
|
is specified.</p>
|
|
|
|
|
|
|
|
|
|
<p>If the byte value(s) terminate(s) a byte sequence, then the byte
|
|
|
|
|
sequence results in the following depending on the action that is announced with
|
|
|
|
|
a period (<code>.</code>) followed by a letter:
|
|
|
|
|
<ul>
|
|
|
|
|
<li><code>u</code> - Unassigned. The byte sequence is valid but does not encode a character.</li>
|
|
|
|
|
<li>(no letter) - valid. If no action letter is specified, then
|
|
|
|
|
the byte sequence is valid and encodes a Unicode character up to
|
|
|
|
|
U+ffff.</li>
|
|
|
|
|
<li><code>p</code> - surrogate Pair. The byte sequence is valid and may result in</li>
|
|
|
|
|
<li><code>i</code> - Illegal. The byte sequence is illegal. This is the default for
|
|
|
|
|
all byte values in a row that are not otherwise specified with
|
|
|
|
|
column entries.</li>
|
|
|
|
|
<li><code>s</code> - State change only. The byte sequence does not encode any character
|
|
|
|
|
but may change the state number. This could be used with simple, stateful
|
|
|
|
|
encodings (using, for example, SI/SO codes),
|
|
|
|
|
but ICU currently does not take advantage of it.</li>
|
1999-12-03 21:29:57 +00:00
|
|
|
</ul>
|
2000-08-18 00:26:52 +00:00
|
|
|
If an action is specified but no next state, then the next state number defaults to 0.
|
|
|
|
|
In other words, a byte value (range) terminates a sequence if there is an action
|
|
|
|
|
specified for it, or when there is neither an action nor a next state - in this case,
|
|
|
|
|
it defaults to "valid, next state is 0" (equivalent to <code>:0.</code>).</p>
|
|
|
|
|
|
|
|
|
|
<p>If a byte value is not specified in any column entry of a row, then it is
|
|
|
|
|
illegal in the current state. If a byte value is specified in more than one column
|
|
|
|
|
entry of the same row, then the last one is used. This allows to specify common
|
|
|
|
|
properties for a wide byte value range followed by a few exceptions and is easier than
|
|
|
|
|
having to specify mutually exclusive ranges, especially if many of them have the
|
|
|
|
|
same properties.</p>
|
|
|
|
|
|
|
|
|
|
<p>The optional keyword at the beginning of a state line has the following effect:
|
|
|
|
|
<ul>
|
|
|
|
|
<li><code>initial</code>: The state machine can start reading byte sequences
|
|
|
|
|
in this state. State 0 is always an initial state. Only initial states can be
|
|
|
|
|
next states for final byte values. In an initial state, the Unicode mappings
|
|
|
|
|
for all final bytes are also stored directly in the state table.</li>
|
|
|
|
|
<li><code>surrogates</code>: All Unicode mappings for final bytes in non-initial
|
|
|
|
|
states are stored in a separate table of 16-bit Unicode (UTF-16) code units.
|
|
|
|
|
Since most legacy codepages map only to Unicode code points up to U+ffff
|
|
|
|
|
(the Basic Multilingual Plane, BMP), the default allocation per mapping
|
|
|
|
|
result is one 16-bit unit. Individual byte values can be specified to map
|
|
|
|
|
to surrogate pairs (= two 16-bit units) with action letter <code>p</code>.
|
|
|
|
|
The <code>surrogates</code> keyword specifies this for the entire state (row).
|
|
|
|
|
Surrogate pair mapping entries can still hold single units depending on the
|
|
|
|
|
actual mapping data, but single-unit mapping entries cannot hold a pair of units.
|
|
|
|
|
Mapping to single-unit entries is the default because the mapping is faster,
|
|
|
|
|
uses half as much memory in the code units table, and is sufficient for most
|
|
|
|
|
legacy codepages.</li>
|
|
|
|
|
</ul>
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>When converting to Unicode, the state machine starts in state number 0.
|
|
|
|
|
In each iteration, it reads one input (codepage) byte and either just goes to
|
|
|
|
|
the next state as specified, or treats it as a final byte with the specified action
|
|
|
|
|
and an optional non-0 next (initial) state. This means that a state table needs to
|
|
|
|
|
have at least as many state rows as the maximum number of bytes per character,
|
|
|
|
|
which is the maximum length of any byte sequence.</p>
|
|
|
|
|
|
|
|
|
|
<h4>Examples for MBCS state tables</h4>
|
|
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
|
<li>US-ASCII:
|
|
|
|
|
<pre>
|
|
|
|
|
0-7f
|
|
|
|
|
</pre>
|
|
|
|
|
This single-row state table describes US-ASCII.
|
|
|
|
|
Byte values from 0 to 0x7f are valid and map to Unicode character up to U+ffff.
|
|
|
|
|
Byte values from 0x80 to 0xff are illegal.<br>
|
|
|
|
|
</li>
|
|
|
|
|
<li>Shift-JIS:
|
|
|
|
|
<pre>
|
|
|
|
|
0-7f, 81-9f:1, a0-df, e0-fc:1
|
|
|
|
|
40-7e, 80-fc
|
|
|
|
|
</pre>
|
|
|
|
|
This two-row state table describes the structure of Shift-JIS, which encodes some characters
|
|
|
|
|
with one byte each, and others with two bytes each.
|
|
|
|
|
Bytes 0 to 0x7f and 0xa0 to 0xdf are valid single-byte encodings.
|
|
|
|
|
Bytes 0x81 to 0x9f and 0xe0 to 0xfc are lead bytes, i.e., they are followed by one of
|
|
|
|
|
the bytes that are specified as valid in state 1.
|
|
|
|
|
A byte sequence of 0x85 0x61 is valid, while a single byte of 0x80 or 0xff is illegal.
|
|
|
|
|
Similarly, a byte sequence of 0x85 0x31 is illegal.<br>
|
|
|
|
|
</li>
|
|
|
|
|
<li>EUC-JP:
|
|
|
|
|
<pre>
|
|
|
|
|
0-8d, 8e:2, 8f:3, 90-9f, a1-fe:1
|
|
|
|
|
a1-fe
|
|
|
|
|
a1-e4
|
|
|
|
|
a1-fe:1, a1:4, a3-af:4, b6:4, d6:4, da-db:4, ed-f2:4
|
|
|
|
|
a1-fe.u
|
|
|
|
|
</pre>
|
|
|
|
|
This fairly complicated state table describes EUC-JP.
|
|
|
|
|
Valid byte sequences are one, two, or three bytes long.
|
|
|
|
|
Two-byte sequences have lead byte 0x8e and end in state 2, or
|
|
|
|
|
lead bytes 0xa1 to 0xfe and end in state 1.
|
|
|
|
|
Three-byte sequences have a lead byte of 0x8f and continue in state 3.
|
|
|
|
|
Some final byte value ranges are entirely unassigned, therefore they end in state 4
|
|
|
|
|
with an action letter of <code>u</code> for "unassigned" to save significant memory
|
|
|
|
|
for the code units table.
|
|
|
|
|
Assigned three-byte sequences end in state 1 like most two-byte sequences.<br>
|
|
|
|
|
<em>Note: </em>This reuse of a final or intermediate state is valid for as long
|
|
|
|
|
as there is no circle in the state chain. The mappings will be unique because of
|
|
|
|
|
the different path to the shared state.
|
|
|
|
|
(Sharing a state saves some memory: Each state table row occupies 1kB in the .cnv file.)<br>
|
|
|
|
|
This table also shows the redefinition of byte value ranges within one state row
|
|
|
|
|
(number 3) as a shorthand.<br>
|
|
|
|
|
</li>
|
1999-12-03 21:29:57 +00:00
|
|
|
</ul>
|
2000-01-15 02:00:06 +00:00
|
|
|
|
1999-12-03 21:29:57 +00:00
|
|
|
</body>
|
|
|
|
|
</html>
|
