The ICU conversion API is a set of C functions used to convert to and from Unicode and various character sets (codepages, encodings, character encoding schemes).
In order to handle many kinds of character encoding schemes, ICU has a number of converter implementations, one per type. Some of these types are for purely algorithmic conversions that do not need to load data. For example, the UTF converters calculate Unicode code points from the input bytes, and vice versa. Also, the ISO_2022 converter starts without any specific conversion data table until it needs to - handling escape sequences and the general structure of ISO 2022 is done with static data.
Many other encodings share common characteristics and need by definition tables to convert text between them and Unicode. A converter object for such an encoding is instantiated by loading a (.cnv) data file (typically from the single, common ICU data file) and associating it with a converter type implementation depending on the type information in the data.
The following describes specifics about each converter type:
The MBCS converter is a data-based converter for Multi-Byte Characater Sets. It has been reimplemented for ICU 1.6 to handle a wider range of such encodings. Its current capabilities and limitations are:
The conversion to Unicode uses a state machine to achieve the above capabilities with
reasonable data file sizes. The state machine information itself is loaded with the
conversion data and defines the structure of the codepage, including which byte sequences
are valid, unassigned, and illegal. This data cannot (or not easily) be computed from
the pure mapping data. Instead, the .ucm files for MBCS encodings have additional entries
that are specific to ICU's makeconv and this converter type. They are additional header lines
that start with <icu:state>. Each such line defines one state of the state machine.
The state machine uses a table of as many rows as there are states (= as many as there are
<icu:state> lines). Each row has 256 entries, one for each possible byte value.
The state table lines in the .ucm header follow the following EBNF-like grammar (whitespace is allowed between all tokens):
row=[firstentry ','] entry (',' entry)*
firstentry="initial" | "surrogates"
(initial state (default for state 0), output is all surrogate pairs)
Each state table row description (that follows the <icu:state>)
begins with an optional initial or surrogates keyword
and is followed by one or more column entries.
For the purpose of MBCS state tables, the states=rows in the table are numbered
beginning with 0 at the first such line in the .ucm file header.
The numbers are assigned implicitly by makeconv in order of the <icu:state>
lines.
entry=range [':' nextstate] ['.' [action]]
range=number ['-' number]
nextstate=number
(0..7f)
action='u' | 's' | 'p' | 'i'
(unassigned, state change only, surrogate pair, illegal)
number=(1- or 2-digit hexadecimal number)
Each column entry consists at least of a hexadecimal byte value or value range
and is separated by the following column entry by a comma.
The column entry specifies how to interpret an input byte in the row's state.
If neither a next state nor an action is explicitly specified - only the byte
value (range) is given - then the byte value terminates the byte sequence,
results in a valid mapping to a Unicode BMP character, and the state number is
reset to 0.
The next state can be explicitly specified with a separating
colon (:) followed by the number of the state (=number/index of the row,
starting at 0). This is mostly used for intermediate byte values, i.e., for
bytes that are not the last ones in a sequence. The state machine needs to
proceed to the next state and read another byte. In this case, no other action
is specified.
If the byte value(s) terminate(s) a byte sequence, then the byte
sequence results in the following depending on the action that is announced with
a period (.) followed by a letter:
u - Unassigned. The byte sequence is valid but does not encode a character.p - surrogate Pair. The byte sequence is valid and may result ini - Illegal. The byte sequence is illegal. This is the default for
all byte values in a row that are not otherwise specified with
column entries.s - State change only. The byte sequence does not encode any character
but may change the state number. This could be used with simple, stateful
encodings (using, for example, SI/SO codes),
but ICU currently does not take advantage of it.:0.).
If a byte value is not specified in any column entry of a row, then it is illegal in the current state. If a byte value is specified in more than one column entry of the same row, then the last one is used. This allows to specify common properties for a wide byte value range followed by a few exceptions and is easier than having to specify mutually exclusive ranges, especially if many of them have the same properties.
The optional keyword at the beginning of a state line has the following effect:
initial: The state machine can start reading byte sequences
in this state. State 0 is always an initial state. Only initial states can be
next states for final byte values. In an initial state, the Unicode mappings
for all final bytes are also stored directly in the state table.surrogates: All Unicode mappings for final bytes in non-initial
states are stored in a separate table of 16-bit Unicode (UTF-16) code units.
Since most legacy codepages map only to Unicode code points up to U+ffff
(the Basic Multilingual Plane, BMP), the default allocation per mapping
result is one 16-bit unit. Individual byte values can be specified to map
to surrogate pairs (= two 16-bit units) with action letter p.
The surrogates keyword specifies this for the entire state (row).
Surrogate pair mapping entries can still hold single units depending on the
actual mapping data, but single-unit mapping entries cannot hold a pair of units.
Mapping to single-unit entries is the default because the mapping is faster,
uses half as much memory in the code units table, and is sufficient for most
legacy codepages.When converting to Unicode, the state machine starts in state number 0. In each iteration, it reads one input (codepage) byte and either just goes to the next state as specified, or treats it as a final byte with the specified action and an optional non-0 next (initial) state. This means that a state table needs to have at least as many state rows as the maximum number of bytes per character, which is the maximum length of any byte sequence.
0-7f
This single-row state table describes US-ASCII.
Byte values from 0 to 0x7f are valid and map to Unicode character up to U+ffff.
Byte values from 0x80 to 0xff are illegal.
0-7f, 81-9f:1, a0-df, e0-fc:1
40-7e, 80-fc
This two-row state table describes the structure of Shift-JIS, which encodes some characters
with one byte each, and others with two bytes each.
Bytes 0 to 0x7f and 0xa0 to 0xdf are valid single-byte encodings.
Bytes 0x81 to 0x9f and 0xe0 to 0xfc are lead bytes, i.e., they are followed by one of
the bytes that are specified as valid in state 1.
A byte sequence of 0x85 0x61 is valid, while a single byte of 0x80 or 0xff is illegal.
Similarly, a byte sequence of 0x85 0x31 is illegal.
0-8d, 8e:2, 8f:3, 90-9f, a1-fe:1
a1-fe
a1-e4
a1-fe:1, a1:4, a3-af:4, b6:4, d6:4, da-db:4, ed-f2:4
a1-fe.u
This fairly complicated state table describes EUC-JP.
Valid byte sequences are one, two, or three bytes long.
Two-byte sequences have lead byte 0x8e and end in state 2, or
lead bytes 0xa1 to 0xfe and end in state 1.
Three-byte sequences have a lead byte of 0x8f and continue in state 3.
Some final byte value ranges are entirely unassigned, therefore they end in state 4
with an action letter of u for "unassigned" to save significant memory
for the code units table.
Assigned three-byte sequences end in state 1 like most two-byte sequences.