Version: Oct 13, 2000
Copyright © 1997-2000 International Business Machines Corporation
and others. All Rights Reserved.
Today's software market is a global one in which it is desirable to develop and maintain one application that supports a wide variety of national languages. International Components for Unicode provides the following tools to help you write language independent applications:
It is possible to support additional locales by adding more locale data files, with no code changes.
Please refer to POSIX programmer's Guide for details on what the ISO locale ID means.
Your comments are important to making this release successful. We are committed to fixing any bugs, and will also use your feedback to help plan future releases.
IMPORTANT: Please make sure you understand the Copyright and License information.
There are two ways to download the ICU releases,
For more details on how to download ICU directly from the web site, please also see http:/oss.software.ibm.com/icu/download/index.html
Below, $Root is the placement of the icu directory in your file system, like "drive:\...\icu" in your environment. "drive:\..." stands for any drive and any directory on that drive that you chose to install icu into.
|
readme.html |
Describes the International Components for Unicode (this file) |
|
license.html |
Contains IBM's public license |
|
$Root/docs |
API documentation for the International Components for Unicode |
|
$Root/source/common/ |
The utility classes, such as ResourceBundle, Unicode, Locale, UnicodeString. The codepage conversion library API, UnicodeConverter. |
|
$Root/source/i18n/ |
The collation source files, Collator, RuleBasedCollator and
CollationKey. |
|
$Root/source/test/intltest/ |
A test suite including all C++ APIs. For information about running the test suite, see docs/intltest.html. |
|
$Root/source/test/cintltst/ |
A test suite including all C APIs. For information about running the test suite, see docs/cintltst.html. |
|
$Root/data/ |
The Unicode 3.0 data file. Please see http://www.unicode.org/ for more
information.
|
|
$Root/source/tools |
Tools for generating the data files. Data files are generated by invoking $Root/source/tools/makedata.bat on Win32 or $Root/source/make install on Unix. |
|
$Root/source/samples |
Various sample programs that use ICU |
|
$Root/include/ |
contains all the public header files. |
|
$output |
contains the libraries for static/dynamic linking or executable programs. |
The following shows the main directory structure of the International Components for Unicode
In the International Components for Unicode, there are two categories:
See International Components for Unicode Coding Guidelines for a discussion of code conventions common to all library classes.
See also ../icuhtml/aindex.html
for an alphabetical index, and ../icuhtml/HIER.html for a hierarchical index
to detailed API documentation.
The platform dependencies have been isolated into the following 4 files:
As it was previously mentioned and proposed on ICU list, ICU libraries on Win32 are now renamed and relocated. The following changes took place:
Also, ctestfw and ex toolutil (now icutu17) libraries appeared in icu\bin and icu\lib dirs, but this shouldn't concern regular users of icu.
When you first try to compile your programs with new version of icu, compilation will fail. The following steps are required:
When a new, binary incompatible version appears after 1.7, the libraries will change the version number, so you will have to rename libraries in your projects again. However, this way you can have several versions of icu on the same machine and change the libraries for different programs without having to change path settings on your machine.
If you are building on the Win32 platform, it is important that you understand a few build details:
DLL directories and the PATH setting: As delivered, the International Components for Unicode build as several DLLs. These DLLs are placed in the directories "icu\bin\Debug" and "icu\bin\Release". You must add either of these directories to the PATH environment variable in your system, or any executables you build will not be able to access International Components for Unicode libraries. Alternatively, you can copy the DLL files into a directory already in your PATH, but we do not recommend this. You can wind up with multiple copies of the DLL and wind up using the wrong one.
To change your PATH: When you are not using the debug version, you will want to change the "Debug" part of the path to "Release" instead (the $Root is the root ICU installation directory e.g. drive:\installation-directory\icu).
Link with Runtime libraries: All the DLLs link with the C
runtime library "Debug Multithreaded DLL" or "Multithreaded DLL." (This
is changed through the Project Settings dialog, on the C/C++ tab, under
Code Generation.) It is important that any executable or other DLL you
build which uses the International Components for Unicode DLLs links with
these runtime libraries as well. If you do not do this, you will
seemingly get memory errors when you run the executable.
If you are building on the OS/390 UNIX System Services platform, it is important that you understand a few details:
iconv -f IBM-1047 -t ISO8859-1 uni-text.txt >
uni-text.txt
OS/390 supports both native S/390 hexadecimal floating point and,
with Version 2.6 and later, IEEE binary floating point. This is a
compile time option. Applications built with IEEE should use ICU dlls
that are built with IEEE (and vice versa). The environment variable
IEEE390=1 will cause the OS/390 version of ICU to be built with IEEE
floating point. The default is native hexadecimal floating point.
Important: Currently (ICU 1.4.2), native floating point
support is sufficient for codepage conversion, resource bundle and
UnicodeString operations, but the Format APIs, especially
ChoiceFormat, require IEEE binary floating point.
Examples for configuring ICU:
Debug build: IEEE390=1 ./configure
Release build: CFLAGS=-2 IEEE390=1 ./configure
ICU Reference Release 1.4.0 contains partial support for the 400 platform, but additional work by the user is currently needed to get it to build completely. A future release of the ICU should work out-of-the-box under OS/400.
Building International Components for Unicode requires:
The steps are:
Note: To set the active configuration, two different possibilities are:
It is also possible to build each library individually, using the
workspaces in each respective directory. They have to be built in the
following order:
There is a set of Makefiles for Unix that supports Linux w/gcc, Solaris w/gcc and Workshop CC, AIX w/xlc and OS/390 with C++.
Building International Components for Unicode on Unix requires:
A UNIX C++ compiler, (gcc, cc, xlc_r, etc...) installed on the target machine. A recent version of GNU make (3.7+). OS/390 gnu utilities for both make (gmake) and zip (gzip/gunzip) can be found at the MKS web site at http://www.mks.com. Please do a search on "os/390".
The steps are:
Regarding the test suite, please read the directions in docs/intltest.html and docs/cintltst.html.
Some platforms use package management tools to control the installation and uninstallation of files on the system, as well as the integrity of the system configuration. You may want to check if ICU can be packaged for your package management tools by looking into the "packaging" directory. (Please note that if you are using a snapshot of ICU from CVS, it is probable that the packaging scripts or related files are not up to date with the contents of ICU at this time, so use them with caution.)
It is also possible to build each library individually, using the Makefiles in each respective directory. They have to be built in the following order:
HP/UX has a documented characteristic where the shl_unload() function always unloads a library, regardless of how many times the library has been loaded. Most operating systems reference-count libraries as they are opened. In the future (Jitterbug 414) this may be corrected in the ICU, but at present we work around this problem by simply NOT ever unloading shared libraries. This means that once a data library is loaded (ex: libicudata.sl) by a process, it cannot be unloaded and replaced without stopping and restarting the process.
To add locale data files to International Components for Unicode do the following:
Adding resource bundle data to your application is simple. Just create the resource bundle files with the right format and names in your application directory tree. For more information on the resource bundle file format see the resource bundle documentation or the User's Guide).
Note: resource bundle tag names should contain only
invariant 7-bit ASCII characters (e.g. ones from the following set:
A-Z, a-z, 0-9, <SP>, ", %, &, `, (, ), *, +, ,, -, ., /,
:, ;, <, =, >, ?, _). Use that same directory name (absolute
path) when instantiating a resource bundle at run time.
Collation data is stored as a binary object inside of the corresponding resource. The source for the tailorings is stored in a corresponding ASCII text file (resource bundle source) indicated by a "CollationElements" tag . For instance, the data for de_CH is stored with a tag "CollationElements" in a file named "de_CH.txt". (The data is stored in a tag named '%%CollationElements'). Reading the collation data from these files can be time-consuming, especially for large pieces of data that occur in languages such as Japanese. For this reason, the Collation Framework implements a second file format, a performance-optimized, non-portable, binary format. These binary formats are generated automatically by the framework when genrb is run on the specific locales.
The charset conversion library provides ways to convert simple text strings (e.g., char*) such as ISO 8859-1 to and from Unicode. The objective is to provide clean, simple, reliable, portable and adaptable data structures and algorithms to support the International Components for Unicode's character codeset Conversion APIs. The conversion data in the library originated from the NLTC lab in IBM. The IBM character set conversion tables are publicly available in the published IBM document called "CHARACTER DATA REPRESENTATION ARCHITECTURE - REFERENCE AND REGISTRY". The character set conversion library includes single-byte, double-byte and some UCS encodings to and from Unicode. This document can be ordered through Mechanicsberg and it comes with 2 CD ROMs which have machine-readable conversion tables on them. The license agreement is included in International Components for Unicode agreement.
Click here to view converters implemented in ICU. To see converters in action, please visit http://oss.software.ibm.com/developerworks/opensource/icu/localeexplorer/?converter&
To order the document in the US you can call 1-800-879-2755 and request document number SC09-2190-00. The cost of this publication is $75.00 US not including tax.
ICU supports extensive versioning of its code and data. Versioning allows clients to determine when parts of ICU change, and what the effect of the change is.
ICU as a whole has a version number. ICU components such as Collator have their own distinct version numbers. Each resource bundle, including all the locale data resource bundles, has its own version number. Individual tagged items within a resource bundle have their own version numbers.
All version numbers are in the form of a UVersionInfo structure, which is an array of four unsigned bytes. These bytes are:
UVersionNumber structures can be converted to and from string representations as dotted integers, such as "1.4.5.0", using the u_versionToString() and u_stringToVersion() functions.
Version numbers monotonically increase as changes are made. Two UVersionInfo structure may be compared using binary comparison (memcmp) to see which is larger (newer). It only makes sense to compare the same flavor of version number; you cannot compare the ICU version number to the Collator version number, for instance.
The interpretation of version numbers depends on what is being described.
0 (Major): Reference release with major feature addition or change.
1 (Minor): Reference release without major feature addition.
2 (Milli): Maintenance update to the reference releases.
3 (Patch): Enhancement/patch update.
0 (Major): Breaking change. Results and data generated by the new version are incompatible with those generated by the preceding version. Example: In ICU 1.5, the implementation of ResourceBundle changed drastically. The data structure, algorithm for parsing data, and so on are completely different in 1.5. This required an increment of the major version number.
1 (Minor): Backward-compatible change. The new version of the code can read or use data generated by the old version, but the old version cannot read or use data generated by the new version. Example: The delimiter in the CollationKey gets changed from 0x0000 to 0xFFFF. The algorithm keeps track of the differences and recognize these two different formats before and after a particular release.
2 (Milli): Compatible change. Results and data generated by the new version are compatible with those generated by the preceding version. Example: A byte in the reserved bytes in the data structure is now used as a flag/bitmask or whatever, e.g. UDataInfo. The size of the data structure is changed and new code is added to check for this flag. No other changes are made.
3 (Patch): Enhancement. A minor change. Example: Performance enhancements applied to the code but no changes other than that.
0 (Major): Incompatible format change. The layout or format of the data has changed. For example, an additional array element has been added, or an additional tag. Example: ICU 1.6 changes the element layout in "CollationElements". We changed this from a tag with plain string value to a tagged array with 3 new subtags, "Version", "Override" and "Sequence". This change is incompatible with pre-1.6 code and data.
1 (Minor): Backward-compatible format change. A change that can be read and used by previous versions of ICU, but that adds data used by newer versions. Example: We added a new tag called "Author" to the data file. The only difference between the previous version of the data files and the current version is this tag.
2 (Milli): Compatible change. A change to the data without modification of the format. Example: We updated the value of a tag "LocaleID" from "041C" to "3801". No other changes were made.
3 (Patch): Enhancement. A minor change. Example: We changed the comments in the data file, perhaps the copyright notices.
The data stored in resource bundles is tagged with version numbers. A resource bundle can contain a tagged string named "Version" that declares the version number in dotted-integer format. Example:
en {
Version { "1.0.3.5" }
...
}
A resource bundle may omit the "Version" element, in which case it will inherit one along the usual chain. Example: If the resource bundle en_US contained no "Version" element, it would inherit "1.0.3.5" from en.
If inheritance passes all the way to the root resource bundle and it contains no "Version" resource, then the default version number 1.0.0.0 is returned.
Elements within a resource bundle may also contain version numbers, for example:
be {
CollationElements {
Version { "1.0.0.0" }
...
}
}
Here the CollationElements data is version 1.0.0.0. This version may differ from the version of the enclosing bundle.
If a resource element lacks a "Version" element, then it inherits the "Version" element of its enclosing resource bundle. (This is a special case; in general, resource bundle elements do not inherit data from enclosing structures.) Example:
en {
Version { "1.0.3.5" }
...
}
en_US {
CollationElements {
...(contains no "Version" element)
}
}
Here, the version of the CollationElements in en_US is 1.0.3.5. It inherits the en_US version, which is inherited from en.
Note: The API and code to fully support the mechanism described above is not in place yet as of ICU 1.6. See Future Enhancements below.
Currently, the following components are versioned.
u_getVersion().ures_getVersion() and
ResourceBundle::getVersion(). This is a data version
number for the bundle as a whole.u_getUnicodeVersion() and
Unicode::getUnicodeVersion(). This version reflects the
numbering of the Unicode releases; see http://www.unicode.org.Collator::getVersion(). This is a code version number for
the collation code and algorithm.In order for the code to be portable, only a subset of the C++ language that will compile correctly on even the oldest of C++ compilers (and also to provide a usable C interface) can be used in the implementation, which means that there's no use the C++ exception mechanism in the code.
After considering many alternatives, the decision was that every function that can fail takes an error-code parameter by reference. This is always the last parameter in the function’s parameter list. The ErrorCode type is defined as a enumerated type. Zero represents no error, positive values represent errors, and negative values represent non-error status codes. Macros were provided, SUCCESS and FAILURE, to check the error code.
The ErrorCode parameter is an input-output parameter. Every function tests the error code before doing anything else, and immediately exits if it’s a FAILURE error code. If the function fails later on, it sets the error code appropriately and exits without doing any other work (except, of course, any cleanup it has to do). If the function encounters a non-error condition it wants to signal (such as "encountered an unmapped character" in transcoding), it sets the error code appropriately and continues. Otherwise, the function leaves the error code unchanged.
Generally, only functions that don’t take an ErrorCode parameter, but call functions that do, have to declare one. Almost all functions that take an ErrorCode parameter and also call other functions that do merely have to propagate the error code they were passed down to the functions they call. Functions that declare a new ErrorCode parameter must initialize it to ZERO_ERROR before calling any other functions.
The rationale here is to allow a function to call several functions (that take error codes) in a row without having to check the error code after each one. [A function usually will have to check the error code before doing any other processing, however, since it is supposed to stop immediately after receiving an error code.] Propagating the error-code parameter down the call chain saves the programmer from having to declare one everywhere, and also allows us to more closely mimic the C++ exception protocol.
Function names. If a function is identical (or almost identical) to an ANSI or POSIX function, we give it the same name and (as much as possible) the same parameter list. A "u" is prepended onto the beginning of the name.
For functions that exist prior to version 1.2.1, that the function name should begin with a lower-case "u". After the "u" is a short code identifying the subsystem it belongs to (e.g., "loc", "rb", "cnv", "coll", etc.). This code is separated from the actual function name by an underscore, and the actual function name can be anything. For example,
UChar* uloc_getLanguage(...); void uloc_setDefaultLocale(...); UChar* ures_getString(...);
Struct and enum type names. For structs and enum types, the rule is that their names begin with a capital "U." There is no underscore for struct names.
UResourceBundle; UCollator; UCollationResult;
Enum value names. Enumeration values have names that begin with "UXXX" where XXX stands for the name of the functional category.
UNUM_DECIMAL; UCOL_GREATER;
Macro names. Macro names are in all caps, but there are currently no other requirements.
Constant names. Many constant names (constants defined with "const", not macros defined with "#define" that are used as constants) begin with a lowercase k, but this isn’t universally enforced.
In ICU's C APIs, the user needs to adhere to the following principles for consistency across all functional categories:
int32_t parameter
that gives the size of the buffer.To find out how large the result buffer should be, ICU provides a preflighting C interface. The interface works like this:
NULL pointer for the buffer pointer,
and the function returns the actual size of the result. You can then
choose to allocate a buffer of the correct size and re-run the
operation if you would like to.The following three options demonstrates how to use the preflighting interface,
/**
* @param result is a pointer to where the actual result will be.
* @param maxResultSize is the number of characters the
* buffer pointed to be result has room for.
* @return The actual length of the result including the
* terminating <code>NULL</code>.
*/
int32_t doSomething( /* input parameters */,
UChar* result,
int32_t maxResultSize,
UErrorCode* err);
In this sample, if the actual result doesn’t fit in the space
available in maxResultSize, this function returns the amount
of space necessary to hold the result, and result holds as many
characters of the actual result as possible. If you don’t care
about this, no further action is necessary. If you do care about
the truncated characters, you can then allocate a buffer on the heap of
the size specified by the return value and call the function again,
passing that buffer’s address for result.
All preflighting functions have a fill-in ErrorCode
parameter (and follow the normal ErrorCode rules), even if
they are not currently doing so. Buffer overflow would be treated as a
FAILURE error condition, but would not be reported when the caller
passes in NULL for actualResultSize
(presumably, a NULL for this parameter means the client
doesn’t care if he got a buffer overflow). All other failing error
conditions will overwrite the "buffer overflow" error, e.g.
MISSING_RESOURCE_ERROR etc..
Returning an array of strings is fairly easy in C++, but very hard in C. Instead of returning the array pointer directly, we opted for an iterative interface instead: split the function into two functions. One returns the number of elements in the array, and the other one returns a single specified element from the array.
int32_t countArrayItems(/* parameters */);
int32_t getArrayElement(int32_t elementIndex,
/* other parameters */,
UChar* result,
int32_t maxResultSize,
UErrorCode* err);
In this case, iterating across all the elements in the array would amount to a call to the count() function followed by multiple calls to the getElement() function.
UChar element[50];
for (i = 0; i < countArrayItems(...); i++) {
getArrayItem(i, ..., element, 50, &err);
/* do something with element */
}
In the case of the resource bundle ures_XXXX functions
returning 2-dimensional arrays, the getElement() function takes both x
and y coordinates for the desired element, and the count() function
returns the number of arrays (x axis). Since the size of each array
element in the resource 2-D arrays should always be the same, this
provides an easy-to-use C interface.
void countArrayItems(int32_t* rows,
int32_t* columns,
/* other parameters */);
int32_t get2dArrayElement(int32_t rowIndex,
int32_t colIndex,
/* other parameters */,
UChar* result,
int32_t maxResultSize,
UErrorCode* err);
We have decided to make a semantic change to the conversion API which
affects applications using ICU that are migrated to use ICU version 1.6
compared to earlier ICU versions:
The error code that is set from streaming conversion like
ucnv_fromUnicode() - ucnv_toUnicode() ucnv_fromUChars() - ucnv_toUChars() scsu_compress() - scsu_decompress()when the target buffer is full but the source not empty is changed from
U_INDEX_OUTOFBOUNDS_ERROR to
U_BUFFER_OVERFLOW_ERROR. This change makes the error codes
more consistent with their names and with their use in other icu
APIs.You need to test for this new error code if your code uses ICU for
conversion and used the old error code. ucnv.h and scsu.h are updated
with this information. Please search in your source code for
U_INDEX_OUTOFBOUNDS_ERROR. If it is used with the above
functions (not with ucnv_getNextUChar()), then you
need to change it to U_BUFFER_OVERFLOW_ERROR in order to get
your code to work with icu 1.6.
See the updated sample code in icu/source/samples. All
samples are updated. See
jitterbug 516 for details. This was discussed in july 2000 on the icu
mailing list. Please see the list archive for the discussion.
http://oss.software.ibm.com/icu/ is a pointer to general information about the International Components for Unicode.
docs/udata.html is a raw draft of ICU data handling.
icuhtml/aindex.html is an
alphabetical index to detailed API documentation.
icuhtml/HIER.html is a hierarchical
index to detailed API documentation.
docs/collate.html is an overview to Collation.
docs/BreakIterator.html is a diagram showing how BreakIterator processes text elements.
http://www.ibm.com/developer/unicode/
is a pointer to information on how to make applications global.
To submit comments, request features and report bugs, please contact us. While we are not able to respond individually to each comment, we do review all comments. Send Internet email to icu@oss.software.ibm.com.
Copyright © 1997-2000 International Business Machines Corporation
and others. All Rights Reserved.
IBM Center for Emerging Technologies Silicon Valley,
10275 N De Anza Blvd., Cupertino, CA 95014
All rights reserved.