Version: May 30, 2000
COPYRIGHT:
Copyright (c) 1997-2000 International Business Machines Corporation and others.
All Rights Reserved.
Today's software market is a global one in which it is desirable to develop and maintain one application that supports a wide variety of national languages. International Components for Unicode provides the following tools to help you write language independent applications:
It is possible to support additional locales by adding more locale data files, with no code changes.
Please refer to POSIX programmer's Guide for details on what the ISO locale ID means.
Your comments are important to making this release successful. We are committed to fixing any bugs, and will also use your feedback to help plan future releases.
IMPORTANT: Please make sure you understand the Copyright and License information.
There are two ways to download the ICU releases,
For more details on how to download ICU directly from the web site, please also see http:/oss.software.ibm.com/icu/download/index.html
Below, $Root is the placement of the icu directory in your file system, like "drive:\...\icu" in your environment. "drive:\..." stands for any drive and any directory on that drive that you chose to install icu into.
The following files describe the code drop:
|
readme.html (this file) |
describes the International Components for Unicode |
|
license.html |
contains IBM's public license |
The following directories contain source code and data files:
|
$Root\source\common\ |
The utility classes, such as ResourceBundle, Unicode, Locale, UnicodeString. The codepage conversion library API, UnicodeConverter. |
|
$Root\source\i18n\ |
The collation source files, Collator, RuleBasedCollator
and CollationKey. |
|
$Root\source\test\intltest\ |
A test suite including all C++ APIs. For information about running the test suite, see docs/intltest.html. |
|
$Root/source/test/cintltst/ |
A test suite including all C APIs. For information about running the test suite, see docs/cintltst.html. |
|
$Root/data/ |
The Unicode 3.0 data file. Please see http://www.unicode.org/ for more
information.
|
|
$Root/source/tools |
Tools for generating the data files. Data files are generated by invoking $Root/source/tools/makedata.bat on Win32 or $Root/source/make install on Unix. |
|
$Root/source/samples |
Various sample programs that use ICU |
The following directories are populated when you've built the
framework:
(on Unix, replace $Root with the value given to the file
"configure")
|
$Root/include/ |
contains all the public header files. |
|
$output |
contains the libraries for static/dynamic linking or executable programs. |
The following diagram shows the main directory structure of the International Components for Unicode:
icu-NNNN
|
output icu
_____|_____ ______________|______________________________
| | | | | | |
libraries programs include data source | |
(built) (built) (built) | readme.html license.html
|
_________________|__________________________
| | | | | |
common i18n test extra tools samples
| |
___|___ ___|_________________
| | | | | |
intltest cintltst makeconv ctestfw genrb ....
In the International Components for Unicode, there are two categories:
See International Components for Unicode Coding Guidelines for a discussion of code conventions common to all library classes.
See also html/aindex.html for an alphabetical
index, and html/HIERjava.html for a
hierarchical index to detailed API documentation.
The platform dependencies have been isolated into the following 4 files:
· XP_CPLUSPLUS is defined for C++
· bool_t, TRUE and FALSE, int8_t, int16_t etc.
· U_EXPORT and U_IMPORT for specifying dynamic library import and export
· icu_isNaN, icu_isInfinite(double), icu_getNaN(); icu_getInfinity for handling special floating point values
· icu_tzset, icu_timezone, icu_tzname and time for reading platform specific time and timezone information
· icu_getDefaultDataDirectory, icu_getDefaultLocaleID for reading the locale setting and data directory
· icu_isBigEndian for finding the endianess of the platform
· icu_nextDouble is used specifically by the ChoiceFormat API.
Win32 Platform
If you are building on the Win32 platform, it is important that you understand a few build details:
DLL directories and the PATH setting: As delivered, the International Components for Unicode build as several DLLs. These DLLs are placed in the directories "icu\bin\Debug" and "icu\bin\Release". You must add either of these directories to the PATH environment variable in your system, or any executables you build will not be able to access International Components for Unicode libraries. Alternatively, you can copy the DLL files into a directory already in your PATH, but we do not recommend this -- you can wind up with multiple copies of the DLL, and wind up using the wrong one.
To change your PATH: Do this under NT by using the System control panel. Pick the "Environment" tab, select the variable PATH in the lower box. In the "value" box, append the string ";drive:\...\icu\bin\Debug" at the end of the path string. If there is nothing there, just type in "drive:\...\icu\bin\Debug". Click the Set button, then the Ok button.
Link with Runtime libraries: All the DLLs link with the C runtime
library "Debug Multithreaded DLL" or "Multithreaded DLL."
(This is changed through the Project Settings dialog, on the C/C++ tab, under
Code Generation.) It is important that any executable or other DLL you build
which uses the International Components for Unicode DLLs links with these
runtime libraries as well. If you do not do this, you will seemingly get memory
errors when you run the executable.
If you are building on the OS/390 UNIX System Services platform, it is important that you understand a few details:
iconv -f IBM-1047 -t ISO8859-1 uni-text.txt > uni-text.txtOS/390 supports both native S/390 hexadecimal floating point and, with Version
2.6 and later, IEEE binary
floating point. This is a compile time option. Applications built with IEEE
should use ICU dlls that are
built with IEEE (and vice versa). The environment variable IEEE390=1 will cause
the OS/390 version
of ICU to be built with IEEE floating point. The default is native hexadecimal
floating point.
Important: Currently (ICU 1.4.2), native floating point support
is sufficient for codepage conversion, resource bundle and UnicodeString operations,
but the Format APIs, especially ChoiceFormat, require IEEE binary floating point.
Examples for configuring ICU:
Debug build: IEEE390=1 ./configure
Release build: CFLAGS=-2 IEEE390=1 ./configure
Building International Components for Unicode requires:
The steps are:
Note: To set the active configuration, two different possibilities are:
It is also possible to build each library individually, using the workspaces
in each respective directory. They have to be built in the following order:
1. common
2. i18n
3. makedata (which invokes makeconv,
genrb, gencol, genccode etc.)
4. ctestfw
5. intltest and cintltst, if you
want to run the test suite.
Regarding the test suite, please read the directions in docs/intltest.html and docs/cintltst.html
There is a set of Makefiles for Unix that supports Linux w/gcc, Solaris w/gcc and Workshop CC, AIX w/xlc and OS/390 with C++.
Building International Components for Unicode on Unix requires:
A UNIX C++ compiler, (gcc, cc, xlc_r, etc...) installed on the target machine. A recent version of GNU make (3.7+). OS/390 gnu utilities for both make (gmake) and zip (gzip/gunzip) can be found at the MKS web site at http://www.mks.com. Please do a search on "os/390".
The steps are:
It is also possible to build each library individually, using the Makefiles
in each respective directory. They have to be built in the following order:
1. common
2. i18n
3. makeconv
4. genrb
5. gencol
6. gentz
7. genccode
8. ctestfw
9. intltest and cintltst, if you
want to run the test suite.
Regarding the test suite, please read the directions in docs/intltest.html and docs/cintltst.html
To add locale data files to International Components for Unicode do the
following: 1. Create a file containing the
key-value pairs which value you are overriding from the parent locale data
file. 2. Name the file with locale ID
you are creating with a ".txt" at the end. e.g.
fr_BF.txt 3. Add the name of that file
(without the ".txt" extension) as a single line in
"index.txt" file in the default locale directory (icu/data/). 4. Regenerate the data DLL
file. Please see "How to Install"
section for more details on how to verify the ICU release. How to add
resource bundle data to your application Adding resource bundle data to your application is quite simple: Create resource bundle files with the right format and names in a directory
for resource bundles you create in your application directory tree.(for more
information of that format of these files see resource bundle documentation
or resource
bundle format). Collation data is stored in a single directory on a local disk. Each
locale's data is stored in a corresponding ASCII text file indicated by a
"CollationElements" tag . For instance, the data for de_CH is stored
with a tag "CollationElements" in a file named "de_CH.txt".
Reading the collation data from these files can be time-consuming, especially
for large pieces of data that occur in languages such as Japanese. For this
reason, the Collation Framework implements a second file format, a
performance-optimized, non-portable, binary format. These binary files are
generated automatically by the framework the first time a collation table is
parsed. They have names of the form "de_CH.col". Once the files are
generated by the framework, future loading of those collations occur from the
binary file, rather than the text file, at much higher speed. In general, you don't have to do anything special with these files. They can
be generated directly by using the "gencol" tool. In addition,
they can also be generated and used automatically by the framework, without
intervention on your part. However, there are situations in which you will have
to regenerate them. To do so, you must manually delete the ".col"
files from your collation data directory and re-run the gencol tool. You will need to regenerate your ".col" files in the following
circumstances: The charset conversion library provides ways to convert simple text strings
(e.g., char*) such as ISO 8859-1 to and from Unicode. The objective is to
provide clean, simple, reliable, portable and adaptable data structures and
algorithms to support the International Components for Unicode's character
codeset Conversion APIs. The conversion data in the library originated from the
NLTC lab in IBM. The IBM character set conversion tables are publicly available
in the published IBM document called "CHARACTER DATA REPRESENTATION
ARCHITECTURE - REFERENCE AND REGISTRY". The character set conversion
library includes single-byte, double-byte and some UCS encodings to and from Unicode.
This document can be ordered through Mechanicsberg and it comes with 2 CD ROMs
which have machine-readable conversion tables on them. The license agreement is
included in International Components for Unicode agreement. Click here to view converters implemented in
ICU. To see converters in action, please visit
http://oss.software.ibm.com/developerworks/opensource/icu/localeexplorer/?converter& To order the document in the US you can call 1-800-879-2755 and request
document number SC09-2190-00. The cost of this publication is $75.00 US not
including tax. In order for the code to be portable, only a subset of the C++ language that
will compile correctly on even the oldest of C++ compilers (and also to provide
a usable C interface) can be used in the implementation, which means that
there's no use the C++ exception mechanism in the code. After considering many alternatives, the decision was that every function
that can fail takes an error-code parameter by reference. This is always the
last parameter in the function’s parameter list. The ErrorCode type is defined
as a enumerated type. Zero represents no error, positive values represent
errors, and negative values represent non-error status codes. Macros were
provided, SUCCESS and FAILURE, to check the error code. The ErrorCode parameter is an input-output parameter. Every function tests
the error code before doing anything else, and immediately exits if it’s a
FAILURE error code. If the function fails later on, it sets the error code
appropriately and exits without doing any other work (except, of course, any
cleanup it has to do). If the function encounters a non-error condition it
wants to signal (such as "encountered an unmapped character" in
transcoding), it sets the error code appropriately and continues. Otherwise,
the function leaves the error code unchanged. Generally, only functions that don’t take an ErrorCode parameter, but call
functions that do, have to declare one. Almost all functions that take an
ErrorCode parameter and also call other functions that do merely have to
propagate the error code they were passed down to the functions they call.
Functions that declare a new ErrorCode parameter must initialize it to
ZERO_ERROR before calling any other functions. The rationale here is to allow a function to call several functions (that
take error codes) in a row without having to check the error code after each
one. [A function usually will have to check the error code before doing any
other processing, however, since it is supposed to stop immediately after
receiving an error code.] Propagating the error-code parameter down the call
chain saves the programmer from having to declare one everywhere, and also
allows us to more closely mimic the C++ exception protocol. Function names. If a function is identical (or almost identical) to
an ANSI or POSIX function, we give it the same name and (as much as possible)
the same parameter list. A "u" is prepended onto the beginning of the
name. For functions that exist prior to version 1.2.1, that the function name
should begin with a lower-case "u". After the "u" is a
short code identifying the subsystem it belongs to (e.g., "loc",
"rb", "cnv", "coll", etc.). This code is
separated from the actual function name by an underscore, and the actual
function name can be anything. For example, Struct and enum type names. For structs and enum types, the rule is
that their names begin with a capital "U." There is no underscore for
struct names. Enum value names. Enumeration values have names that begin with
"UXXX" where XXX stands for the name of the functional category. Macro names. Macro names are in all caps, but there are currently no
other requirements. Constant names. Many constant names (constants defined with
"const", not macros defined with "#define" that are used as
constants) begin with a lowercase k, but this isn’t universally enforced. In ICU's C APIs, the user needs to adhere to the following principles for
consistency across all functional categories: To find out how large the result buffer should be, ICU provides a preflighting
C interface. The interface works like this: The following three options demonstrates how to use the preflighting
interface, In this sample, if the actual result doesn’t fit in the space available in maxResultSize, this
function returns the amount of space necessary to hold the result, and result
holds as many characters of the actual result as possible. If you don’t care
about this, no further action is necessary. If you do care about the
truncated characters, you can then allocate a buffer on the heap of the size
specified by the return value and call the function again, passing that buffer’s
address for result. All preflighting functions have a fill-in ErrorCode parameter (and follow the normal ErrorCode rules),
even if they are not currently doing so. Buffer overflow would be treated as a
FAILURE error condition, but would not be reported when the caller
passes in NULL for actualResultSize
(presumably, a NULL for this parameter means the client doesn’t care if he got
a buffer overflow). All other failing error conditions will overwrite the
"buffer overflow" error, e.g. MISSING_RESOURCE_ERROR
etc.. Returning an array of strings is fairly easy in C++, but very hard in C.
Instead of returning the array pointer directly, we opted for an iterative
interface instead: split the function into two functions. One returns the
number of elements in the array, and the other one returns a single specified
element from the array. In this case, iterating across all the elements in the array would amount to
a call to the count() function followed by multiple calls to the getElement()
function. In the case of the resource bundle ures_XXXX
functions returning 2-dimensional arrays, the getElement() function takes both
x and y coordinates for the desired element, and the count() function returns
the number of arrays (x axis). Since the size of each array element in
the resource 2-D arrays should always be the same, this provides an easy-to-use
C interface. We have decided to make a semantic change to the conversion API
which affects applications using ICU that are migrated to use ICU version 1.6 compared to earlier ICU versions:Note about HP/UX data libraries:
How ICU handles data
How to add a locale data file
Make sure the filename is the locale ID with the extension
".txt". We recommend you copy parent file and change the values
that need to be changed, remove all other key-pairs. Be sure
to update the locale ID key (the outmost brace) with
the name of the locale id your a creating.
Would create a locale that inherits all the key-value pairs from fr.txt.
Please note that resource bundle tag names should contain only invariant 7-bit
ASCII characters (e.g. ones from the following set: A-Z, a-z, 0-9, <SP>,
", %, &, `, (, ), *, +, ,, -, ., /, :, ;, <, =, >, ?, _).
Use that same directory name (absolute path) when instantiating a resource
bundle at run time.Where Collation Data is stored
Character Set Conversion Information
Programming Notes
Reporting Errors
C Function and Data Type Naming
UChar* uloc_getLanguage(...);void uloc_setDefaultLocale(...);UChar* ures_getString(...);
UResourceBundle; UCollator; UCollationResult;
UNUM_DECIMAL;UCOL_GREATER;
Preflighting and Overflow Handling
/** * @param result is a pointer to where the actual result will be. * @param maxResultSize is the number of characters the buffer pointed to be result has room for. * @return The actual length of the result (counting the terminating null) */int32_t doSomething( /* input params */, UChar* result, int32_t maxResultSize, UErrorCode* err);
Arrays as return types
int32_t countArrayItems(/* params */);int32_t getArrayElement(int32_t elementIndex, /* other params */, UChar* result, int32_t maxResultSize, UErrorCode* err);
for (i = 0; i < countArrayItems(...); i++) { UChar element[50]; getArrayItem(i, ..., element, 50, &err); /* do something with element */}
void countArrayItems(int32_t* rows, int32_t* columns, /* other params */); int32_t get2dArrayElement(int32_t rowIndex, int32_t colIndex, /* other params */, UChar* result, int32_t maxResultSize, UErrorCode* err);
Important change of error codes from streaming conversion functions
The error code that is set from streaming conversion like
ucnv_fromUnicode() - ucnv_toUnicode()
ucnv_fromUChars() - ucnv_toUChars()
scsu_compress() - scsu_decompress()
when the target buffer is full but the source not empty is changed from
U_INDEX_OUTOFBOUNDS_ERROR to U_BUFFER_OVERFLOW_ERROR.
This change makes the error codes more consistent with their names and
with their use in other icu APIs.
You need to test for this new error code if your code uses ICU for conversion
and used the old error code. ucnv.h and scsu.h are updated with this information.
Please search in your source code for U_INDEX_OUTOFBOUNDS_ERROR.
If it is used with the above functions (not with ucnv_getNextUChar()),
then you need to change it to U_BUFFER_OVERFLOW_ERROR in order to
get your code to work with icu 1.6.
See the updated sample code in icu/source/samples. All samples are updated.
See jitterbug 516 for details.
This was discussed in july 2000 on the icu mailing list.
Please see the list archive for the discussion.
http://oss.software.ibm.com/icu/ is a pointer to general information about the International Components for Unicode.
docs/udata.html is a raw draft of ICU data handling.
html/aindex.html is an alphabetical
index to detailed API documentation.
html/HIERjava.html is a hierarchical
index to detailed API documentation.
docs/collate.html is an overview to Collation.
docs/BreakIterator.html is a diagram showing how BreakIterator processes text elements.
http://www.ibm.com/developer/unicode/ is a
pointer to information on how to make applications global.
To submit comments, request features and report bugs, please contact us. While we are not able to respond individually to each comment, we do review all comments. Send Internet email to icu@oss.software.ibm.com.
Copyright © 1997-2000 International Business Machines Corporation and
others. All Rights Reserved.
IBM Center for Java Technology Silicon Valley,
10275 N De Anza Blvd., Cupertino, CA 95014
All rights reserved.