International Components for Unicode
ICU 2.1 ReadMe

Version: 2002-May-13
Copyright © 1997-2002 International Business Machines Corporation and others. All Rights Reserved.


Table of Contents


Introduction

Today's software market is a global one in which it is desirable to develop and maintain one application (single source/single binary) that supports a wide variety of languages. The International Components for Unicode (C/C++) provides tools to help write platform-independent applications that are internationalized and localized, with support for:

ICU has a sister project ICU4J that extends the internationalization capabilities of Java to a level similar to ICU. The ICU C/C++ project is also called ICU4C when a distinction is necessary.

Getting started

This document describes how to build and install ICU on your machine. For other information about ICU please see the following table of links.
The ICU homepage also links to related information about writing internationalized software.

Here are some useful links regarding ICU and internationalization in general.
ICU Homepage http://oss.software.ibm.com/icu/
ICU4J Homepage http://oss.software.ibm.com/icu4j/
FAQ - Frequently Asked Questions about ICU http://oss.software.ibm.com/icu/userguide/icufaq.html
ICU User's Guide http://oss.software.ibm.com/icu/userguide/
Download ICU Releases http://oss.software.ibm.com/icu/download/
API Documentation Online http://oss.software.ibm.com/icu/apiref/
Online ICU Demos http://oss.software.ibm.com/icu/demo/
Contacts and Bug Reports/Feature Requests http://oss.software.ibm.com/icu/archives/

Important: Please make sure you understand the Copyright and License Information.

What is new in this release?

The following list concentrates on changes that affect existing applications migrating from previous ICU releases. For more news about this release, see the ICU 2.1 download page.

Support for Unicode 3.1.1

ICU 2.0 has been upgraded to support Unicode 3.1.1, which includes the addition of 44,946 new encoded characters. These characters cover several historic scripts, several sets of symbols, and a very large collection of additional CJK ideographs.

As part of this upgrade, a number of ICU services have been reviewed and improved with regards to handling supplementary characters (surrogate pairs). Especially, normalization is revamped for support of supplementary characters and higher performance.

ICU 2.1 also includes Corrigendum #3: U+F951 Normalization.

Memory and resource cleanup

ICU is carefully tested for memory leaks. Some memory is held in internal caches that do not normally get released during normal operation. These are not leaks because ICU continues to use them as necessary.

For testing purposes (for memory leaks) and for a small number of applications it can be useful to close all the memory that is allocated for a library. ICU 2.0 supports this with a new function u_cleanup() that may be called after an application has released all ICU objects. u_cleanup() will then release all of ICU's internal memory. The ICU libraries can then even be unloaded cleanly without shutting down the process.

Data loading changed

ICU data loading is simplified for most users. By default, the ICU build creates a DLL/shared library that is linked directly with the common library ([lib]icuuc). By placing all ICU libraries including the data library into the same folder, ICU should start up and find its data immediately. Dynamic loading of data from DLLs/shared libraries is not supported any more.

Before ICU 2.0, ICU did not itself link directly with its data library, but some ICU applications did (like the Xerces XML parser) and called udata_setCommonData(). This is not necessary any more in the default case.
On the other hand, this same technique can now be used to efficiently load application data (e.g., for its own localization). An application can build a data DLL/library of its own, link it, and call the new API udata_setAppData().

For details on finding and loading ICU data and on options for portable, common data files etc. see the User's Guide ICU Data Chapter.

Library linking changed

Multithreaded usage is safer

It was discovered that some parts of ICU were not initialized in a thread safe manner. This has been fixed.


How To Download the Source Code

There are two ways to download ICU releases:

ICU Source Code Organization

In the descriptions below, <ICU> is the full path name of the icu directory (the top level directory from the distribution archives) in your file system. You can also view the User's Guide to see which libraries you need for your software product. You need at least the data ([lib]icudt) and the common ([lib]icuuc) libraries in order to use ICU.

The following files describe the code drop.
readme.html Describes the International Components for Unicode (this file)
license.html Contains the text of the ICU license


The following directories contain source code and data files.
<ICU>/source/common/ The core Unicode and support functionality, such as resource bundles, character properties, locales, codepage conversion, normalization, Unicode properties, Locale, and UnicodeString.
<ICU>/source/i18n/ Modules in i18n are generally the more data-driven, that is to say resource bundle driven, components. These deal with higher-level internationalization issues such as formatting, collation, text break analysis, and transliteration.
<ICU>/source/data This directory contains the source data in text format, which is compiled into binary form during the ICU build process. It contains several subdirectories, in which the data files are grouped by function. Note that the build process must be run again after any changes are made to this directory.
  • brkitr/ Data files for character, word, sentence, title casing and line boundary analysis.
  • locales/ These .txt files contain ICU language and culture-specific localization data. Two special bundles are root, which is the fallback data and parent of other bundles, and index, which contains a list of installed bundles. The makefile resfiles.mk contains the list of resource bundle files.
  • mappings/ Here are the code page converter tables, .ucm files containing mappings to and from Unicode. These are compiled into .cnv files. convrtrs.txt is the alias mapping table from various converter name formats to ICU internal format and vice versa. It produces cnvalias.dat. The makefiles ucmfiles.mk, ucmcore.mk, and ucmebcdic.mk contain the list of converters to be built.
  • translit/ This directory contains transliterator rules as resource bundles, a makefile trnsfiles.mk containing the list of installed system translitaration files, and as well the special bundle translit_index which lists the system transliterator aliases.
  • unidata/ This directory contains the Unicode data files. Please see http://www.unicode.org/ for more information.
  • misc/ The misc directory contains other data files which did not fit into the above categories. Currently it only contains timezone.txt, a generated file, which is compiled into tz.dat. The tz.dat file contains time zone information.
  • out/ This directory contains the assembled memory mapped files.
  • out/build This directory contains intermediate (compiled) files, such as .cnv, .res, etc.
<ICU>/source/test/intltest/ A test suite including all C++ APIs. For information about running the test suite, see the users' guide.
<ICU>/source/test/cintltst/ A test suite written in C, including all C APIs. For information about running the test suite, see the users' guide.
<ICU>/source/test/testdata/ Source text files for data, which are read by the tests. It contains the subdirectories out/build/ which is used for intermediate files, and out/ which contains the files test1.cnv through test4.cnv, and testdata.dat. Note that the tests call u_setDataDirectory("<ICU>/source/test/testdata/lib"), so that ICU will load these files as if they were part of the ICU data package, for testing purposes. This was formerly accomplished by setting the ICU_DATA environment variable to point at these files. ICU_DATA should not be set under normal circumstances.
<ICU>/source/tools Tools for generating the data files. Data files are generated by invoking <ICU>/source/data/build/makedata.bat on Win32 or <ICU>/source/make on Unix.
<ICU>/source/samples Various sample programs that use ICU
<ICU>/source/extra Non-supported API additions. Currently, it contains the 'ustdio' file i/o library
<ICU>/source/layout/ Contains the ICU layout engine (not a rasterizer).
<ICU>/packaging/
<ICU>/debian/
These directories contain scripts and tools for packaging the final ICU build for various release platforms.
<ICU>/source/config/ Contains helper makefiles for platform specific build commands. Used by 'configure'.
<ICU>/source/allinone/ Contains top-level ICU workspace and project files, for instance to build all of ICU under one MSVC project.
<ICU>/bin/ Contains the libraries and executables for using ICU on Windows.
<ICU>/include/ Contains the headers needed for developing software that uses ICU on Windows.

How To Build And Install ICU

Supported Platforms

Here is a status of functionality of ICU on several different platforms.
Operating system Compiler Testing frequency
Windows NT/2000/XP Microsoft Visual C++ 6.0 Reference platform
Red Hat Linux 7.2 gcc 2.96 Reference platform
AIX 5.1.0 L Visual Age C++ 5.0 Reference platform
Solaris 7 (SunOS 5.7) Workshop Pro CC 6.0 Reference platform
HP/UX 11.01 aCC A.12.10 Reference platform
Windows 98 Microsoft Visual C++ 6.0 Regularly tested
AIX 4.3.3 xlC 3.6.6 Regularly tested
Solaris 8 (SunOS 5.8) Workshop Pro CC 4.2 Regularly tested
Solaris 7 (SunOS 5.7) Workshop Pro CC 4.2 Regularly tested
Solaris 2.6 (SunOS 5.6) gcc 2.95.2 Regularly tested
FreeBSD 4.4 gcc 2.95.3 Regularly tested
Mac OS X (10.1) gcc-932.1, based on gcc version 2.95.2
(Developer Tools December 2001)
Regularly tested
OS/390 (zSeries) CC r10 Regularly tested
HP/UX 11.01 CC A.03.10 Rarely tested
AS/400 (iSeries) V5R1 iCC Rarely tested
NetBSD, OpenBSD   Rarely tested
SGI/IRIX   Rarely tested
Tru64 (OSF) Compaq's cxx compiler Rarely tested


Key to testing frequency

Reference platform
ICU will work on these platforms with these compilers
Regularly tested
ICU should work on these platforms with these compilers
Rarely tested
ICU has been ported to these platforms but may not have been tested there recently

How To Build And Install On Windows

Building International Components for Unicode requires:

(If you want to build with Microsoft Visual C++ .NET, please refer to the note about building with Visual Studio .NET below.)

The steps are:

  1. Unzip the icu-XXXX.zip file into any convenient location. Using command line zip, type "unzip -a icu-XXXX.zip -d drive:\directory", or just use WinZip.
  2. Be sure that the ICU binary directory, <ICU>\bin\, is included in the PATH environment variable. The tests will not work without the location of the ICU dll files in the path.
  3. Set the TZ environment variable to PST8PDT. The tests will not work in any other time zone.
  4. Open the "<ICU>\source\allinone\allinone.dsw" workspace file in Microsoft Visual C++ 6.0. (This workspace includes all the International Components for Unicode libraries, necessary ICU building tools, and the intltest and cintltest test suite projects). Please see the note below if you want to build from the command line instead.
  5. Set the active Project to the "all" project. To do this: Choose "Project" menu, and select "Set active project". In the submenu, select the "all" workspace.
  6. Set the active configuration to "Win32 Debug" or "Win32 Release" (See note below).
  7. Choose the "Build" menu and select "Rebuild All". If you want to build the Debug and Release at the same time, see the note below.
  8. Run the C++ test suite, "intltest". To do this: set the active project to "intltest", and press F5 to run it.
  9. Run the C test suite, "cintltst". To do this: set the active project to "cintltst", and press F5 to run it.
  10. Make sure that both "cintltst" and "intltest" passed without any errors. The return codes are non-zero when they do not pass. Visual C++ will display the return codes in the debug tag of the output window. When "intltest" and "cintltest" return 0, it means that everything is installed correctly. You can press Ctrl+F5 on the test project to run the test and see what error messages were displayed (if any tests failed).
  11. Reset the TZ environment variable to its original value, unless you plan on testing ICU any further.
  12. You are now able to develop applications with ICU by using the libraries and tools in <ICU>\bin\. The headers are in <ICU>\include\ and the link libraries are in <ICU>\lib\. To install the ICU runtime on a machine, or ship it with your application, copy the needed components from <ICU>\bin\ to a location on the system PATH or to your application directory.

Using MSDEV At The Command Line Note: You can build ICU from the command line. Assuming that you have properly installed Microsoft Visual C++ to support command line execution, you can run the following command, 'msdev <ICU>\source\allinone\allinone.dsw /MAKE "ALL"'.

Setting Active Configuration Note: To set the active configuration, two different possibilities are:

Batch Configuration Note: If you want to build the Debug and Release configurations at the same time, choose "Build" menu and select "Batch Build..." instead (and mark all configurations as checked), then click the button named "Rebuild All". The "all" workspace will build all the libraries, test programs and various ICU tools (e.g. genrb for generating binary locale data files).

Microsoft Visual Studio .NET Note: ICU will build with Microsoft Visual Studio .NET, though this is not a supported platform at the time of this writing. Visual Studio .NET will offer to convert the project files when you open the existing workspace file. Choose "Yes to All" in the dialog asking whether to convert the files or not (this creates new files) and then follow the rest of the build instructions.

How To Build And Install On Unix

Building International Components for Unicode on Unix requires:

A UNIX C++ compiler, (gcc, CC, xlC_r, etc...) installed on the target machine. A recent version of GNU make (3.7+). For a list of OS/390 tools please view the OS/390 build section of this document for further details.

The steps are:

  1. Decompress the icuXXXX.tar (or icuXXXX.tgz) file. For example, gunzip -d < icuXXXX.tgz | tar xvf -
  2. Change directory to the "icu/source".
  3. Run "chmod +x runConfigureICU configure install-sh" as these files may have the wrong permissions.
  4. Run the runConfigureICU script for your platform. (See note below).
  5. Type "gmake" (or "make" if GNU make is the default make on your platform) to compile the libraries and all the data files. The proper name of the GNU make command is printed at the end of the configuration run, as in "You must use gmake to compile ICU".
  6. Optionally, type "gmake check" (or "make check") to run the test suite, which checks for ICU's functionality integrity (See note below).
  7. Type "gmake install" (or "make install") to install. The install targets support the use of the DESTDIR variable to create the installation tree under a specific destination directory. (See note below).

Configuring ICU NOTE: Type "./runConfigureICU --help" for help on how to run it and a list of supported platforms. You may also want to type "./configure --help" to print the available configure options that you may want to give runConfigureICU. If you are not using the runConfigureICU script, or your platform is not supported by the script, you may need to set your CC,CXX, CFLAGS and CXXFLAGS environment variables, and type "./configure". Some of the more frequently used options to configure are --disable-64bit-libs to create 32-bit libraries, and --srcdir to do out of source builds (build the libraries in the current location).

Running The Tests From The Command Line NOTE: You may have to set certain variables if you with to run test programs individually, that is apart from "make check". The TZ environment variable needs to be set to PST8PDT. Also, the environment variable ICU_DATA can be set to the full pathname of the data directory to indicate where the locale data files and conversion mapping tables are. The trailing "/" is required after the directory name (e.g. "$Root/source/data/" will work, but the value "$Root/source/data" is not acceptable). You do not need to set ICU_DATA if the complete data library is in your library path.

Installing ICU NOTE: If you are using the "gmake install" command, using the "--prefix" option on configure or runConfigureICU will install ICU to the specified location.

Some platforms use package management tools to control the installation and uninstallation of files on the system, as well as the integrity of the system configuration. You may want to check if ICU can be packaged for your package management tools by looking into the "packaging" directory. (Please note that if you are using a snapshot of ICU from CVS, it is probable that the packaging scripts or related files are not up to date with the contents of ICU at this time, so use them with caution).

OS/390 (zSeries) Platform

If you are building on the OS/390 UNIX System Services platform, it is important that you understand a few details:

OS/390 Batch (PDS) support

By default, ICU builds its libraries into the HFS. However, there is a 390-specific switch to build some libraries into PDS files. The switch is the environmental variable OS390BATCH, and if set, the following libraries are built into PDS files: libicuucXX.dll, libicudtXXe.dll, libicudtXXe_390.dll, and libtestdata.dll. Turning on OS390BATCH does not turn off the normal HFS build, thus the HFS dlls will always be created.

The names of the PDS files are determined by the value of the environmental variables LOADMOD and LOADEXP. These variables must contain the target PDS names whenever the OS390BATCH variable is set. LOADMOD is the library (.dll) target dataset and LOADEXP is the side deck (.x) target dataset.

The PDS member names are as follows:

IXMIXXUC --> libicuucXX.dll
IXMIXXDA --> libicudtXXe.dll
IXMIXXD1 --> libicudtXXe_390.dll

Example PDS attributes are as follows:

Data Set Name . . . : USER.ICU.LOAD
General Data
Management class. . : **None**
Storage class . . . : BASE
Volume serial . . . : TSO007
Device type . . . . : 3390
Data class. . . . . : LOAD
Organization  . . . : PO
Record format . . . : U
Record length . . . : 0
Block size  . . . . : 32760
1st extent cylinders: 40
Secondary cylinders : 59
Data set name type  : PDS

Data Set Name . . . : USER.ICU.EXP
General Data
Management class. . : **None**
Storage class . . . : BASE
Volume serial . . . : TSO007
Device type . . . . : 3390
Data class. . . . . : **None**
Organization  . . . : PO
Record format . . . : FB
Record length . . . : 80
Block size  . . . . : 3200
1st extent cylinders: 3
Secondary cylinders : 3
Data set name type  : PDS

OS/400 (iSeries) Platform

ICU Reference Release 1.8.1 contains partial support for the 400 platform, but additional work by the user is currently needed to get it to build properly. A future release of ICU should work out-of-the-box under OS/400.

Important Notes About Using ICU

Using ICU in a Multithreaded Environment

If you are using ICU in a multithreaded application, there may be a chance that the ICU global mutex is not initialized properly. Normally the ICU global mutex is initialized during C++ static initialization, but there are some compilers and linkers that do not properly perform C++ static initialization in a library (this sometimes happens on HP/UX and on MacOS X).

Upon the first usage of most ICU APIs, the global mutex will get initialized. For example, you could call uloc_countAvailable() or uloc_getDefault() from your main() function before any threads are created. Those functions will initialize the global mutex. Without one of these function calls from a single thread, the data caches inside ICU may get initialized more than once, which may cause memory leaks and other problems. This problem normally does not happen when C++ static initialization works properly.

ICU does not use C++ static initialization for anything else, and disabling threads in ICU will disable all C++ static initialization in ICU.

Windows Platform

If you are building on the Win32 platform, it is important that you understand a few of the following build details.

DLL directories and the PATH setting

As delivered, the International Components for Unicode build as several DLLs, which are placed in the "<ICU>\bin" directory. You must add this directory to the PATH environment variable in your system, or any executables you build will not be able to access International Components for Unicode libraries. Alternatively, you can copy the DLL files into a directory already in your PATH, but we do not recommend this. You can wind up with multiple copies of the DLL and wind up using the wrong one.

Changing your PATH

Note: When packaging a Windows application for distribution and installation on user systems, copies of the ICU DLLs should be included with the application, and installed for exclusive use by the application. This is the only way to insure that your app is running with the same version of ICU, built with exactly the same options, that you developed and tested with. Refer to Microsoft's guidelines on the usage of DLLs, or search for the phrase "DLL hell" on msdn.microsoft.com.

Linking with Runtime libraries

All the DLLs link with the C runtime library "Debug Multithreaded DLL" or "Multithreaded DLL." (This is changed through the Project Settings dialog, on the C/C++ tab, under Code Generation.) It is important that any executable or other DLL you build which uses the International Components for Unicode DLLs links with these runtime libraries as well. If you do not do this, you will get random memory errors when you run the executable.

Unix Type Platform

If you are building on a Unix platform, and if you are installing ICU in a non-standard location, you may need to add the location of your ICU libraries to your LD_LIBRARY_PATH environment variable (or the equivalent runtime library path environment variable for your system). The ICU libraries may not link or load properly without doing this.

Note that if you do not want to have to set this variable, you may instead use the --enable-rpath option at configuration time. This option will instruct the linker to always look for the libraries where they are installed. You will need to use the appropriate linker options when linking your own applications and libraries against ICU, too. Please refer to your system's linker manual for information about runtime paths. The use of rpath also means that when building a new version of ICU you should not have an older version installed in the same place as the new version's installation directory, as the older libraries will used during the build, instead of the new ones, likely leading to an incorrectly build ICU. (This is the proper behavior of rpath.)

Using the default codepage

ICU has code to determine the default codepage of the system or process. This default codepage can be used to convert char * strings to and from Unicode.

Depending on system design, setup and APIs, it may not always be possible to find a default codepage that fully works as expected. For example,

If you have means of detecting a default codepage name that are more appropriate for your application, then you should set that name with ucnv_setDefaultName() as the first ICU function call. This makes sure that the internally cached default converter will be instantiated from your preferred name.

Starting in ICU 2.0, when a converter for the default codepage cannot be opened, a fallback default codepage name and converter will be used. On most platforms, this will be US-ASCII. For OS/390 (z/OS), ibm-1047-s390 is the default fallback codepage. For AS/400 (iSeries), ibm-37 is the default fallback codepage. This default fallback codepage is used when the operating system is using a non-standard name for a default codepage, or the converter was not packaged with ICU. The feature allows ICU to run in unusual computing environments without completely failing.

Methods for enabling deprecated APIs

C

Some deprecated C APIs can be enabled without recompiling the ICU libraries. This can be achieved by defining certain symbols before including the ICU header files. For example, to enable deprecated C APIs for formatting.

#ifndef U_USE_DEPRECATED_FORMAT_API
#  define U_USE_DEPRECATED_FORMAT_API 1
#endif

#include "unicode/udat.h"

int main(){
    UDateFormat *def, *fr, *fr_pat ;
    UErrorCode status = U_ZERO_ERROR;
    UChar temp[30];

    fr = udat_open(UDAT_FULL, UDAT_DEFAULT, "fr_FR", NULL,0, &status);
    if(U_FAILURE(status)){
        printf("Error creating the french dateformat using full time style\n %s\n",
            myErrorName(status) );
    }
    /* This is supposed to open default date format,
       but later on it treats it like it is "en_US".
       This is very bad when you try to run the tests
       on a machine where the default locale is NOT "en_US"
    */
    def = udat_open(UDAT_SHORT, UDAT_SHORT, "en_US", NULL, 0, &status);
    if(U_FAILURE(status)){
        .... /* handle the error */
    }
}

C++

Deprecated C++ APIs cannot be enabled without recompiling ICU libraries. Every service has a specific symbol that should be defined to enable the deprecated API of that service. For example: To enable deprecated APIs in transliteration service, the U_USE_DEPRECATED_TRANSLITERATOR_API symbol should be defined before compiling ICU.

Platform Dependencies

The platform dependencies have been mostly isolated into the following files in the common library. This information can be useful if you are porting ICU to a new platform.

It is possible to build each library individually. They must be built in the following order:

  1. stubdata
  2. common
  3. i18n
  4. toolutil
  5. makeconv
  6. gencnval
  7. genrb
  8. gentz
  9. gennames
  10. genuca
  11. gennorm
  12. genccode
  13. gencmn
  14. pkgdata
  15. makedata (a project on Windows, or source/data/Makefile on Unix)
  16. ctestfw, intltest and cintltst, if you want to run the test suite.

Copyright © 1997-2002 International Business Machines Corporation and others. All Rights Reserved.
IBM Globalization Center of Competency - San Jose,
5600 Cottle Road, San José, CA 95193
All rights reserved.