International Components for Unicode
ICU 3.6 ReadMe

Version: 2006-May-26
Copyright © 1997-2006 International Business Machines Corporation and others. All Rights Reserved.

Table of Contents

• Introduction
• Getting Started
• What Is New In This release?
• How To Download the Source Code
• ICU Source Code Organization
• How To Build And Install ICU
  • Supported Platforms
  • Windows
  • Cygwin
  • UNIX
  • z/OS (os/390)
  • i5/OS (OS/400 iSeries)
• How To Package ICU
• Important Notes About Using ICU
  • Using ICU in a Multithreaded Environment
  • Windows Platform
  • UNIX Type Platforms
• Platform Dependencies
  • Porting To A New Platform
  • Platform Dependent Implementations

Introduction

Today's software market is a global one in which it is desirable to develop and maintain one application (single source/single binary) that supports a wide variety of languages. The International Components for Unicode (ICU) libraries provide robust and full-featured Unicode services on a wide variety of platforms to help this design goal. The ICU libraries provide support for:

• The latest version of the Unicode standard
• Character set conversions with support for over 200 codepages
• Locale data for more than 230 locales
• Language sensitive text collation (sorting) and searching based on the Unicode Collation Algorithm (=ISO 14651)
• Regular expression matching and Unicode sets
• Transformations for normalization, upper/lowercase, script transliterations (50+ pairs)
• Resource bundles for storing and accessing localized information
• Date/Number/Message formatting and parsing of culture specific input/output formats
• Calendar specific date and time manipulation
• Complex text layout for Arabic, Hebrew, Indic and Thai
• Text boundary analysis for finding characters, word and sentence boundaries

ICU has a sister project ICU4J that extends the internationalization capabilities of Java to a level similar to ICU. The ICU C/C++ project is also called ICU4C when a distinction is necessary.

Getting started

This document describes how to build and install ICU on your machine. For other information about ICU please see the following table of links.
The ICU homepage also links to related information about writing internationalized software.

Important: Please make sure you understand the Copyright and License Information.

What is new in this release?

The following list concentrates on changes that affect existing applications migrating from previous ICU releases. For more news about this release, see the ICU 3.6 download page.

Changes to packaging resource bundles

Since ICU 3.0, the old style of packaging was deprecated, and an alternate packaging mode was made available. In this release, this compatibility mode of packaging has been removed from ICU. If you're using the genrb -P or -t options, you are probably using the older compatible mode of data packaging. This compatibility file naming mode was removed for portability and performance reasons.

Code changes should not be required to use the newer data file naming scheme, but you will need to update your makefile scripts, if you're using the older data file naming scheme. The example of using the new data file naming scheme can be found in the ufortune sample program.

Changes to wchar_t type for the Microsoft Visual Studio builds

Previous versions of ICU built with Microsoft Visual Studio were not built with the /Zc:wchar_t compiler option. ICU now builds with this option turned on by default. This allows the built libraries to be compatible with Visual Studio 2005, and this makes it easier for ICU users to use MFC in their projects, which requires this option to be turned by default. If you do not use the C++ API of ICU, you are not affected by this change.

If you receive any errors while linking ICU into your project, please make sure that you have set "Treat wchar_t as Built-in Type" to "Yes (/Zc:wchar_t)" in your project files.

Source download contains .dat package for ICU data

The ICU4C 3.6 source downloads contain a pre-built .dat package with ICU's data rather than the data source files. This is to simplify the build process for the majority of users and to reduce platform porting issues. If you need the data source files for customization, then please download the ICU source code from CVS.

How To Download the Source Code

There are two ways to download ICU releases:

• Official Release Snapshot:
  If you want to use ICU (as opposed to developing it), you should download an official packaged version of the ICU source code. These versions are tested more thoroughly than day-to-day development builds of the system, and they are packaged in zip and tar files for convenient download. These packaged files can be found at http://www.ibm.com/software/globalization/icu/downloads.jsp.
  The packaged snapshots are named icu-nnnn.zip or icu-nnnn.tgz, where nnnn is the version number. The .zip file is used for Windows platforms, while the .tgz file is preferred on most other platforms.
  Please unzip this file. It will reconstruct the source directory, which includes anonymous CVS control directories (see below).
• CVS Source Repository:
  If you are interested in developing features, patches, or bug fixes for ICU, you should probably be working with the latest version of the ICU source code. You will need to check the code out of our CVS repository to ensure that you have the most recent version of all of the files. See our CVS page for details.

ICU Source Code Organization

In the descriptions below, <ICU> is the full path name of the ICU directory (the top level directory from the distribution archives) in your file system. You can also view the ICU Architectural Design section of the User's Guide to see which libraries you need for your software product. You need at least the data ([lib]icudt) and the common ([lib]icuuc) libraries in order to use ICU.

How To Build And Install ICU

Supported Platforms

Key to testing frequency

Reference platform

ICU will work on these platforms with these compilers

Regularly tested

ICU should work on these platforms with these compilers

Rarely tested

ICU has been ported to these platforms but may not have been tested there recently

How To Build And Install On Windows

Building International Components for Unicode requires:

• Microsoft Windows 2000 or above
• Microsoft Visual C++ 2003
• Cygwin is required when other versions of Microsoft Visual C++ and other compilers are used to build ICU.

The steps are:

1. Unzip the icu-XXXX.zip file into any convenient location. Using command line zip, type "unzip -a icu-XXXX.zip -d drive:\directory", or just use WinZip.
2. Be sure that the ICU binary directory, <ICU>\bin\, is included in the PATH environment variable. The tests will not work without the location of the ICU DLL files in the path.
3. Open the "<ICU>\source\allinone\allinone.sln" workspace file in Microsoft Visual Studio .NET 2003. (This solution includes all the International Components for Unicode libraries, necessary ICU building tools, and the test suite projects). Please see the command line note below if you want to build from the command line instead.
4. Set the active configuration to "Debug" or "Release" (See Windows configuration note below).
5. Choose the "Build" menu and select "Rebuild Solution". If you want to build the Debug and Release at the same time, see the batch configuration note below.
6. Run the C++ test suite, "intltest". To do this: set the active startup project to "intltest", and press Ctrl+F5 to run it. Make sure that it passes without any errors.
7. Run the C test suite, "cintltst". To do this: set the active startup project to "cintltst", and press Ctrl+F5 to run it. Make sure that it passes without any errors.
8. Run the I/O test suite, "iotest". To do this: set the active startup project to "iotest", and press Ctrl+F5 to run it. Make sure that it passes without any errors.
9. You are now able to develop applications with ICU by using the libraries and tools in <ICU>\bin\. The headers are in <ICU>\include\ and the link libraries are in <ICU>\lib\. To install the ICU runtime on a machine, or ship it with your application, copy the needed components from <ICU>\bin\ to a location on the system PATH or to your application directory.

Using MSDEV At The Command Line Note: You can build ICU from the command line. Assuming that you have properly installed Microsoft Visual C++ to support command line execution, you can run the following command, 'devenv.com <ICU>\source\allinone\allinone.sln /build Release'. You can also use Cygwin with this compiler to build ICU, and you can refer to the How To Build And Install On Windows with Cygwin section for more details.

Setting Active Configuration Note: To set the active configuration, two different possibilities are:

• Choose "Build" menu, select "Configuration Manager...", and select "Release" or "Debug" for the Active Configuration Solution.
• Another way is to select the desired build configuration from "Solution Configurations" dropdown menu from the standard toolbar. It will say "Release" or "Debug" in the dropdown list.

Batch Configuration Note: If you want to build the Debug and Release configurations at the same time, choose "Build" menu, and select "Batch Build...". Click the "Select All" button, and then click the "Rebuild" button.

How To Build And Install On Windows with Cygwin

Building International Components for Unicode with this configuration requires:

• Microsoft NT 4.0 or above, or Windows 98 or above
• Microsoft Visual C++ 6.0 or above (when gcc isn't used).
• Cygwin with the following installed:
  • bash
  • GNU make
  • man (if you plan to look at the man pages)

There are two ways you can build ICU with Cygwin. You can build with gcc or Microsoft Visual C++. If you use gcc, the resulting libraries and tools will depend on the Cygwin environment. If you use Microsoft Visual C++, the resulting libraries and tools do not depend on Cygwin and can be more easily distributed to other Windows computers (the generated man pages and shell scripts still need Cygwin). To build with gcc, please follow the "How To Build And Install On UNIX" instructions, while you are inside a Cygwin bash shell. To build with Microsoft Visual C++, please use the following instructions:

1. Start the Windows "Command Prompt" window. This is different from the gcc build, which requires the Cygwin Bash command prompt. The Microsoft Visual C++ compiler will not work with a bash command prompt.
2. If the computer isn't set up to use Visual C++ from the command line, you need to run VCVARS32.BAT (for example: "C:\Program Files\Microsoft Visual Studio\VC98\Bin\VCVARS32.BAT").
3. Unzip the icu-XXXX.zip file into any convenient location. Using command line zip, type "unzip -a icu-XXXX.zip -d drive:\directory", or just use WinZip.
4. Change directory to "icu/source", which is where you unzipped ICU.
5. Run "bash ./runConfigureICU Cygwin/MSVC" (See Windows configuration note and non-functional configure options below).
6. Type "make" to compile the libraries and all the data files. This make command should be GNU make.
7. Optionally, type "make check" to run the test suite, which checks for ICU's functionality integrity (See testing note below).
8. Type "make install" to install ICU. If you used the --prefix= option on configure or runConfigureICU, ICU will be installed to the directory you specified. (See installation note below).

Configuring ICU on Windows NOTE: In addition to the Unix configuration note the following configure options currently do not work on Windows with Microsoft's compiler. Some options can work by manually editing icu/source/common/unicode/pwin32.h, but manually editing the files is not recommended.

• --disable-renaming
• --disable-threading
• --disable-tracing
• --enable-64bit-libs
• --enable-rpath
• --with-iostream
• --enable-static (Requires that U_STATIC_IMPLEMENTATION be defined in user code that links against ICU's static libraries.)
• --with-data-packaging=files (The pkgdata tool currently does not work in this mode. Manual packaging is required to use this mode.)

How To Build And Install On UNIX

Building International Components for Unicode on UNIX requires:

• A C++ compiler installed on the target machine (for example: gcc, CC, xlC_r, aCC, cxx, etc...).
• An ANSI C compiler installed on the target machine (for example: cc).
• A recent version of GNU make (3.77+).
• For a list of z/OS tools please view the z/OS build section of this document for further details.

Here are the steps to build ICU:

1. Decompress the icu-X.Y.tgz (or icu-X.Y.tar.gz) file. For example, "gunzip -d < icu-X.Y.tgz | tar xvf -"
2. Change directory to the "icu/source".
3. Run "chmod +x runConfigureICU configure install-sh" because these files may have the wrong permissions.
4. Run the runConfigureICU script for your platform. (See configuration note below).
5. Type "gmake" (or "make" if GNU make is the default make on your platform) to compile the libraries and all the data files. The proper name of the GNU make command is printed at the end of the configuration run, as in "You must use gmake to compile ICU".
6. Optionally, type "gmake check" to run the test suite, which checks for ICU's functionality integrity (See testing note below).
7. Type "gmake install" to install ICU. If you used the --prefix= option on configure or runConfigureICU, ICU will be installed to the directory you specified. (See installation note below).

Configuring ICU NOTE: Type "./runConfigureICU --help" for help on how to run it and a list of supported platforms. You may also want to type "./configure --help" to print the available configure options that you may want to give runConfigureICU. If you are not using the runConfigureICU script, or your platform is not supported by the script, you may need to set your CC, CXX, CFLAGS and CXXFLAGS environment variables, and type "./configure". Some of the more frequently used options to configure are --disable-64bit-libs to create 32-bit libraries, and --srcdir to do out of source builds (build the libraries in the current location). HP-UX user's, please see this note regarding HP-UX multithreaded build issues with newer compilers. Solaris user's, please see this note regarding Solaris multithreaded build issues.

Running The Tests From The Command Line NOTE: You may have to set certain variables if you with to run test programs individually, that is apart from "gmake check". The environment variable ICU_DATA can be set to the full pathname of the data directory to indicate where the locale data files and conversion mapping tables are when you are not using the shared library (e.g. by using the .dat archive or the individual data files). The trailing "/" is required after the directory name (e.g. "$Root/source/data/out/" will work, but the value "$Root/source/data/out" is not acceptable). You do not need to set ICU_DATA if the complete shared data library is in your library path.

Installing ICU NOTE: Some platforms use package management tools to control the installation and uninstallation of files on the system, as well as the integrity of the system configuration. You may want to check if ICU can be packaged for your package management tools by looking into the "packaging" directory. (Please note that if you are using a snapshot of ICU from CVS, it is probable that the packaging scripts or related files are not up to date with the contents of ICU at this time, so use them with caution).

How To Build And Install On z/OS (OS/390)

You can install ICU on z/OS or OS/390 (the previous name of z/OS), but IBM tests only the z/OS installation. These platforms commonly are called "MVS". You install ICU in a z/OS UNIX system services file system such as HFS or zFS. On this platform, it is important that you understand a few details:

• APAR PQ58392 may be needed by z/OS 1.2 or 1.3 in order to get some ICU number formatting functions to work properly. The APAR affects C and C++ code.
• The makedep executable that is used with the z/OS ICU build process is not shipped with ICU. It is available at the z/OS UNIX - Tools and Toys site. The PATH environment variable should be updated to contain the location of this executable prior to build. Alternatively, makedep may be moved into an existing PATH directory.
• The gnu utilities gmake and gzip/gunzip are needed and can be obtained for z/OS from z/OS UNIX - Tools and Toys.
• Since the default make on z/OS is not gmake, the pkgdata tool requires that the "make" command is aliased to your installed version of gmake. You may also need to set $MAKE equal to the fully qualified path of GNU make. GNU make is available with the "z/OS UNIX - Tools and Toys" that was mentioned above. ICU requires the same GNU make as described in the UNIX build instructions.
• Since USS does not support using the mmap() function over NFS, it is recommended that you build ICU on a local filesystem. Once ICU has been built, you should not have this problem while using ICU when the data library has been built as a shared library, which is this is the default setting.
• Encoding considerations: The source code assumes that it is compiled with codepage ibm-1047 (to be exact, the UNIX System Services variant of it). The pax command converts all of the source code files from ASCII to codepage ibm-1047 (USS) EBCDIC. However, some files are binary files and must not be converted, or must be converted back to their original state. You can use the unpax-icu.sh script to do this for you automatically. It will unpackage the tar file and convert all the necessary files for you automatically.
• z/OS supports both native S/390 hexadecimal floating point and (with OS/390 2.6 and later) IEEE 754 binary floating point. This is a compile time option. Applications built with IEEE should use ICU DLLs that are built with IEEE (and vice versa). The environment variable IEEE390=0 will cause the z/OS version of ICU to be built without IEEE floating point support and use the native hexadecimal floating point. By default ICU is built with IEEE 754 support. Native floating point support is sufficient for codepage conversion, resource bundle and UnicodeString operations, but the Format APIs require IEEE binary floating point.

• z/OS introduced the concept of Extra Performance Linkage (XPLINK) to bring performance improvement opportunities to call-intensive C and C++ applications such as ICU. XPLINK is enabled on a DLL-by-DLL basis, so if you are considering using XPLINK in your application that uses ICU, you should consider building the XPLINK-enabled version of ICU. You need to set ICU's environment variable OS390_XPLINK=1 prior to invoking the make process to produce binaries that are enabled for XPLINK.

  Note: XPLINK, which is enabled for z/OS 1.2 and later, requires the PTF PQ69418 to build XPLINK enabled binaries.

• Currently in ICU 3.0, there is an issue with building on z/OS without XPLINK and with the C++ iostream. By default, the iostream library on z/OS is XPLINK enabled. If you are not building an XPLINK enabled version of ICU, you should use the --with-iostream=old configure option when using runConfigureICU. This will prevent applications that use the icuio library from crashing.

• When you build ICU on a system such as z/OS 1.2, the binaries that result can run on that level of the operating system and later, such as z/OS 1.3 and z/OS 1.4. It's possible that you may have a z/OS 1.4 system, but you may need to deliver binaries on z/OS 1.2 and above. z/OS gives you this ability by targeting the complier and linker to run at the older level, thereby producing the desired binaries.

  To set the compiler and LE environment to OS/390 2.10, specify the following, "./runConfigureICU OS390V2R10"

  To set the compiler and LE environment to z/OS 1.2 specify the following, "./runConfigureICU zOSV1R2"

• The rest of the instructions for building and testing ICU on z/OS with UNIX System Services are the same as the How To Build And Install On UNIX section.

z/OS (Batch/PDS) support outside the UNIX system services environment

By default, ICU builds its libraries into the UNIX file system (HFS). In addition, there is a z/OS specific environment variable (OS390BATCH) to build some libraries into the z/OS native file system. This is useful, for example, when your application is externalized via Job Control Language (JCL).

The OS390BATCH environment variable enables non-UNIX support including the batch environment. When OS390BATCH is set, the libicuucXX.dll, libicudtXXe.dll, and libicudtXXe_stub.dll binaries are built into data sets (the native file system). Turning on OS390BATCH does not turn off the normal z/OS UNIX build. This means that the z/OS UNIX (HFS) DLLs will always be created.

Two additional environment variables indicate the names of the z/OS data sets to use. The LOADMOD environment variable identifies the name of the data set that contains the dynamic link libraries (DLLs) and the LOADEXP environment variable identifies the name of the data set that contains the side decks, which are normally the files with the .x suffix in the UNIX file system.

A data set is roughly equivalent to a UNIX or Windows file. For most kinds of data sets the operating system maintains record boundaries. UNIX and Windows files are byte streams. Two kinds of data sets are PDS and PDSE. Each data set of these two types contains a directory. It is like a UNIX directory. Each "file" is called a "member". Each member name is limited to eight bytes, normally EBCDIC.

Here is an example of some environment variables that you can set prior to building ICU:

OS390BATCH=1
LOADMOD=USER.ICU.LOAD
LOADEXP=USER.ICU.EXP

The PDS member names for the DLL file names are as follows:

IXMIXXIN --> libicui18nXX.dll
IXMIXXUC --> libicuucXX.dll
IXMIXXDA --> libicudtXXe.dll
IXMIXXD1 --> libicudtXXe_stub.dll (Only when OS390_STUBDATA=1)

You should point the LOADMOD environment variable at a partitioned data set extended (PDSE) and point the LOADEXP environment variable at a partitioned data set (PDS). The PDSE can be allocated with the following attributes:

Data Set Name . . . : USER.ICU.LOAD
Management class. . : **None**
Storage class . . . : BASE
Volume serial . . . : TSO007
Device type . . . . : 3390
Data class. . . . . : LOAD
Organization  . . . : PO
Record format . . . : U
Record length . . . : 0
Block size  . . . . : 32760
1st extent cylinders: 1
Secondary cylinders : 5
Data set name type  : LIBRARY

The PDS can be allocated with the following attributes:

Data Set Name . . . : USER.ICU.EXP
Management class. . : **None**
Storage class . . . : BASE
Volume serial . . . : TSO007
Device type . . . . : 3390
Data class. . . . . : **None**
Organization  . . . : PO
Record format . . . : FB
Record length . . . : 80
Block size  . . . . : 3200
1st extent cylinders: 3
Secondary cylinders : 3
Data set name type  : PDS

How To Build And Install On i5/OS (OS/400 iSeries)

Before you start building ICU, ICU requires the following:

• QSHELL interpreter installed (install base option 30, operating system)
• ILE C/C++ Compiler for iSeries, LPP 5722-WDS
• The latest GNU facilities (You can get the GNU facilities for i5/OS from http://www.ibm.com/servers/enable/site/porting/iseries/overview/gnu_utilities.html). Older versions may not work properly.

The following describes how to setup and build ICU. For background information, you should look at the UNIX build instructions.

1. Create i5/OS target library. This library will be the target for the resulting modules, programs and service programs. You will specify this library on the OUTPUTDIR environment variable in step 2.

   CRTLIB LIB(libraryname)
   

2. Set up the following environment variables in your build process (use the libraryname from the previous step). The libraryname identifies target i5/OS library for *module, *pgm and *srvpgm objects.

   ADDENVVAR ENVVAR(CC) VALUE('/usr/bin/icc')
   ADDENVVAR ENVVAR(CXX) VALUE('/usr/bin/icc')
   ADDENVVAR ENVVAR(MAKE) VALUE('/usr/bin/gmake')
   ADDENVVAR ENVVAR(OUTPUTDIR) VALUE('libraryname')
   

3. Run 'CHGJOB CCSID(37)'
4. Run 'QSH'
5. Run gunzip on the ICU source code compressed tar archive (icu-X.Y.tgz).
6. Run unpax-icu.sh on the tar file generated from the previous step.
7. Change your current directory to icu/source.
8. Run 'export CFLAGS=-O4 CXXFLAGS=-O4' to optimize your build of ICU. If the build fails, rerun these build steps without this step before asking the icu-support mailing list for help.
9. Run './configure'
10. Run 'gmake' to build ICU.
11. Run 'gmake check' to build the tests.
12. The "utility/MultithreadTest" test in intltest may have failed during 'gmake check'. In order to make this test pass, please use 'gmake check QIBM_MULTI_THREADED=Y' after you built the tests with 'gmake check' from the previous step. You can look at the iSeries Information Center for more details.

How To Package ICU

There are many ways that a person can package ICU with their software products. Usually only the libraries need to be considered for packaging.

On UNIX, you should use "gmake install" to make it easier to develop and package ICU. The bin, lib and include directories are needed to develop applications that use ICU. These directories will be created relative to the "--prefix=dir" configure option (See the UNIX build instructions). When ICU is built on Windows, a similar directory structure is built.

When changes have been made to the standard ICU distribution, it is recommended that at least one of the following guidelines be followed for special packaging.

1. Add a suffix name to the library names. This can be done with the --with-library-suffix configure option.
2. The installation script should install the ICU libraries into the application's directory.

Following these guidelines prevents other applications that use a standard ICU distribution from conflicting with any libraries that you need. On operating systems that do not have a standard C++ ABI (name mangling) for compilers, it is recommended to do this special packaging anyway. More details on customizing ICU are available in the User's Guide. The ICU Source Code Organization section of this readme.html gives a more complete description of the libraries.

Normally only the above ICU libraries need to be considered for packaging. The versionless symbolic links to these libraries are only needed for easier development. The X, Y and Z parts of the name are the version numbers of ICU. For example, ICU 2.0.2 would have the name libicuuc.so.20.2 for the common library. The exact format of the library names can vary between platforms due to how each platform can handles library versioning.

Important Notes About Using ICU

Using ICU in a Multithreaded Environment

Some versions of ICU require calling the u_init() function from uclean.h to ensure that ICU is initialized properly. In those ICU versions, u_init() must be called before ICU is used from multiple threads. There is no harm in calling u_init() in a single-threaded application, on a single-CPU machine, or in other cases where u_init() is not required.

In addition to ensuring thread safety, u_init() also attempts to load at least one ICU data file. Assuming that all data files are packaged together (or are in the same folder in files mode), a failure code from u_init() usually means that the data cannot be found. In this case, the data may not be installed properly, or the application may have failed to call udata_setCommonData() or u_setDataDirectory() which specify to ICU where it can find its data.

Since u_init() will load only one or two data files, it cannot guarantee that all of the data that an application needs is available. It cannot check for all data files because the set of files is customizable, and some ICU services work without loading any data at all. An application should always check for error codes when opening ICU service objects (using ucnv_open(), ucol_open(), C++ constructors, etc.).

ICU 3.4 and later

ICU 3.4 self-initializes properly for multi-threaded use. It achieves this without performance penalty by hardcoding the core Unicode properties data, at the cost of some flexibility. (For details see Jitterbug 4497.)

u_init() can be used to check for data loading. It tries to load the converter alias table (cnvalias.icu).

ICU 2.6..3.2

These ICU versions require a call to u_init() before multi-threaded use. The services that are directly affected are those that don't have a service object and need to be fast: normalization and character properties.

u_init() loads and initializes the data files for normalization and character properties (unorm.icu and uprops.icu) and can therefore also be used to check for data loading.

ICU 2.4 and earlier

ICU 2.4 and earlier versions were not prepared for multithreaded use on multi-CPU platforms where the CPUs implement weak memory coherency. These CPUs include: Power4, Power5, Alpha, Itanium. u_init() was not defined yet.

Using ICU in a Multithreaded Environment on HP-UX

If you are building ICU with a newer aCC compiler and you are planning on using the older <iostream.h> instead of the newer <iostream>, you will need to use a special configure flag before building ICU. By default, the aCC -AA flag is used on HP-UX when the compiler supports that option in order to make ICU thread safe with RogueWave and other libraries using the 2.0 Standard C++ library. Your applications that use ICU will also need to use the -AA compiler flag. To turn off this behavior in ICU, you will need to use the --with-iostream= old configure option when you first use runConfigureICU.

Using ICU in a Multithreaded Environment on Solaris

ICU's tests may hang on Solaris 8 and Earlier

ICU's tests use usleep(), which is multithread unsafe on versions of Solaris before version 9. This does not mean that ICU is not thread safe because only ICU's test code uses usleep(). The sleep() and nanosleep() functions could be used in ICU's multithreaded tests, but sleep() and nanosleep() are not a stable API between versions of Solaris. Solaris 9 fixes usleep so that it is multithread safe.

This hanging behavior tends to appear on multi-CPU machines. Single CPU Solaris 8 machines do not seem to show this behavior.

In a future version of ICU, we hope to find a portable solution to this problem that will work between the modern versions of Solaris.

Solaris Deadlock Issues in Solaris 8 (2.8) and Earlier

Solaris 8, and earlier, has outstanding thread deadlocking issues that may be problematic for applications using either native, or POSIX, threading on these platforms. Sun states that Solaris 9 does not have the deadlock problems. Deadlocks may occur either during initialization of the Solaris threading library, or at any other time.

Sun Microsystems has provided a Sun Alert Notification regarding the issue. Users should consider applying the latest OS patches to their Solaris installations in order to help avoid deadlock. Further information regarding the issue, and links to applicable patches, may be found at:

[1] "Applications Linked to libthread May Hang", Sun Alert Notification, Sun Microsystems, Inc., 04-Sep-2002
http://sunsolve.sun.com/search/document.do?assetkey=1-26-46867

Sun is not providing patches for Solaris 6 (2.6), or earlier.

Sun states that by applying the patch users will avoid the deadlock issues. However, with all applicable patches applied, deadlock may still be seen, as demonstrated by the ICU Mutex unit tests. The unit test will hang indefinitely. No bug exists in ICU. However, a latent bug still exists in Solaris, which Sun Microsystems has yet to resolve. In order to avoid this, users are suggested to modify their LD_LIBRARY_PATH according to the guidelines specified by Sun Microsystems in the Sun Alert Notification.

Linking on Solaris

In order to avoid synchronization and threading issues, developers are suggested to strictly follow the compiling and linking guidelines for multithreaded applications, specified in the following document from Sun Microsystems. Most notably, pay strict attention to the following statements from Sun:

To use libthread, specify -lthread before -lc on the ld command line, or last on the cc command line.

To use libpthread, specify -lpthread before -lc on the ld command line, or last on the cc command line.

Failure to do this may cause spurious lock conflicts, recursive mutex failure, and deadlock.

[2] "Solaris Multithreaded Programming Guide, Compiling and Debugging", Sun Microsystems, Inc., Apr 2004
http://docs.sun.com/db/doc/806-6867/6jfpgdcob?a=view

Windows Platform

If you are building on the Win32 platform, it is important that you understand a few of the following build details.

DLL directories and the PATH setting

As delivered, the International Components for Unicode build as several DLLs, which are placed in the "<ICU>\bin" directory. You must add this directory to the PATH environment variable in your system, or any executables you build will not be able to access International Components for Unicode libraries. Alternatively, you can copy the DLL files into a directory already in your PATH, but we do not recommend this. You can wind up with multiple copies of the DLL and wind up using the wrong one.

Changing your PATH

• Windows 2000/XP: Use the System Icon in the Control Panel. Pick the "Advanced" tab. Select the "Environment Variables..." button. Select the variable PATH in the lower box, and select the lower "Edit..." button. In the "Variable Value" box, append the string ";<ICU>\bin" to the end of the path string. If there is nothing there, just type in "<ICU>\bin". Click the Set button, then the OK button.
• Windows 95/98/ME: Edit the autoexec.bat, and add the following line to the end of file, "SET PATH=%PATH%;<ICU>\bin"

Note: When packaging a Windows application for distribution and installation on user systems, copies of the ICU DLLs should be included with the application, and installed for exclusive use by the application. This is the only way to insure that your application is running with the same version of ICU, built with exactly the same options, that you developed and tested with. Refer to Microsoft's guidelines on the usage of DLLs, or search for the phrase "DLL hell" on msdn.microsoft.com.

UNIX Type Platform

If you are building on a UNIX platform, and if you are installing ICU in a non-standard location, you may need to add the location of your ICU libraries to your LD_LIBRARY_PATH or LIBPATH environment variable (or the equivalent runtime library path environment variable for your system). The ICU libraries may not link or load properly without doing this.

Note that if you do not want to have to set this variable, you may instead use the --enable-rpath option at configuration time. This option will instruct the linker to always look for the libraries where they are installed. You will need to use the appropriate linker options when linking your own applications and libraries against ICU, too. Please refer to your system's linker manual for information about runtime paths. The use of rpath also means that when building a new version of ICU you should not have an older version installed in the same place as the new version's installation directory, as the older libraries will used during the build, instead of the new ones, likely leading to an incorrectly build ICU. This is the proper behavior of rpath.

Platform Dependencies

Porting To A New Platform

If you are using ICU's Makefiles to build ICU on a new platform, there are a few places where you will need to add or modify some files. If you need more help, you can always ask the icu-support mailing list. Once you have finished porting ICU to a new platform, it is recommended that you contribute your changes back to ICU via the icu-support mailing list. This will make it easier for everyone to benefit from your work.

Data For a New Platform

For some people, it may not be necessary for completely build ICU. Most of the makefiles and build targets are for tools that are used for building ICU's data, and an application's data (when an application uses ICU resource bundles for its data).

Data files can be built on a different platform when both platforms share the same endianness and the same charset family. This assertion does not include platform dependent DLLs/shared/static libraries. For details see the User Guide ICU Data chapter.

ICU 2.8 removes the requirement that ICU be completely built in the native operating environment. It adds the icuswap tool which can be run on any platform to turn binary ICU data files from any one of the three formats into any one of the other data formats. This allows a application to use ICU data built anywhere to be used for any other target platform.

WARNING! Building ICU without running the tests is not recommended. The tests verify that ICU is safe to use. It is recommended that you try to completely port and test ICU before using the libraries for your own application.

Adapting Makefiles For a New Platform

Try to follow the build steps from the UNIX build instructions. If the configure script fails, then you will need to modify some files. Here are the usual steps for porting to a new platform:

1. Create an mh file in icu/source/config/. You can use mh-linux or a similar mh file as your base configuration.
2. Modify icu/source/aclocal.m4 to recognize your platform's mh file.
3. Modify icu/source/configure.in to properly set your platform C Macro define.
4. Run autoconf in icu/source/ without any options. The autoconf tool is standard on most Linux systems.
5. If you have any optimization options that you want to normally use, you can modify icu/source/runConfigureICU to specify those options for your platform.
6. Build and test ICU on your platform. It is very important that you run the tests. If you don't run the tests, there is no guarentee that you have properly ported ICU.

Platform Dependent Implementations

The platform dependencies have been mostly isolated into the following files in the common library. This information can be useful if you are porting ICU to a new platform.

• unicode/platform.h.in (autoconf'ed platforms)
  unicode/pXXXX.h (others: pwin32.h, pmacos.h, ..): Platform-dependent typedefs and defines:
  
  
  • XP_CPLUSPLUS for C++ only.
  • Generic types like UBool, int8_t, int16_t, int32_t, int64_t, uint64_t etc.
  • U_EXPORT and U_IMPORT for specifying dynamic library import and export
  • <iostream> usability
  
  
• unicode/putil.h, putil.c: platform-dependent implementations of various functions that are platform dependent:
  
  
  • uprv_isNaN, uprv_isInfinite, uprv_getNaN and uprv_getInfinity for handling special floating point values.
  • uprv_tzset, uprv_timezone, uprv_tzname and time for getting platform specific time and time zone information.
  • u_getDataDirectory for getting the default data directory.
  • uprv_getDefaultLocaleID for getting the default locale setting.
  • uprv_getDefaultCodepage for getting the default codepage encoding.
  
  
• umutex.h, umutex.c: Code for doing synchronization in multithreaded applications. If you wish to use International Components for Unicode in a multithreaded application, you must provide a synchronization primitive that the classes can use to protect their global data against simultaneous modifications. See Users' guide for more information.
  
  
  • We supply sample implementations for Windows, Sun Solaris, Linux, AIX, HP-UX, BSD, Mac OS X, z/OS and many others.
  
  
• umapfile.h, umapfile.c: functions for mapping or otherwise reading or loading files into memory. All access by ICU to data from files makes use of these functions.
  
  
• Using platform specific #ifdef macros are highly discouraged outside of the scope of these files. When the source code gets updated in the future, these #ifdef's can cause testing problems for your platform.

Copyright © 1997-2006 International Business Machines Corporation and others. All Rights Reserved.
IBM Globalization Center of Competency - San José
5600 Cottle Road
San José, CA 95193
USA