International Components for Unicode
ReadMe

Version: 2001-Aug-02
Copyright © 1995-2001 International Business Machines Corporation and others. All Rights Reserved.

Table of Contents

• Introduction
• Late Breaking News And What Is New?
  • Support for Unicode 3.1
  • License Change from IPL to the X license
  • Collation Improvements
• What the International Components for Unicode Contain
• Platform Dependencies
• How to Build And Install ICU
  • Supported Platforms
  • Windows
  • Unix
  • OS/390 (zSeries)
  • OS/400 (iSeries)
• Important Notes About Using ICU
  • Windows Platform
  • Unix Type Platforms
• Getting More Information About ICU
• Submitting Comments, Requesting Features and Reporting Bugs

Introduction

Today's software market is a global one in which it is desirable to develop and maintain one application that supports a wide variety of national languages. International Components for Unicode provides the following tools to help you write language independent applications:

• Support for the latest Unicode standard
• Resource bundles for storing and accessing localized information
• Number formatters for converting binary numbers into text strings for meaningful display
• Date and time formatters for converting internal time data into text strings for meaningful display
• Message formatters for putting together sequences of strings, numbers dates and other format to create messages
• Text collation supporting language sensitive comparison of strings
• Text boundary analysis for finding characters, word and sentence boundaries
• Changing simple data files rather than modifying program code easily localizes applications written using these tools
• Over 160 locales supported. Visit the LocaleExplorer on the ICU website for a demonstration and a full list of supported locales or see the index file with the supported locales.

It is possible to support additional locales by adding more locale data files, with no code changes. Please refer to POSIX programmer's Guide for details on what the ISO locale ID means.

This document will go into more detail on how to build and install ICU on your machine. Once you start using ICU, the Where To Find More Information section of this document will be very helpful resource.

Your comments are important to making this release successful. We are committed to fixing any bugs, and will also use your feedback to help plan future releases.

IMPORTANT: Please make sure you understand the Copyright and License Information.

Late Breaking News And What Is New?

For more news about this release, see the online release notes.

Support for Unicode 3.1

The ICU 2.0 data has been upgraded to support Unicode 3.1. This means that the character property data and normalization has changed. Recent versions of ICU already supported Unicode 3.0 data with UTF-16 surrogate pairs.

License Change

The ICU projects (ICU4C and ICU4J) have changed their licenses from the IPL (IBM Public License) to the X license. The X license is a non-viral and recommended free software license that is compatible with the GNU GPL license. This is effective starting with release 1.8.1 of ICU4C and release 1.3.1 of ICU4J. All previous ICU releases will continue to utilize the IPL. New ICU releases will adopt the X license. The users of previous releases of ICU will need to accept the terms and conditions of the X license in order to adopt the new ICU releases.

The main effect of the change is to provide GPL compatibility. The X license is listed as GPL compatible, see the gnu page at http://www.gnu.org/philosophy/license-list.html#GPLCompatibleLicenses.

The text of the X license is available at http://www.x.org/terms.htm. The IBM version contains the essential text of the license, omitting the X-specific trademarks and copyright notices.

For more details please see the press announcement and the Project FAQ.

Collation Improvements

The collation framework has been reimplemented to make it faster, Unicode Collation Algorithm compliant, and to make the locale-specific collation data smaller (by separating it from the shared UCA data).
Sort keys and even some collation results have changed from ICU 1.6 and ICU 1.7.
For details, see our collation design document.

What the International Components for Unicode Contain

There are two ways to download the ICU releases,

• Official Release Snapshot:
  If you want to use ICU (as opposed to developing it), you should download an official packaged version of the ICU source code. These versions are tested more thoroughly than day-to-day development builds of the system, and they are packaged in zip and tar files for convenient download. These packaged files can be found at http://oss.software.ibm.com/icu/download/.
  If packaged snapshot is named ICUXXXXXX.zip or ICUXXXXXX.tgz, XXXXXX is the release version number.
  Please unzip this file. It will reconstruct the source directory, including anonymous CVS control directories (see below).
• CVS Source Repository:
  If you are interested in developing features, patches, or bug fixes for ICU, you should probably be working with the latest version of the ICU source code. You will need to check the code out of our CVS repository to ensure that you have the most recent version of all of the files. There are several ways to do this:
  • WebCVS:
    If you want to browse the code and only make occasional downloads, you may want to use WebCVS. It provides a convenient, web-based interface for browsing and downloading the latest version of the ICU source code and documentation. You can also view each file's revision history, display the differences between individual revisions, determine which revisions were part of which official release, and so on.
  • WinCVS:
    If you will be doing serious work on ICU, you should probably install a CVS client on your own machine so that you can do batch operations without going through the WebCVS interface. On Windows, we suggest the WinCVS client. The following is the example instruction on how to download ICU via WinCVS:
    1. Install the WinCVS client, which you can download from the WinCVS home page.
    2. In the WinCVS preferences, specify your CVSRoot to be ":pserver:anoncvs@oss.software.ibm.com:/usr/cvs/icu"
       with the password "anoncvs". To enter the CVSRoot value, select "Preferences" from the "Cvs Admin" pull-down menu. Authentication should be set to "'passwd' file on the cvs server".
    3. To "extract" the most recent version of ICU, select "Checkout module" from the "Cvs Admin" menu. Specify "icu" for the module name.
  • CVS command line:
    You can also check out the repository anonymously on UNIX using the following commands, after first setting your CVSROOT to point to the ICU repository:
    
    export CVSROOT=:pserver:anoncvs@oss.software.ibm.com:/usr/cvs/icu
    cvs login CVS password: anoncvs
    cvs checkout icu
    cvs logout

For more details on how to download ICU directly from the web site, please also see http:/oss.software.ibm.com/icu/download/

Below, $Root is the placement of the icu directory in your file system, like "drive:\...\icu" in your environment. "drive:\..." stands for any drive and any directory on that drive that you chose to install icu into.

Platform Dependencies

The platform dependencies have been mostly isolated into the following files in the common library. This information can be useful if you are porting ICU to a new platform.

• unicode/platform.h.in (autoconf'ed platforms)
  unicode/pXXXX.h (others: pwin32.h, pmacos.h, ..): Platform-dependent typedefs and defines:
  
  
  • XP_CPLUSPLUS for C++ only.
  • TRUE and FALSE, UBool, int8_t, int16_t etc.
  • U_EXPORT and U_IMPORT for specifying dynamic library import and export
  
  
• unicode/putil.h, putil.c: platform-dependent implementations of various functions that are platform dependent:
  
  
  • uprv_isNaN, uprv_isInfinite, uprv_getNaN and uprv_getInfinity for handling special floating point values.
  • uprv_tzset, uprv_timezone, uprv_tzname and time for getting platform specific time and timezone information.
  • u_getDataDirectory for getting the default data directory.
  • uprv_getDefaultLocaleID for getting the default locale setting.
  • uprv_getDefaultCodepage for getting the default codepage encoding.
  
  
• umutex.h, umutex.c: Code for doing synchronization in multithreaded applications. If you wish to use International Components for Unicode in a multithreaded application, you must provide a synchronization primitive that the classes can use to protect their global data against simultaneous modifications. See Users' guide for more information.
  
  
  • We supply sample implementations for WinNT, Win95, Win98, Sun/Solaris, RedHat/Linux, HP-UX and for AIX on an RS/6000.
  
  
• unicode/udata.h, udata.c: The data-accessing interface in ICU is implemented such that there is a lot of flexibility for reading a data file. Each platform can tune the performance of file accessing for its environment by choosing to implement one of the following options:
  
  
  • DLL
  • Memory map
  • Individual files
  
  
• For the Intltest test suite, intltest.cpp in "icu/source/test/intltest/" contains the method pathnameInContext, which must also be adapted to any new platform.
• Using platform specific #ifdef macros are highly discouraged outside of the scope of these files. When the source code gets updated in the future, these #ifdef's can cause testing problems for your platform.

It is possible to build each library individually. They must be built in the following order:

1. stubdata
2. common
3. i18n
4. toolutil
5. makeconv
6. genrb
7. gentz
8. genccode
9. gennames
10. genuca
11. gennorm
12. makedata (a project on Windows, or source/data/Makefile on Unix)
13. ctestfw, intltest and cintltst, if you want to run the test suite.

How To Build And Install ICU

Supported Platforms

Key to testing frequency

Reference platform

ICU will work on these platforms with these compilers

Regularly tested

ICU should work on these platforms with these compilers

Rarely tested

ICU may not work on these platforms

How To Build And Install On Windows

Building International Components for Unicode requires:

• Microsoft NT 3.51 and above, or Windows 95 and above
• Microsoft Visual C++ 6.0 (Service Pack 2 is required to work with the release build of max speed optimization).

The steps are:

1. Unzip the icu-XXXX.zip file, type "unzip -a icu-XXXX.zip -d drive:\directory" under command prompt or use WinZip. drive:\directory\icu is the root ($Root) directory (you may but don't need to place "icu" into another directory). If you change the root, you will change the project settings accordingly in EACH makefile in the project, updating the "include" and "library" paths.
2. Set the environment variable ICU_DATA to the full pathname of the data directory. The trailing "\" is required after the directory name (e.g. "$Root\source\data\" will work, but the value "$Root\source\data" is not acceptable). This environment variable indicates where the locale data files and conversion mapping tables are located.
3. Be sure that the ICU binary directory, $Root\bin\, is included in the PATH environment variable. The tests may not work without the DLL files in the path.
4. Set the TZ environment variable to PST8PDT. The tests will not work in any other timezone.
5. Use Microsoft Visual C++ 6.0 to open the "$Root\source\allinone\allinone.dsw" workspace (This workspace includes all the International Components for Unicode libraries, necessary ICU building tools, and the intltest and cintltest test suite projects). Please see the note below if you want to build from the command line instead.
6. Set the active Project to the "all" project. To do this: Choose "Project" menu, and select "Set active project". In the submenu, select the "all" workspace.
7. Set the active configuration to "Win32 Debug" or "Win32 Release" (See note below).
8. Choose the "Build" menu and select "Rebuild All". If you want to build the Debug and Release at the same time, see the note below.
9. Run the C++ test suite, "intltest". To do this: set the active project to "intltest", and press F5 to run it.
10. Run the C test suite, "cintltst". To do this: set the active project to "cintltst", and press F5 to run it.
11. Make sure that both "cintltst" and "intltest" passed without any errors. The return codes are non-zero when they do not pass. Visual C++ will display the return codes in the debug tag of the output window. When "intltest" and "cintltest" return 0, it means that everything is installed correctly. You can press Ctrl+F5 on the test project to run the test and see what error messages were displayed (if any tests failed).
12. Reset the TZ environment variable to its original value, unless you plan on testing ICU any further.
13. You are now able to develop applications with ICU.

Using MSDEV At The Command Line Note: You can build ICU from the command line. Assuming that you have properly installed Microsoft Visual C++ to support command line execution, you can run the following command, 'msdev $Root\source\allinone\allinone.dsw /MAKE "ALL"'.

Setting Active Configuration Note: To set the active configuration, two different possibilities are:

• Choose "Build" menu, select "Set Active Configuration", and select "Win32 Release" or "Win32 Debug".
• Another way is to select "Customize" in the "Tools" menu, select the "Toolbars" tab, enable "Build" instead of "Build Minibar", and click on "Close". This will bring up a toolbar which you can move aside the other permanent toolbars at the top of the MSVC window. The advantage is that you now have an easy-to-reach pop-up menu that will always show the currently selected active configuration. Or, you can drag the project and configuration selections and drop them on the menu bar for later selection.

Batch Configuration Note: If you want to build the Debug and Release configurations at the same time, choose "Build" menu and select "Batch Build..." instead (and mark all configurations as checked), then click the button named "Rebuild All". The "all" workspace will build all the test programs as well as the tools for generating binary locale data files. The "makedata" project will be run automatically to convert the locale data files from text format into icudata.dll.

How To Build And Install On Unix

Building International Components for Unicode on Unix requires:

A UNIX C++ compiler, (gcc, cc, xlc_r, etc...) installed on the target machine. A recent version of GNU make (3.7+). For a list of OS/390 tools please view the OS/390 build section of this document for further details.

The steps are:

1. Decompress the icuXXXX.tar (or icuXXXX.tgz) file and use pax.
2. Before running the test programs or samples, please set the environment variable ICU_DATA, the full pathname of the data directory, to indicate where the locale data files and conversion mapping tables are. If this variable is not set, the default user data directory will be used. The trailing "/" is required after the directory name (e.g. "$Root/source/data/" will work, but the value "$Root/source/data" is not acceptable). When you are running individual tests, the TZ environment variable needs to be set to PST8PDT. Normally "make check" does this for you automatically.
3. Change directory to the "icu/source".
4. If it is not already set, please set the executable flag for the following files (by executing 'chmod +x' command): runConfigureICU, configure, install-sh and config.*,
5. Run the runConfigureICU script for your platform. If you are not using the runConfigureICU script or your platform is not supported by the script, you need to set your CC, CXX, CFLAGS and CXXFLAGS environment variables, and type "./configure". You can type "./configure --help" to print the available options.
6. Type "gmake" to compile the libraries and all the data files.
   Note: On OS/390, both IEEE binary floating point and native S/390 hexadecimal floating point calculations are supported. The default is to build with native floating-point support. Please set the environment variable IEEE390=1 if you would like to make the ICU DLLs with IEEE floating point support.
7. Optionally, type "gmake check" to verify the test suite.
8. Type "gmake install" to install.

Some platforms use package management tools to control the installation and uninstallation of files on the system, as well as the integrity of the system configuration. You may want to check if ICU can be packaged for your package management tools by looking into the "packaging" directory. (Please note that if you are using a snapshot of ICU from CVS, it is probable that the packaging scripts or related files are not up to date with the contents of ICU at this time, so use them with caution.)

OS/390 (zSeries) Platform

If you are building on the OS/390 UNIX System Services platform, it is important that you understand a few details:

• The gnu utilities gmake and gzip/gunzip are needed and can be obtained for OS/390 from z/OS Unix - Tools and Toys. Documentation on these tools can be found at the Open Source Software for OS/390 UNIX Red Book.
• Encoding considerations: The source code assumes that it is compiled with codepage ibm-1047 (to be exact, the UNIX System Services variant of it). The pax command converts all of the source code files from ASCII to codepage ibm-1047 (USS) EBCDIC. However, some files are binary files and must not be converted, or must be converted back to their original state. You can use the unpax-icu.sh script to do this for you automatically. It will unpackage the tar file and convert all the necessary files for you automatically. The files that must not be converted to ibm-1047 are the following:
  • All UTF-8 files
  • icu/data/*.brk
  • icu/source/test/testdata/uni-text.bin
  • icu/source/test/testdata/th18057.txt
  Such a conversion can be done using iconv:
  iconv -f IBM-1047 -t ISO8859-1 uni-text.bin > uni-text.bin
• DLL directories and the LIBPATH setting: Building and testing ICU needs the ICU libraries on the LIBPATH. In other words, the LIBPATH should contain (each path prepended with the root directory that contains the icu directory):
  • icu/source/common
  • icu/source/i18n
  • icu/source/tools/ctestfw
  • icu/source/tools/toolutil
  • icu/source/extra/ustdio

• OS/390 supports both native S/390 hexadecimal floating point and, (with Version 2.6 and later) IEEE binary floating point. This is a compile time option. Applications built with IEEE should use ICU dlls that are built with IEEE (and vice versa). The environment variable IEEE390=1 will cause the OS/390 version of ICU to be built with IEEE floating point. The default is native hexadecimal floating point.
  Important: Currently (ICU 1.4.2), native floating point support is sufficient for codepage conversion, resource bundle and UnicodeString operations, but the Format APIs, especially ChoiceFormat, require IEEE binary floating point.

  Examples for configuring ICU:
  Debug build: IEEE390=1 ./configure
  Release build: CFLAGS=-2 IEEE390=1 ./configure

• Since the default make on OS/390 is not gmake, the pkgdata tool requires that the "make" command is aliased to your installed version of gmake.
• The makedep executable that is used with the OS/390 ICU build process is not shipped with ICU. It is available at the z/OS Unix - Tools and Toys site. The PATH environment variable should be updated to contain the location of this executable prior to build. Alternatively, makedep may be moved into an existing PATH directory.
• To run all of the tests for ICU, use "gmake check". When running individual tests of the test suite, the TZ environment variable should be set to export TZ="PST8PDT" so that time zone comparisons are correct.

OS/390 Batch (PDS) support

By default, ICU builds its libraries into the HFS. However, there is a 390-specific switch to build some libraries into PDS files. The switch is the environmental variable OS390BATCH, and if set, the following libraries are built into PDS files: libicuucXX.dll, libicudtXXe.dll, libicudtXXe_390.dll, and libtestdata.dll. Turning on OS390BATCH does not turn off the normal HFS build, thus the HFS dlls will always be created.

The names of the PDS files are determined by the value of the environmental variables LOADMOD and LOADEXP. These variables must contain the target PDS names whenever the OS390BATCH variable is set. LOADMOD is the library (.dll) target dataset and LOADEXP is the side deck (.x) target dataset.

The PDS member names are as follows:

IXMICUUC --> libicuucXX.dll
IXMICUDA --> libicudtXXe.dll
IXMICUD1 --> libicudtXXe_390.dll
IXMICUTE --> libtestdata.dll

Example PDS attributes are as follows:

Data Set Name . . . : USER.ICU.LOAD
General Data
Management class. . : **None**
Storage class . . . : BASE
Volume serial . . . : TSO007
Device type . . . . : 3390
Data class. . . . . : LOAD
Organization  . . . : PO
Record format . . . : U
Record length . . . : 0
Block size  . . . . : 32760
1st extent cylinders: 40
Secondary cylinders : 59
Data set name type  : PDS

Data Set Name . . . : USER.ICU.EXP
General Data
Management class. . : **None**
Storage class . . . : BASE
Volume serial . . . : TSO007
Device type . . . . : 3390
Data class. . . . . : **None**
Organization  . . . : PO
Record format . . . : FB
Record length . . . : 80
Block size  . . . . : 3200
1st extent cylinders: 3
Secondary cylinders : 3
Data set name type  : PDS

OS/400 (iSeries) Platform

ICU Reference Release 1.8.1 contains partial support for the 400 platform, but additional work by the user is currently needed to get it to build properly. A future release of ICU should work out-of-the-box under OS/400.

• Requirements:
  • QSHELL interpreter installed (install base option 30, operating system)
  • QShell Utilities, PRPQ 5799-XEH (not required for V4R5)
  • ILE C++ for AS/400, PRPQ 5799-GDW (the latest cum package and PTF SF62241 must be installed)
  • GNU facilities (You can get the GNU facilities for OS/400 from http://www.as400.ibm.com/developer/porting/gnu_utilities.html).
• Build environment setup:
  1. Create AS400 target library. This library will be the target for the resulting modules, programs and service programs. You will specify this library on the OUTPUTDIR environment variable in step 2.
     
     CRTLIB LIB(libraryname)
     
     
  2. Set up the following environment variables in your build process (use the libraryname from the previous step)
     ADDENVVAR ENVVAR(ICU_DATA) VALUE('/icu/source/data')
     ADDENVVAR ENVVAR(CC) VALUE('/usr/bin/icc')
     ADDENVVAR ENVVAR(CXX) VALUE('/usr/bin/icc')
     ADDENVVAR ENVVAR(MAKE) VALUE('/usr/bin/gmake')
     ADDENVVAR ENVVAR(OUTPUTDIR) VALUE('libraryname') identifies target as400 library for *module, *pgm and *srvpgm objects
     
     
  3. Add QCXXN, to your build process library list. This results in the resolution of CRTCPPMOD used by the icc compiler
  4. In order to get the tests to run correctly, the QUTCOFFSET needs to be set to the Pacific Time Zone offset.
     
     To check your QUTCOFFSET:
     DSPSYSVAL SYSVAL(QUTCOFFSET)
     
     To change your QUTCOFFSET:
     
     CHGSYSVAL SYSVAL(QUTCOFFSET) VALUE('-0800') or -0700 for daylight savings.
     
     
  5. Run 'CHGJOB CCSID(37)'
  6. Run 'QSH'
  7. Run gunzip on the ICU source code compressed tar archive (icu-X-Y.tar.gz or icu-X-Y.tgz).
  8. Run unpax-icu.sh on the tar file from the ICU download page.
  9. Change your current directory to icu/source.
  10. Configure the Makefiles with the as/400 configure script from the ICU download page. Note: Verify that the mh-os400 configure file is used.
      • Run 'configure --host=as400-os400'
      • The 'clean' and 'install' targets will not work without changes because of symbolic links. To delete the target module, program, or service programs replace rm -rf with $(RMV), and in the library installation targets (install-library) change $(INSTALL) to $(INSTALL-S).
  11. Run 'gmake -e'. The '-e' option is needed to pickup the compilers.
  12. Run 'gmake -e check' to run the tests.

Important Notes About Using ICU

Windows Platform

If you are building on the Win32 platform, it is important that you understand a few of the following build details.

DLL directories and the PATH setting

As delivered, the International Components for Unicode build as several DLLs. These DLLs are placed in the "icu\bin" directory. You must add this directory to the PATH environment variable in your system, or any executables you build will not be able to access International Components for Unicode libraries. Alternatively, you can copy the DLL files into a directory already in your PATH, but we do not recommend this. You can wind up with multiple copies of the DLL and wind up using the wrong one.

Changing your PATH

• Windows 2000: Use the System Icon in the Control Panel. Pick the "Advanced" tab. Select the "Environment Variables..." button. Select the variable PATH in the lower box, and select the lower "Edit..." button. In the "Variable Value" box, append the string ";$Root\bin" to the end of the path string. If there is nothing there, just type in "$Root\bin". Click the Set button, then the OK button.
• Windows NT: Use the System Icon in the Control Panel. Pick the "Environment" tab, and select the variable PATH in the lower box. In the "value" box, append the string ";$Root\bin" at the end of the path string. If there is nothing there, just type in "$Root\bin". Click the Set button, then the OK button.
• Windows 95/98/ME: Edit the autoexec.bat, and add the following line to the end of file, "SET PATH=%PATH%;$Root\bin"

Linking with Runtime libraries

All the DLLs link with the C runtime library "Debug Multithreaded DLL" or "Multithreaded DLL." (This is changed through the Project Settings dialog, on the C/C++ tab, under Code Generation.) It is important that any executable or other DLL you build which uses the International Components for Unicode DLLs links with these runtime libraries as well. If you do not do this, you will get random memory errors when you run the executable.

Unix Type Platform

If you are building on a Unix platform, it is important that you add the location of your ICU libraries (including the data library) to your LD_LIBRARY_PATH environment variable. The ICU libraries may not link or load properly without doing this.

Getting More Information About ICU

Submitting Comments, Requesting Features and Reporting Bugs

To submit comments, request features and report bugs, please contact us. The best forum is the ICU mailing list. See the information on how to browse and join the list. If you find a bug in the code that has not been submitted and/or fixed yet, then please submit a jitterbug.

Copyright © 1997-2001 International Business Machines Corporation and others. All Rights Reserved.
IBM Center for Emerging Technologies Silicon Valley,
10275 N De Anza Blvd., Cupertino, CA 95014
All rights reserved.