International Components for Unicode
ReadMe

Version: 2000-Dec-15
Copyright © 1997-2000 International Business Machines Corporation and others. All Rights Reserved.



Contents

Late Breaking News And What Is New?

Introduction

Today's software market is a global one in which it is desirable to develop and maintain one application that supports a wide variety of national languages. International Components for Unicode provides the following tools to help you write language independent applications:

It is possible to support additional locales by adding more locale data files, with no code changes.

Please refer to POSIX programmer's Guide for details on what the ISO locale ID means.

Your comments are important to making this release successful. We are committed to fixing any bugs, and will also use your feedback to help plan future releases.

IMPORTANT: Please make sure you understand the Copyright and License information.


What the International Components for Unicode Contain

There are two ways to download the ICU releases,

For more details on how to download ICU directly from the web site, please also see http:/oss.software.ibm.com/icu/download/

Below, $Root is the placement of the icu directory in your file system, like "drive:\...\icu" in your environment. "drive:\..." stands for any drive and any directory on that drive that you chose to install icu into.

The following files describe the code drop

readme.html

Describes the International Components for Unicode (this file)

license.html

Contains IBM's public license


The following directories contain source code and data files

$Root/source/common/

The core Unicode and support functionality, such as resource bundles, character properties, locales, codepage conversion, and normalization.Unicode, Locale, UnicodeString.

$Root/source/i18n/

Modules in i18n are generally the more data-driven, that is to say resource bundle driven, components. These deal with higher level internationalization issues such as formatting, collation, text break analysis, and transliteration.

$Root/source/test/intltest/

A test suite including all C++ APIs. For information about running the test suite, see the users' guide.

$Root/source/test/cintltst/

A test suite written in C, including all C APIs. For information about running the test suite, see the users' guide.

$Root/data/
    This directory contains the source data in text format, which is compiled into binary form during the ICU build process. The output from these files is stored in $Root/source/data/build while awaiting further packaging.
  • unidata/ This directory contains the Unicode data files. Please see http://www.unicode.org/ for more information.
  • Resource Bundle sources .txt files containing ICU language and culture-specific localization data. Two special bundles are root which is the fallback data and parent of other bundles, and index which contains a list of installed bundles. resfiles.txt contains the list of resource bundle files.

    Also here are transliteration bundles, and the list of installed transliteration files in translit_index.txt.

    All resource bundles are compiled into .res files. ucmfiles.txt contains the list of converter files.

  • Code page converter tables .ucm files containing mappings to and from unicode. These are compiled into .cnv files.
  • convrtrs.txt is the alias mapping table from various converter name formats to ICU internal format and vice versa. It produces cnvalias.dat.
  • timezone.txt is a generated file which is compiled into tz.dat, containing time zone information.

$Root/source/data

This directory is where the final, packaged version of the ICU binary data ends up. If the ICU_DATA environment variable is used, then it should be set to this directory. The intermediate individual data files (.res, .cnv) are kept in the subdirectory "$Root/source/data/build" prior to packaging.

$Root/source/tools

Tools for generating the data files. Data files are generated by invoking $Root/source/data/build/makedata.bat on Win32 or $Root/source/make on unix.

$Root/source/samples

Various sample programs that use ICU

$Root/source/extra

Non-supported API additions. Currently, it contains the 'ustdio' file i/o library

$Root/source/layout

Contains the ICU layout engine (not a rasterizer).

$Root/packaging
$Root/debian

These directories contain scripts and tools for packaging the final ICU build for various release platforms.

$Root/source/config

Contains helper makefiles for platform specific build commands. Used by 'configure'.

$Root/source/allinone

Contains top level ICU project files, for instance to build all of ICU under one MSVC project.


Platform Dependencies

The platform dependencies have been mostly isolated into the following files:

Important Installation Notes

Win32 Platform

BREAKING NEWS: Library names are changed and libraries are moved on Win32

As it was previously mentioned and proposed on ICU list, ICU libraries on Win32 are now renamed and relocated. The following changes took place:

  1. in icu\lib:
  2. in icu\bin:

Also, ctestfw and ex toolutil (now icutu17) libraries appeared in icu\bin and icu\lib dirs, but this shouldn't concern regular users of icu.

What to do, how to cope?

When you first try to compile your programs with new version of icu, compilation will fail. The following steps are required:

  1. change your path from ...\icu\bin\debug or ...\icu\bin\release to just ...\icu\bin
  2. make the same change for MSVC executable directory setting (tools->options->directories)
  3. In all your .dsp files (project settings), on linker tab change input libraries according to the above scheme. You want to do it for both debug and release versions.

When a new, binary incompatible version appears after 1.7, the libraries will change the version number, so you will have to rename libraries in your projects again. However, this way you can have several versions of icu on the same machine and change the libraries for different programs without having to change path settings on your machine.

If you are building on the Win32 platform, it is important that you understand a few build details:

DLL directories and the PATH setting: As delivered, the International Components for Unicode build as several DLLs. These DLLs are placed in the directories "icu\bin\Debug" and "icu\bin\Release". You must add either of these directories to the PATH environment variable in your system, or any executables you build will not be able to access International Components for Unicode libraries. Alternatively, you can copy the DLL files into a directory already in your PATH, but we do not recommend this. You can wind up with multiple copies of the DLL and wind up using the wrong one.

To change your PATH: When you are not using the debug version, you will want to change the "Debug" part of the path to "Release" instead (the $Root is the root ICU installation directory e.g. drive:\installation-directory\icu).

Link with Runtime libraries: All the DLLs link with the C runtime library "Debug Multithreaded DLL" or "Multithreaded DLL." (This is changed through the Project Settings dialog, on the C/C++ tab, under Code Generation.) It is important that any executable or other DLL you build which uses the International Components for Unicode DLLs links with these runtime libraries as well. If you do not do this, you will seemingly get memory errors when you run the executable.

Collation Improvements

We are reimplementing the collation implementation to make it faster (much of this is done with ICU 1.7), to comply with the Unicode Collation Algorithm (ICU 1.8), and also to make the locale-specific collation data smaller (by separating it from the shared UCA data, also for ICU 1.8).
This means that sort keys and even some collation results are changing from ICU 1.6 and will change again for ICU 1.8.
For details, see our collation design document.

API documentation

We are updating our API documentation, generated JavaDoc-style from the public header files. It is available for online browsing as well as for download from the release folder.

Also, as a more general and comprehensive documentation, we are working on improving the ICU User Guide.

OS/390 Platform

Note that ICU 1.7 is not supported on OS/390! ICU 1.6 is supported, and ICU 1.8 is scheduled to be.

If you are building on the OS/390 UNIX System Services platform, it is important that you understand a few details:

OS/400 Platform

ICU Reference Release 1.4.0 contains partial support for the 400 platform, but additional work by the user is currently needed to get it to build completely. A future release of the ICU should work out-of-the-box under OS/400.

Note: About the NULL pointer checks
In common/ucnv.c and common/unistr.c (search for U_MAX_PTR), there are additional checks for NULL pointers. This is because pointer comparison works differently on the AS/400 architecture.

How To Build And Install ICU

How To Build And Install On Windows

Building International Components for Unicode requires:

The steps are:

  1. Unzip the icu-XXXX.zip file, type "unzip -a icu-XXXX.zip -d drive:\directory" under command prompt or use WinZip. drive:\directory\icu is the root ($Root) directory (you may but don't need to place "icu" into another directory). If you change the root, you will change the project settings accordingly in EACH makefile in the project, updating the "include" and "library" paths.
  2. Set the environment variable ICU_DATA to the full pathname of the data directory. The trailing "\" is required after the directory name (e.g. "$Root\source\data\" will work, but the value "$Root\source\data" is not acceptable). This environment variable indicates where the locale data files and conversion mapping tables are located.
  3. Be sure that the ICU binary directory, $Root\bin\[Release|Debug], is included in the PATH environment variable. The tests may not work without the DLL files in the path.
  4. Set the TZ environment variable to PST8PDT. The tests will not work in any other timezone.
  5. Use Microsoft Visual C++ 6.0 to open the "$Root\source\allinone\allinone.dsw" workspace (This workspace includes all the International Components for Unicode libraries, necessary ICU building tools, and the intltest and cintltest test suite projects).
  6. Set the active Project to the "all" project. To do this: Choose "Project" menu, and select "Set active project". In the submenu, select the "all" workspace.
  7. Set the active configuration to "Win32 Debug" or "Win32 Release" (See note below).
  8. Choose the "Build" menu and select "Rebuild All". If you want to build the Debug and Release configurations at the same time, choose "Build" menu and select "Batch Build..." instead (and mark all configurations as checked), then click the button named "Rebuild All". The "all" workspace will build all the test programs as well as the tools for generating binary locale data files. The "makedata" project will be run automatically to convert the locale data files from text format into icudata.dll.
  9. Run the C++ test suite, "intltest". To do this: set the active project to "intltest", and press F5 to run it.
  10. Run the C test suite, "cintltst". To do this: set the active project to "cintltst", and press F5 to run it.
  11. Make sure that both "cintltst" and "intltest" passed without any errors. The return codes are non-zero when they do not pass. Visual C++ will display the return codes in the debug tag of the output window. When "intltest" and "cintltest" return 0, it means that everything is installed correctly.
  12. Reset the TZ environment variable to its original value, unless you plan on testing ICU any further.
  13. You are now able to develop applications with ICU.

Note: To set the active configuration, two different possibilities are:

It is also possible to build each library individually, using the workspaces in each respective directory. They have to be built in the following order:

  1. common
  2. i18n
  3. makedata (which invokes makeconv, genrb, genccode, etc.)
  4. ctestfw
  5. intltest and cintltst, if you want to run the test suite.

How To Build And Install On Unix

There is a set of Makefiles for Unix that supports Linux w/gcc, Solaris w/gcc and Workshop CC, AIX w/xlc and OS/390 with C++.

Building International Components for Unicode on Unix requires:

A UNIX C++ compiler, (gcc, cc, xlc_r, etc...) installed on the target machine. A recent version of GNU make (3.7+). OS/390 gnu utilities for both make (gmake) and zip (gzip/gunzip) can be found at the MKS web site at http://www.mks.com. Please do a search on "os/390".

The steps are:

  1. Decompress the icuXXXX.tar (or icuXXXX.tgz) file.
  2. Before running the test programs or samples, please set the environment variable ICU_DATA, the full pathname of the data directory, to indicate where the locale data files and conversion mapping tables are. If this variable is not set, the default user data directory will be used. The trailing "/" is required after the directory name (e.g. "$Root/source/data/" will work, but the value "$Root/source/data" is not acceptable). The TZ environment variable does not need to be set.
  3. Change directory to the "icu/source".
  4. If it is not already set, please set the executable flag for the following files (by executing 'chmod +x' command): runConfigureICU, configure, install.sh and config.*,
  5. You also need to set other environment variables for different build systems. See the User Guide or the provided script.
  6. Type "./configure" or type "./configure --help" to print the available options.
  7. Type "make" to compile the libraries and all the data files. On OS/390, both IEEE binary floating point and native S/390 hexadecimal floating point calculations are supported. The default is to build with native floating-point support. Please set the environment variable IEEE390=1 if you would like to make the ICU DLLs with IEEE floating point support.
  8. Optionally, type "make check" to verify the test suite.
  9. Type "make install" to install.

Some platforms use package management tools to control the installation and uninstallation of files on the system, as well as the integrity of the system configuration. You may want to check if ICU can be packaged for your package management tools by looking into the "packaging" directory. (Please note that if you are using a snapshot of ICU from CVS, it is probable that the packaging scripts or related files are not up to date with the contents of ICU at this time, so use them with caution.)

It is also possible to build each library individually, using the Makefiles in each respective directory. They have to be built in the following order:

  1. common
  2. i18n
  3. makeconv
  4. genrb
  5. gentz
  6. genccode
  7. ctestfw
  8. intltest and cintltst, if you want to run the test suite.

Using Shared Data Libraries

HP/UX has a documented characteristic where the shl_unload() function always unloads a library, regardless of how many times the library has been loaded. Most operating systems reference-count libraries as they are opened. In the future (Jitterbug 414) this may be corrected in the ICU, but at present we work around this problem by simply NOT ever unloading shared libraries. This means that once a data library is loaded (ex: libicudata.sl) by a process, it cannot be unloaded and replaced without stopping and restarting the process.

Important Change Of Error Codes From Streaming Conversion Functions (change for ICU 1.6)

We have decided to make a semantic change to the conversion API which affects applications using ICU that are migrated to use ICU versions 1.6 or later compared to earlier ICU versions before 1.6:
The error code that is set from streaming conversion like

ucnv_fromUnicode() - ucnv_toUnicode()
ucnv_fromUChars()  - ucnv_toUChars()
scsu_compress()    - scsu_decompress()
when the target buffer is full but the source not empty is changed from U_INDEX_OUTOFBOUNDS_ERROR to U_BUFFER_OVERFLOW_ERROR. This change makes the error codes more consistent with their names and with their use in other icu APIs.

You need to test for this new error code if your code uses ICU for conversion and used the old error code. ucnv.h and scsu.h are updated with this information. Please search in your source code for U_INDEX_OUTOFBOUNDS_ERROR. If it is used with the above functions (not with ucnv_getNextUChar()), then you need to change it to U_BUFFER_OVERFLOW_ERROR in order to get your code to work with icu 1.6 or later.

See the updated sample code in icu/source/samples. All samples are updated. See jitterbug 516 for details. This was discussed in July 2000 on the icu mailing list. Please see the list archive for the discussion.

Where To Find More Information

http://oss.software.ibm.com/icu/ is the homepage of the International Components for Unicode.

The API Documentation is available online or for download. The (draft of) the User Guide is available for browsing and as a PDF file.

http://www.ibm.com/developer/unicode/ is a pointer to information on how to make applications global.

Submitting Comments, Requesting Features and Reporting Bugs

To submit comments, request features and report bugs, please contact us. The best forum is the ICU mailing list. See the information on how to browse and join the list. If you find a bug in the code that has not been submitted and/or fixed yet, then please submit a jitterbug.

Copyright © 1997-2000 International Business Machines Corporation and others. All Rights Reserved.
IBM Center for Emerging Technologies Silicon Valley,
10275 N De Anza Blvd., Cupertino, CA 95014
All rights reserved.