UnicodeTools

This file provides instructions for building and running the UnicodeTools, which
can be used to:

WARNING!!

Instructions:

0. You will need to get ICU4J on your system, using CVS.

The rest of this will assume that you have set up CVS so that you load the ICU4J project into C:\ICU4J

You need both the main icu4j and a subproject called unicodetools. See: http://www.ibm.com/software/globalization/icu/repository.jsp. Inside unicodetools, look at com/ibm/text. The main directories of interest are UCD, UCA and utility.

0a. If you are using Eclipse for your IDE, look at the instructions on http://icu.sourceforge.net/docs/eclipse_howto/eclipse_howto.html

Set up Eclipse to build two projects: ICU4J and UnicodeTools:

Project Name: ICU4J
Directory: C:\ICU4J\icu4j
Default output folder = ICU4J/classes

Project Name: unicodetools
Create project from existing source: C:\ICU4J\unicodetools
Default Output Folder: unicodetools/classes

After Eclipse is set up with these, exclude certain files from unicodetools:

Right-Click UnicodeTools > Properties > Java Build Path > Exclusions
com/ibm/rbm/
com/ibm/text/utility/UnicodeMapInt.java
com/ibm/text/utility/TestUtility.java
com/ibm/text/UCD/GenerateThaiBreaks-old.java/
com/ibm/text/UCD/ProcessUnihan.java/
com/ibm/text/UCA/WriteHTMLCollation.java/

UnicodeTools must also include the ICU4J project, with

Right-Click UnicodeTools > Properties > Java Build Path > Projects

1. In UCD, you must edit UCD_Types.java at the top, to set the directories for the build:

public static final String DATA_DIR = "C:\\DATA\\";
public static final String UCD_DIR = BASE_DIR + "UCD\\";
public static final String BIN_DIR = DATA_DIR + "BIN\\";
public static final String GEN_DIR = DATA_DIR + "GEN\\";

Make sure that each of these directories exist. Also make sure that the following
exist:

<GEN_DIR>/DerivedData
<GEN_DIR>/DerivedData/ExtractedProperties
<UCD_DIR>/EXTRAS-Update

2. Download all of the UnicodeData files for each version into UCD_DIR.

The folder names must be of the form: "3.2.0-Update", so rename the folders on the
Unicode site to this format. If the folder contains ucd, then make the contents of that directory be the contents of the x.x.x-Update directory. That is, each directory will directly contain files like PropList....txt

2a Ensure Complete Release

If you are downloading any "incomplete" release (one that does not contain a complete set of data files for that release, you need to also download the previous complete release). Most of the N.M-Update directoriess are complete, *except*:

4.0-Update, which does not contain a copy of Unihan.txt and some other files
3.1-Update, which does not contain a copy of BidiMirroring.txt

Also, make the following changes to UnicodeData for 1.1.5:

Delete

3400;HANGUL SYLLABLE KIYEOK A;Lo;0;L;1100 1161;;;;N;;;;;
...
4DFF;HANGUL SYLLABLE MIEUM WEO RIEUL-THIEUTH;Lo;0;L;1106 116F 11B4;;;;N;;;;;
4E00;;Lo;0;L;;;;;N;;;;;

Add:

4E00;;Lo;0;L;;;;;N;;;;;
9FA5;;Lo;0;L;;;;;N;;;;;
E000;;Co;0;L;;;;;N;;;;;
F8FF;;Co;0;L;;;;;N;;;;;

And from a late version of Unicode, add:

F900;CJK COMPATIBILITY IDEOGRAPH-F900;Lo;0;L;8C48;;;;N;;;;;
...
FA2D;CJK COMPATIBILITY IDEOGRAPH-FA2D;Lo;0;L;9DB4;;;;N;;;;;

2b. UCA data

If you are building any of the UCA tools, you need to get a copy of the UCA data file
from http://www.unicode.org/reports/tr10/#AllKeys. The default location for this is:

BASE_DIR + "Collation\allkeys" + VERSION + ".txt".

If you have it in a different location, change that value for KEYS in UCA.java, and
the value for BASE_DIR

2c. Here is an example of the default directory structure with files. All of the yellow ones should exist

C://DATA/

        BIN/
    
        Collation/
            allkeys-3.1.1.txt
        
        GEN/
            DerivedData/
        UCD/
            3.0.0-Update/
                Unihan-3.2.0.txt
                ...
            3.0.1-Update/
                ...
            3.1.0-Update/
                ...
            3.1.1-Update/
                ...
            3.2.0-Update/
                ...
            4.0.0-Update/
                ArabicShaping-4.0.0d14b.txt
                BidiMirroring-4.0.0d1b.txt
                ...
            EXTRAS-Update/

3. Versions

All of the following have "version X" in the options you give to Java (either on the  command line, or in the Eclipse 'run' options. If you want a specific version like 3.1.0, then you would write "version 3.1.1". If you want the latest version (4.1.0), you can omit the "version X".

4. Building Files

  1. Setup
    1. In Eclipse, open the Package Explorer (Use Window>Show View if you don't see it)
    2. Open UnicodeTools
      • com.ibm.text.UCD
        • MakeUnicodeFiles.txt

          This file drives the production of the derived Unicode files. The first three lines contain parameters that you may want to modify at some times:

          Generate: .*script.* // this is a regular expression. Use .* for all files
DeltaVersion: 10     // This gets appended to the file name. Pick 1+ the highest value in Public
CopyrightYear: 2006  // Pick the current year
    3. Open in Package Explorer
      • com.ibm.text.UCD
        • Main
    4. Run>Run As...
      1. Choose Java Application
        • it will fail, don't worry; you need to set some parameters.
    5. Run>Run...
      • Select the Arguments tab, and fill in the following
        • Program arguments:
          build 5.0.0 MakeUnicodeFiles
        • VM arguments: 
          -Xms512m -Xmx512m
      • Close and Save
  2. Run
    1. You'll see it build the 5.0 files, with something like the following results:
      Writing UCD_Data5.0.0
Data Size: 109,802
Wrote Data 109802
    2. For each version, the tools build a set of binary data in BIN that contain the information for that release. This is done automatically, or you can manually do it with the Program Arguments
      version X build

      This builds an compressed format of all the UCD data (except blocks and Unihan) into the BIN directory. Don't worry about the voluminous console messages, unless one says "FAIL".

      You have to manually do this if you change any of the data files in that version!

      Note: if for any reason you modify the binary format of the BIN files, you also have to bump the value in that file:

      static final byte BINARY_FORMAT = 8; // bumped if binary format of UCD changes
  3. Results in C:\DATA\GEN\DerivedData
    1. The files will be in this directory.
    2. There are also DIFF folders, that contain BAT files that you can run on Windows with CompareIt. (You can modify the code to build BATs with another Diff program if you want).
      1. For any file with a significant difference, it will build two BAT files, such as the first two below.
        Diff_PropList-5.0.0d10.txt.bat
OLDER-Diff_PropList-5.0.0d10.txt.bat

UNCHANGED-Diff_PropertyValueAliases-5.0.0d10.txt.bat
    3. Any files without significant changes will have "UNCHANGED" as a prefix: ignore them.  The OLDER prefix is the comparison to the last version of Unicode.
    4. On Windows you can run these BATs to compare files:
  4. NFSkippable
    1. A file is needed by ICU that is generated with the same tool. Just use the input parameter "NFSkippable" to generate the file NFSafeSets.txt, also in file:///C:/DATA/GEN

5. Invariant Checking

  1. Setup
    1. Open in Package Explorer
      • com.ibm.text.UCD
        • TestUnicodeInvariants.java
    2. Run>Run As... Java Application
      Will create the following file of results:
      C:\DATA\GEN\UnicodeInvariantResults.txt\

      And on the console will list whether any problems are found. Thus in the following case there was one failure:

      ParseErrorCount=0
TestFailureCount=1
    3. The header of the result file explains the syntax of the tests.
    4. Open that file and search for "**** START Error Info ****". Each such point provides a dump of comparison information.
      1. Failures print a list of differences between two sets being compared. So if A and B are being compared, it prints all the items in A-B, then in B-A, then in A&B.
      2. For example, here is a listing of a problem that must be corrected. Note that usually there is a comment that explains what the following line or lines are supposed to test. Then will come FALSE (indicating that the test failed), then the detailed error report.
        # Canonical decompositions (minus exclusions) must be identical across releases
[$Decomposition_Type:Canonical - $Full_Composition_Exclusion] = [$×Decomposition_Type:Canonical - $×Full_Composition_Exclusion]

FALSE
**** START Error Info ****

In [$×Decomposition_Type:Canonical - $×Full_Composition_Exclusion], but not in [$Decomposition_Type:Canonical - $Full_Composition_Exclusion] :

# Total code points: 0

Not in [$×Decomposition_Type:Canonical - $×Full_Composition_Exclusion], but in [$Decomposition_Type:Canonical - $Full_Composition_Exclusion] :
1B06           # Lo       BALINESE LETTER AKARA TEDUNG
1B08           # Lo       BALINESE LETTER IKARA TEDUNG
1B0A           # Lo       BALINESE LETTER UKARA TEDUNG
1B0C           # Lo       BALINESE LETTER RA REPA TEDUNG
1B0E           # Lo       BALINESE LETTER LA LENGA TEDUNG
1B12           # Lo       BALINESE LETTER OKARA TEDUNG
1B3B           # Mc       BALINESE VOWEL SIGN RA REPA TEDUNG
1B3D           # Mc       BALINESE VOWEL SIGN LA LENGA TEDUNG
1B40..1B41     # Mc   [2] BALINESE VOWEL SIGN TALING TEDUNG..BALINESE VOWEL SIGN TALING REPA TEDUNG
1B43           # Mc       BALINESE VOWEL SIGN PEPET TEDUNG

# Total code points: 11

In both [$×Decomposition_Type:Canonical - $×Full_Composition_Exclusion], and in [$Decomposition_Type:Canonical - $Full_Composition_Exclusion] :
00C0..00C5     # L&   [6] LATIN CAPITAL LETTER A WITH GRAVE..LATIN CAPITAL LETTER A WITH RING ABOVE
00C7..00CF     # L&   [9] LATIN CAPITAL LETTER C WITH CEDILLA..LATIN CAPITAL LETTER I WITH DIAERESIS
00D1..00D6     # L&   [6] LATIN CAPITAL LETTER N WITH TILDE..LATIN CAPITAL LETTER O WITH DIAERESIS
...
30F7..30FA     # Lo   [4] KATAKANA LETTER VA..KATAKANA LETTER VO
30FE           # Lm       KATAKANA VOICED ITERATION MARK
AC00..D7A3     # Lo [11172] HANGUL SYLLABLE GA..HANGUL SYLLABLE HIH

# Total code points: 12089
**** END Error Info ****
    5. Options:
      1. -r    Print the failures as a range list.
      2. -fxxx    Use a different input file, such as -fInvariantTest.txt

6. Options

  1. If you want to see files that are opened while processing, do the following:
    1. Run>Run
    2. Select the Arguments tab, and add the following
      1. VM arguments: 
        -DSHOW_FILES

5. UCA

  1. You will use com.ibm.text.UCA.Main as your main class, creating along the same lines as above.
  2. To test whether the UCA files are valid, use the options (note: you must also build the ICU files below, since they test other aspects).
    writeCollationValidityLog

    It will create a file:

    C:\DATA\GEN\collation\5.0.0\CheckCollationValidity.html
    1. Review this file. It will list errors. Some of those are actually warnings, and indicate possible problems (this is indicated in the text, such as by: "These are not necessarily errors, but should be examined for possible errors"). In those cases, the items should be reviewed to make sure that there are no inadvertent problems.
    2. If it is not so marked, it is a true error, and must be fixed.
    3. At the end, there is section 11. Coverage. There are two sections:
      1. In UCDxxx, but not in allkeys. Check this over to make sure that these are all the characters that should get implicit weights.
      2. In allkeys, but not in UCD. These should be only contractions. Check them over to make sure they look right also.

  3. To build all the charts (including for the UCA), use the options: 

    normalizationChart caseChart scriptChart indexChart

    They will be built into

    C:\DATA\GEN\charts

    Once UCA is released, then copy those files up to the right spots in the Unicode site:

  4. To build all the UCA files used by ICU, use the option:

    ICU

    They will be built into:

    C:\DATA\GEN\collation\5.0.0
  5. You should then build a set of the ICU files for the previous version, if you don't have them. Use the options:
    version 4.1.0 ICU

    Or whatever the last version was.

  6. Now, you will want to compare versions. The key file is UCA_Rules_NoCE.txt. It contains the rules expressed in ICU format, which allows for comparison across versions of UCA without spurious variations of the numbers getting in the way.
    1. Do a Diff between the last and current versions of these files, and verify that all the differences are either new characters, or were authorized to be changed by the UTC.