Transliterator is an abstract class that
* transliterates text from one format to another. The most common
* kind of transliterator is a script, or alphabet, transliterator.
* For example, a Russian to Latin transliterator changes Russian text
* written in Cyrillic characters to phonetically equivalent Latin
* characters. It does not translate Russian to English!
* Transliteration, unlike translation, operates on characters, without
* reference to the meanings of words and sentences.
*
* Although script conversion is its most common use, a
* transliterator can actually perform a more general class of tasks.
* In fact, Transliterator defines a very general API
* which specifies only that a segment of the input text is replaced
* by new text. The particulars of this conversion are determined
* entirely by subclasses of Transliterator.
*
*
Transliterators are stateless * *
Transliterator objects are stateless; they
* retain no information between calls to
* transliterate(). (However, this does not
* mean that threads may share transliterators without synchronizing
* them. Transliterators are not immutable, so they must be
* synchronized when shared between threads.) This1 might seem to
* limit the complexity of the transliteration operation. In
* practice, subclasses perform complex transliterations by delaying
* the replacement of text until it is known that no other
* replacements are possible. In other words, although the
* Transliterator objects are stateless, the source text
* itself embodies all the needed information, and delayed operation
* allows arbitrary complexity.
*
*
Batch transliteration * *
The simplest way to perform transliteration is all at once, on a
* string of existing text. This is referred to as batch
* transliteration. For example, given a string input
* and a transliterator t, the call
*
*
String result = t.transliterate(input);
* Keyboard transliteration * *
Somewhat more involved is keyboard, or incremental * transliteration. This is the transliteration of text that is * arriving from some source (typically the user's keyboard) one * character at a time, or in some other piecemeal fashion. * *
In keyboard transliteration, a Replaceable buffer
* stores the text. As text is inserted, as much as possible is
* transliterated on the fly. This means a GUI that displays the
* contents of the buffer may show text being modified as each new
* character arrives.
*
*
Consider the simple RuleBasedTransliterator:
*
*
* th>{theta}
* t>{tau}
*
* t>|{tau}
* {tau}h>{theta}
* transliterate(). Typically, the cursor will
* be coincident with the insertion point, but in a case like the one
* above, it will precede the insertion point.
*
* Keyboard transliteration methods maintain a set of three indices
* that are updated with each call to
* transliterate(), including the cursor, start,
* and limit. Since these indices are changed by the method, they are
* passed in an int[] array. The START index
* marks the beginning of the substring that the transliterator will
* look at. It is advanced as text becomes committed (but it is not
* the committed index; that's the CURSOR). The
* CURSOR index, described above, marks the point at
* which the transliterator last stopped, either because it reached
* the end, or because it required more characters to disambiguate
* between possible inputs. The CURSOR can also be
* explicitly set by rules in a RuleBasedTransliterator.
* Any characters before the CURSOR index are frozen;
* future keyboard transliteration calls within this input sequence
* will not change them. New text is inserted at the
* LIMIT index, which marks the end of the substring that
* the transliterator looks at.
*
*
Because keyboard transliteration assumes that more characters
* are to arrive, it is conservative in its operation. It only
* transliterates when it can do so unambiguously. Otherwise it waits
* for more characters to arrive. When the client code knows that no
* more characters are forthcoming, perhaps because the user has
* performed some input termination operation, then it should call
* finishTransliteration() to complete any
* pending transliterations.
*
*
Inverses * *
Pairs of transliterators may be inverses of one another. For
* example, if transliterator A transliterates characters by
* incrementing their Unicode value (so "abc" -> "def"), and
* transliterator B decrements character values, then A
* is an inverse of B and vice versa. If we compose A
* with B in a compound transliterator, the result is the
* indentity transliterator, that is, a transliterator that does not
* change its input text.
*
* The Transliterator method getInverse()
* returns a transliterator's inverse, if one exists, or
* null otherwise. However, the result of
* getInverse() usually will not be a true
* mathematical inverse. This is because true inverse transliterators
* are difficult to formulate. For example, consider two
* transliterators: AB, which transliterates the character 'A'
* to 'B', and BA, which transliterates 'B' to 'A'. It might
* seem that these are exact inverses, since
*
*
"A" x AB -> "B"* * where 'x' represents transliteration. However, * *
* "B" x BA -> "A"
"ABCD" x AB -> "BBCD"* * so AB composed with BA is not the * identity. Nonetheless, BA may be usefully considered to be * AB's inverse, and it is on this basis that * AB
* "BBCD" x BA -> "AACD"
.getInverse() could legitimately return
* BA.
*
* IDs and display names * *
A transliterator is designated by a short identifier string or * ID. IDs follow the format source-destination, * where source describes the entity being replaced, and * destination describes the entity replacing * source. The entities may be the names of scripts, * particular sequences of characters, or whatever else it is that the * transliterator converts to or from. For example, a transliterator * from Russian to Latin might be named "Russian-Latin". A * transliterator from keyboard escape sequences to Latin-1 characters * might be named "KeyboardEscape-Latin1". By convention, system * entity names are in English, with the initial letters of words * capitalized; user entity names may follow any format so long as * they do not contain dashes. * *
In addition to programmatic IDs, transliterator objects have * display names for presentation in user interfaces, returned by * {@link #getDisplayName}. * *
Factory methods and registration * *
In general, client code should use the factory method
* getInstance() to obtain an instance of a
* transliterator given its ID. Valid IDs may be enumerated using
* getAvailableIDs(). Since transliterators are mutable,
* multiple calls to getInstance() with the same ID will
* return distinct objects.
*
*
In addition to the system transliterators registered at startup,
* user transliterators may be registered by calling
* registerInstance() at run time. A registered instance
* acts a template; future calls to getInstance() with the ID
* of the registered object return clones of that object. Thus any
* object passed to registerInstance() must implement
* clone() propertly. To register a transliterator subclass
* without instantiating it (until it is needed), users may call
* registerClass(). In this case, the objects are
* instantiated by invoking the zero-argument public constructor of
* the class.
*
*
Subclassing
*
* Subclasses must implement the abstract method
* handleTransliterate().
Subclasses should override
* the transliterate() method taking a
* Replaceable and the transliterate()
* method taking a String and StringBuffer
* if the performance of these methods can be improved over the
* performance obtained by the default implementations in this class.
*
* @author Alan Liu
* @draft
*/
class U_I18N_API Transliterator {
public:
/**
* Direction constant indicating the direction in a transliterator, e.g.,
* the forward or reverse rules of a RuleBasedTransliterator. An "A-B"
* transliterator transliterates A to B when operating in the forward
* direction, and B to A when operating in the reverse direction.
* @draft
*/
enum Direction {
FORWARD,
REVERSE
};
/**
* @draft
*/
class Position {
public:
/**
* In transliterate(), the beginning index, inclusive
*/
int32_t start;
/**
* In transliterate(), the ending index, exclusive
*/
int32_t limit;
/**
* In transliterate(), the next character to be
* considered for transliteration
*/
int32_t cursor;
/**
* In transliterate(), the limit character to be
* considered for transliteration
*/
int32_t end;
/**
* Constructor from start, limit. Sets cursor to start and
* end to limit.
*/
Position(int32_t start, int32_t limit);
/**
* Constructor from start, limit, cursor. Sets
* end to limit.
*/
Position(int32_t start, int32_t limit, int32_t cursor);
/**
* Constructor from start, limit, cursor, end.
*/
Position(int32_t start, int32_t limit,
int32_t cursor, int32_t end);
};
private:
/**
* Programmatic name, e.g., "Latin-Arabic".
*/
UnicodeString ID;
/**
* This transliterator's filter. Any character for which
* filter.contains() returns false will not be
* altered by this transliterator. If filter is
* null then no filtering is applied.
*/
UnicodeFilter* filter;
int32_t maximumContextLength;
/**
* Dictionary of known transliterators. Keys are String
* names, values are one of the following:
*
*
Transliterator objects
*
* Class objects. Such objects must represent
* subclasses of Transliterator, and must satisfy the
* constraints described in registerClass()
*
* RULE_BASED_PLACEHOLDER, in which case the ID
* will have its first '-' removed and be appended to
* RB_RULE_BASED_PREFIX to form a resource bundle name from which
* the RB_RULE key is looked up to obtain the rule.
*
* REVERSE_RULE_BASED_PLACEHOLDER. Like
* RULE_BASED_PLACEHOLDER, except the entity names in
* the ID are reversed, and the argument
* RuleBasedTransliterator.REVERSE is pased to the
* RuleBasedTransliterator constructor.
* 0 <= start
* <= limit.
* @param limit the ending index, exclusive; start <= limit
* <= text.length().
* @param filter the filter. Any character for which
* filter.contains() returns false will not be
* altered by this transliterator. If filter is
* null then no filtering is applied.
* @return The new limit index. The text previously occupying [start,
* limit) has been transliterated, possibly to a string of a different
* length, at [start, new-limit), where
* new-limit is the return value.
* @draft
*/
virtual int32_t transliterate(Replaceable& text,
int32_t start, int32_t limit) const;
/**
* Transliterates an entire string in place. Convenience method.
* @param text the string to be transliterated
* @draft
*/
virtual void transliterate(Replaceable& text) const;
/**
* Transliterates the portion of the text buffer that can be
* transliterated unambiguosly after new text has been inserted,
* typically as a result of a keyboard event. The new text in
* insertion will be inserted into text
* at index.limit, advancing
* index.limit by insertion.length().
* Then the transliterator will try to transliterate characters of
* text between index.cursor and
* index.limit. Characters before
* index.cursor will not be changed.
*
* Upon return, values in index will be updated.
* index.start will be advanced to the first
* character that future calls to this method will read.
* index.cursor and index.limit will
* be adjusted to delimit the range of text that future calls to
* this method may change.
*
*
Typical usage of this method begins with an initial call
* with index.start and index.limit
* set to indicate the portion of text to be
* transliterated, and index.cursor == index.start.
* Thereafter, index can be used without
* modification in future calls, provided that all changes to
* text are made via this method.
*
*
This method assumes that future calls may be made that will * insert new text into the buffer. As a result, it only performs * unambiguous transliterations. After the last call to this * method, there may be untransliterated text that is waiting for * more input to resolve an ambiguity. In order to perform these * pending transliterations, clients should call {@link * #finishTransliteration} after the last call to this * method has been made. * * @param text the buffer holding transliterated and untransliterated text * @param index an array of three integers. * *
index.start: the beginning index,
* inclusive; 0 <= index.start <= index.limit.
*
* index.limit: the ending index, exclusive;
* index.start <= index.limit <= text.length().
* insertion is inserted at
* index.limit.
*
* index.cursor: the next character to be
* considered for transliteration; index.start <=
* index.cursor <= index.limit. Characters before
* index.cursor will not be changed by future calls
* to this method.index.limit. If null then no text
* is inserted.
* @see #handleTransliterate
* @exception IllegalArgumentException if index
* is invalid
* @draft
*/
virtual void transliterate(Replaceable& text, Position& index,
const UnicodeString& insertion,
UErrorCode& status) const;
/**
* Transliterates the portion of the text buffer that can be
* transliterated unambiguosly after a new character has been
* inserted, typically as a result of a keyboard event. This is a
* convenience method; see {@link
* #transliterate(Replaceable, int[], String)} for details.
* @param text the buffer holding transliterated and
* untransliterated text
* @param index an array of three integers. See {@link
* #transliterate(Replaceable, int[], String)}.
* @param insertion text to be inserted and possibly
* transliterated into the translation buffer at
* index.limit.
* @see #transliterate(Replaceable, int[], String)
* @draft
*/
virtual void transliterate(Replaceable& text, Position& index,
UChar insertion,
UErrorCode& status) const;
/**
* Transliterates the portion of the text buffer that can be
* transliterated unambiguosly. This is a convenience method; see
* {@link #transliterate(Replaceable, int[], String)} for
* details.
* @param text the buffer holding transliterated and
* untransliterated text
* @param index an array of three integers. See {@link
* #transliterate(Replaceable, int[], String)}.
* @see #transliterate(Replaceable, int[], String)
* @draft
*/
virtual void transliterate(Replaceable& text, Position& index,
UErrorCode& status) const;
/**
* Finishes any pending transliterations that were waiting for
* more characters. Clients should call this method as the last
* call after a sequence of one or more calls to
* transliterate().
* @param text the buffer holding transliterated and
* untransliterated text.
* @param index the array of indices previously passed to {@link
* #transliterate}
* @draft
*/
virtual void finishTransliteration(Replaceable& text,
Position& index) const;
private:
/**
* This internal method does incremental transliteration. If the
* 'insertion' is non-null then we append it to 'text' before
* proceeding. This method calls through to the pure virtual
* framework method handleTransliterate() to do the actual
* work.
*/
void _transliterate(Replaceable& text,
Position& index,
const UnicodeString* insertion,
UErrorCode &status) const;
protected:
/**
* Abstract method that concrete subclasses define to implement
* keyboard transliteration. This method should transliterate all
* characters between index.cursor and
* index.limit that can be unambiguously
* transliterated, regardless of future insertions of text at
* index.limit. index.cursor should
* be advanced past committed characters (those that will not
* change in future calls to this method).
* index.limit should be updated to reflect text
* replacements that shorten or lengthen the text between
* index.cursor and index.limit. Upon
* return, neither index.cursor nor
* index.limit should be less than the initial value
* of index.cursor. index.start
* should not be changed.
*
* @param text the buffer holding transliterated and
* untransliterated text
* @param index an array of three integers. See {@link
* #transliterate(Replaceable, int[], String)}.
* @see #transliterate
*/
virtual void handleTransliterate(Replaceable& text,
Position& index,
bool_t incremental) const = 0;
// C++ requires this friend declaration so CompoundTransliterator
// can access handleTransliterate. Alternatively, we could
// make handleTransliterate public.
friend class CompoundTransliterator;
public:
/**
* Returns the length of the longest context required by this transliterator.
* This is preceding context. The default implementation supplied
* by Transliterator returns zero; subclasses
* that use preceding context should override this method to return the
* correct value. For example, if a transliterator translates "ddd" (where
* d is any digit) to "555" when preceded by "(ddd)", then the preceding
* context length is 5, the length of "(ddd)".
*
* @return The maximum number of preceding context characters this
* transliterator needs to examine
* @draft
*/
int32_t getMaximumContextLength(void) const;
protected:
/**
* Method for subclasses to use to set the maximum context length.
* @see #getMaximumContextLength
*/
void setMaximumContextLength(int32_t maxContextLength);
public:
/**
* Returns a programmatic identifier for this transliterator.
* If this identifier is passed to getInstance(), it
* will return this object, if it has been registered.
* @see #registerInstance
* @see #registerClass
* @see #getAvailableIDs
* @draft
*/
virtual const UnicodeString& getID(void) const;
/**
* Returns a name for this transliterator that is appropriate for
* display to the user in the default locale. See {@link
* #getDisplayName(Locale)} for details.
* @draft
*/
static UnicodeString& getDisplayName(const UnicodeString& ID,
UnicodeString& result);
/**
* Returns a name for this transliterator that is appropriate for
* display to the user in the given locale. This name is taken
* from the locale resource data in the standard manner of the
* java.text package.
*
* If no localized names exist in the system resource bundles,
* a name is synthesized using a localized
* MessageFormat pattern from the resource data. The
* arguments to this pattern are an integer followed by one or two
* strings. The integer is the number of strings, either 1 or 2.
* The strings are formed by splitting the ID for this
* transliterator at the first '-'. If there is no '-', then the
* entire ID forms the only string.
* @param inLocale the Locale in which the display name should be
* localized.
* @see java.text.MessageFormat
* @draft
*/
static UnicodeString& getDisplayName(const UnicodeString& ID,
const Locale& inLocale,
UnicodeString& result);
/**
* Returns the filter used by this transliterator, or NULL
* if this transliterator uses no filter.
* @draft
*/
virtual const UnicodeFilter* getFilter(void) const;
/**
* Returns the filter used by this transliterator, or NULL if this
* transliterator uses no filter. The caller must eventually delete the
* result. After this call, this transliterator's filter is set to
* NULL. Calls adoptFilter().
*/
UnicodeFilter* orphanFilter(void);
/**
* Changes the filter used by this transliterator. If the filter
* is set to null then no filtering will occur.
*
*
Callers must take care if a transliterator is in use by
* multiple threads. The filter should not be changed by one
* thread while another thread may be transliterating.
* @draft
*/
virtual void adoptFilter(UnicodeFilter* adoptedFilter);
/**
* Returns this transliterator's inverse. See the class
* documentation for details. This implementation simply inverts
* the two entities in the ID and attempts to retrieve the
* resulting transliterator. That is, if getID()
* returns "A-B", then this method will return the result of
* getInstance("B-A"), or null if that
* call fails.
*
*
This method does not take filtering into account. The * returned transliterator will have no filter. * *
Subclasses with knowledge of their inverse may wish to
* override this method.
*
* @return a transliterator that is an inverse, not necessarily
* exact, of this transliterator, or null if no such
* transliterator is registered.
* @see #registerInstance
* @draft
*/
Transliterator* createInverse(void) const;
/**
* Returns a Transliterator object given its ID.
* The ID must be either a system transliterator ID or a ID registered
* using registerInstance().
*
* @param ID a valid ID, as enumerated by getAvailableIDs()
* @return A Transliterator object with the given ID
* @exception IllegalArgumentException if the given ID is invalid.
* @see #registerInstance
* @see #getAvailableIDs
* @see #getID
* @draft
*/
static Transliterator* createInstance(const UnicodeString& ID,
Direction dir = FORWARD,
ParseError* parseError = 0);
private:
/**
* This is the path to the subdirectory within the locale data
* directory that contains the rule-based transliterator resource
* bundle files. This is constructed dynamically the first time
* Transliterator::getDataDirectory() is called.
*/
static char* DATA_DIR;
/**
* This is the name of a subdirectory within the locale data directory
* that contains the rule-based transliterator resource bundle files.
*/
static const char* RESOURCE_SUB_DIR;
/**
* Returns the directory in which the transliterator resource bundle
* files are located. This is a subdirectory, named RESOURCE_SUB_DIR,
* under Locale::getDataDirectory(). It ends in a path separator.
*/
static const char* getDataDirectory(void);
/**
* Returns a transliterator object given its ID. Unlike getInstance(),
* this method returns null if it cannot make use of the given ID.
*/
static Transliterator* _createInstance(const UnicodeString& ID,
ParseError* parseError = 0);
public:
/**
* Registers a instance obj of a subclass of
* Transliterator with the system. When
* createInstance() is called with an ID string that is
* equal to obj->getID(), then obj->clone() is
* returned.
*
* After this call the Transliterator class owns the adoptedObj
* and will delete it.
*
* @param obj an instance of subclass of
* Transliterator that defines clone()
* @see #getInstance
* @see #registerClass
* @see #unregister
* @draft
*/
static void registerInstance(Transliterator* adoptedObj,
UErrorCode& status);
private:
/**
* This internal method registers a prototype instance in the cache.
* The CALLER MUST MUTEX using cacheMutex before calling this method.
*/
static void _registerInstance(Transliterator* adoptedPrototype,
UErrorCode &status);
public:
/**
* Unregisters a transliterator or class. This may be either
* a system transliterator or a user transliterator or class.
*
* @param ID the ID of the transliterator or class
* @return the Object that was registered with
* ID, or null if none was
* @see #registerInstance
* @see #registerClass
* @draft
*/
static void unregister(const UnicodeString& ID);
private:
/**
* Unregisters a transliterator or class. Internal method.
* Prerequisites: The cache must be initialized, and the
* caller must own the cacheMutex.
*/
static void _unregister(const UnicodeString& ID);
/**
* Returns an enumeration over the programmatic names of registered
* Transliterator objects. This includes both system
* transliterators and user transliterators registered using
* registerInstance(). The enumerated names may be
* passed to getInstance().
*
* @return An Enumeration over String objects
* @see #getInstance
* @see #registerInstance
*/
// virtual Enumeration getAvailableIDs();
/**
* Vector of registered IDs.
*/
static UVector cacheIDs;
public:
/**
* Return the number of IDs currently registered with the system.
* To retrieve the actual IDs, call getAvailableID(i) with
* i from 0 to countAvailableIDs() - 1.
* @draft
*/
static int32_t countAvailableIDs(void);
/**
* Return the index-th available ID. index must be between 0
* and countAvailableIDs() - 1, inclusive. If index is out of
* range, the result of getAvailableID(0) is returned.
* @draft
*/
static const UnicodeString& getAvailableID(int32_t index);
protected:
/**
* Method for subclasses to use to obtain a character in the given
* string, with filtering.
*/
UChar filteredCharAt(const Replaceable& text, int32_t i) const;
private:
/**
* Comparison function for UVector. Compares two UnicodeString
* objects given void* pointers to them.
*/
static bool_t compareIDs(void* a, void* b);
static void initializeCache(void);
/* IDs take the form