RuleBasedTransliterator is a transliterator
* that reads a set of rules in order to determine how to perform
* translations. Rule sets are stored in resource bundles indexed by
* name. Rules within a rule set are separated by semicolons (';').
* To include a literal semicolon, prefix it with a backslash ('\').
* Whitespace, as defined by Character.isWhitespace(),
* is ignored. If the first non-blank character on a line is '#',
* the entire line is ignored as a comment.
*
* Each set of rules consists of two groups, one forward, and one * reverse. This is a convention that is not enforced; rules for one * direction may be omitted, with the result that translations in * that direction will not modify the source text. In addition, * bidirectional forward-reverse rules may be specified for * symmetrical transformations.
* *Rule syntax
* *Rule statements take one of the following forms:
* *$alefmadda=\u0622;$alefmadda", will be replaced by
* the Unicode character U+0622. Variable names must begin
* with a letter and consist only of letters, digits, and
* underscores. Case is significant. Duplicate names cause
* an exception to be thrown, that is, variables cannot be
* redefined. The right hand side may contain well-formed
* text of any length, including no text at all ("$empty=;").
* The right hand side may contain embedded UnicodeSet
* patterns, for example, "$softvowel=[eiyEIY]".ai>$alefmadda;ai<$alefmadda;ai<>$alefmadda;Translation rules consist of a match pattern and an output
* string. The match pattern consists of literal characters,
* optionally preceded by context, and optionally followed by
* context. Context characters, like literal pattern characters,
* must be matched in the text being transliterated. However, unlike
* literal pattern characters, they are not replaced by the output
* text. For example, the pattern "abc{def}"
* indicates the characters "def" must be
* preceded by "abc" for a successful match.
* If there is a successful match, "def" will
* be replaced, but not "abc". The final '}'
* is optional, so "abc{def" is equivalent to
* "abc{def}". Another example is "{123}456"
* (or "123}456") in which the literal
* pattern "123" must be followed by "456".
*
The output string of a forward or reverse rule consists of
* characters to replace the literal pattern characters. If the
* output string contains the character '|', this is
* taken to indicate the location of the cursor after
* replacement. The cursor is the point in the text at which the
* next replacement, if any, will be applied. The cursor is usually
* placed within the replacement text; however, it can actually be
* placed into the precending or following context by using the
* special character '@'. Examples:
** **
a {foo} z > | @ bar; # foo -> bar, move cursor * before a
* {foo} xyz > bar @@|; # foo -> bar, cursor between * y and z
UnicodeSet
* *UnicodeSet patterns may appear anywhere that
* makes sense. They may appear in variable definitions.
* Contrariwise, UnicodeSet patterns may themselves
* contain variable references, such as "$a=[a-z];$not_a=[^$a]",
* or "$range=a-z;$ll=[$range]".
UnicodeSet patterns may also be embedded directly
* into rule strings. Thus, the following two rules are equivalent:
** **
$vowel=[aeiou]; $vowel>'*'; # One way to do this
* [aeiou]>'*'; * # * Another way
See {@link UnicodeSet} for more documentation and examples.
* *Segments
* *Segments of the input string can be matched and copied to the * output string. This makes certain sets of rules simpler and more * general, and makes reordering possible. For example:
* *** **
([a-z]) > $1 $1; * # * double lowercase letters
* ([:Lu:]) ([:Ll:]) > $2 $1; # reverse order of Lu-Ll pairs
The segment of the input string to be copied is delimited by
* "(" and ")". Up to
* nine segments may be defined. Segments may not overlap. In the
* output string, "$1" through "$9"
* represent the input string segments, in left-to-right order of
* definition.
Anchors
* *Patterns can be anchored to the beginning or the end of the text. This is done with the
* special characters '^' and '$'. For example:
** **
^ a > 'BEG_A'; # match 'a' at start of text
* a > 'A'; # match other instances * of 'a'
* z $ > 'END_Z'; # match 'z' at end of text
* z > 'Z'; # match other instances * of 'z'
It is also possible to match the beginning or the end of the text using a UnicodeSet.
* This is done by including a virtual anchor character '$' at the end of the
* set pattern. Although this is usually the match chafacter for the end anchor, the set will
* match either the beginning or the end of the text, depending on its placement. For
* example:
** **
$x = [a-z$]; # match 'a' through 'z' OR anchor
* $x 1 > 2; # match '1' after a-z or at the start
* 3 $x > 4; # match '3' before a-z or at the end
Example
* *The following example rules illustrate many of the features of * the rule language.
* *| Rule 1. | *abc{def}>x|y |
*
| Rule 2. | *xyz>r |
*
| Rule 3. | *yz>q |
*
Applying these rules to the string "adefabcdefz"
* yields the following results:
|adefabcdefz |
* Initial state, no rules match. Advance * cursor. | *
a|defabcdefz |
* Still no match. Rule 1 does not match * because the preceding context is not present. | *
ad|efabcdefz |
* Still no match. Keep advancing until * there is a match... | *
ade|fabcdefz |
* ... | *
adef|abcdefz |
* ... | *
adefa|bcdefz |
* ... | *
adefab|cdefz |
* ... | *
adefabc|defz |
* Rule 1 matches; replace "def"
* with "xy" and back up the cursor
* to before the 'y'. |
*
adefabcx|yz |
* Although "xyz" is
* present, rule 2 does not match because the cursor is
* before the 'y', not before the 'x'.
* Rule 3 does match. Replace "yz"
* with "q". |
*
adefabcxq| |
* The cursor is at the end; * transliteration is complete. | *
The order of rules is significant. If multiple rules may match * at some point, the first matching rule is applied.
* *Forward and reverse rules may have an empty output string. * Otherwise, an empty left or right hand side of any statement is a * syntax error.
* *Single quotes are used to quote any character other than a
* digit or letter. To specify a single quote itself, inside or
* outside of quotes, use two single quotes in a row. For example,
* the rule "'>'>o''clock" changes the
* string ">" to the string "o'clock".
*
Notes
* *While a RuleBasedTransliterator is being built, it checks that * the rules are added in proper order. For example, if the rule * "a>x" is followed by the rule "ab>y", * then the second rule will throw an exception. The reason is that * the second rule can never be triggered, since the first rule * always matches anything it matches. In other words, the first * rule masks the second rule.
* * @author Alan Liu * @internal Use transliterator factory methods instead since this class will be removed in that release. */ class U_I18N_API RuleBasedTransliterator : public Transliterator { /** * The data object is immutable, so we can freely share it with * other instances of RBT, as long as we do NOT own this object. */ TransliterationRuleData* data; /** * If true, we own the data object and must delete it. */ UBool isDataOwned; public: /** * Constructs a new transliterator from the given rules. * @param rules rules, separated by ';' * @param direction either FORWARD or REVERSE. * @exception IllegalArgumentException if rules are malformed. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ RuleBasedTransliterator(const UnicodeString& id, const UnicodeString& rules, UTransDirection direction, UnicodeFilter* adoptedFilter, UParseError& parseError, UErrorCode& status); /** * Constructs a new transliterator from the given rules. * @param rules rules, separated by ';' * @param direction either FORWARD or REVERSE. * @exception IllegalArgumentException if rules are malformed. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ RuleBasedTransliterator(const UnicodeString& id, const UnicodeString& rules, UTransDirection direction, UnicodeFilter* adoptedFilter, UErrorCode& status); /** * Covenience constructor with no filter. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ RuleBasedTransliterator(const UnicodeString& id, const UnicodeString& rules, UTransDirection direction, UErrorCode& status); /** * Covenience constructor with no filter and FORWARD direction. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ RuleBasedTransliterator(const UnicodeString& id, const UnicodeString& rules, UErrorCode& status); /** * Covenience constructor with FORWARD direction. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ RuleBasedTransliterator(const UnicodeString& id, const UnicodeString& rules, UnicodeFilter* adoptedFilter, UErrorCode& status); private: friend class TransliteratorRegistry; // to access TransliterationRuleData convenience ctor /** * Covenience constructor. * @param id the id for the transliterator. * @param theData the rule data for the transliterator. * @param adoptedFilter the filter for the transliterator */ RuleBasedTransliterator(const UnicodeString& id, const TransliterationRuleData* theData, UnicodeFilter* adoptedFilter = 0); friend class Transliterator; // to access following ct /** * Internal constructor. * @param id the id for the transliterator. * @param theData the rule data for the transliterator. * @param isDataAdopted determine who will own the 'data' object. True, the caller should not delete 'data'. */ RuleBasedTransliterator(const UnicodeString& id, TransliterationRuleData* data, UBool isDataAdopted); public: /** * Copy constructor. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ RuleBasedTransliterator(const RuleBasedTransliterator&); virtual ~RuleBasedTransliterator(); /** * Implement Transliterator API. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ Transliterator* clone(void) const; protected: /** * Implements {@link Transliterator#handleTransliterate}. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ virtual void handleTransliterate(Replaceable& text, UTransPosition& offsets, UBool isIncremental) const; public: /** * Return a representation of this transliterator as source rules. * These rules will produce an equivalent transliterator if used * to construct a new transliterator. * @param result the string to receive the rules. Previous * contents will be deleted. * @param escapeUnprintable if TRUE then convert unprintable * character to their hex escape representations, \uxxxx or * \Uxxxxxxxx. Unprintable characters are those other than * U+000A, U+0020..U+007E. * @internal Use transliterator factory methods instead since this class will be removed in that release. */ virtual UnicodeString& toRules(UnicodeString& result, UBool escapeUnprintable) const; protected: /** * Implement Transliterator framework */ virtual void handleGetSourceSet(UnicodeSet& result) const; public: /** * Override Transliterator framework */ virtual UnicodeSet& getTargetSet(UnicodeSet& result) const; /** * Return the class ID for this class. This is useful only for * comparing to a return value from getDynamicClassID(). For example: *
* . Base* polymorphic_pointer = createPolymorphicObject();
* . if (polymorphic_pointer->getDynamicClassID() ==
* . Derived::getStaticClassID()) ...
*
* @return The class ID for all objects of this class.
* @internal Use transliterator factory methods instead since this class will be removed in that release.
*/
static UClassID getStaticClassID(void) { return (UClassID)&fgClassID; }
/**
* Returns a unique class ID polymorphically. This method
* is to implement a simple version of RTTI, since not all C++
* compilers support genuine RTTI. Polymorphic operator==() and
* clone() methods call this method.
*
* Concrete subclasses of Transliterator that wish clients to * be able to identify them should implement getDynamicClassID() * and also a static method and data member: * *
* static UClassID getStaticClassID() { return (UClassID)&fgClassID; }
* static char fgClassID;
*
*
* Subclasses that do not implement this method will have a
* dynamic class ID of Transliterator::getStatisClassID().
*
* @return The class ID for this object. All objects of a given
* class have the same class ID. Objects of other classes have
* different class IDs.
*/
virtual UClassID getDynamicClassID(void) const { return getStaticClassID(); };
private:
/**
* Class identifier for RuleBasedTransliterator.
*/
static const char fgClassID;
void _construct(const UnicodeString& rules,
UTransDirection direction,
UParseError& parseError,
UErrorCode& status);
};
/**
* Constructs a new transliterator from the given rules.
* @param id the id for the transliterator.
* @param rules rules, separated by ';'
* @param direction either FORWARD or REVERSE.
* @param adoptedFilter the filter for this transliterator.
* @param parseError Struct to recieve information on position
* of error if an error is encountered
* @param status Output param set to success/failure code.
* @exception IllegalArgumentException if rules are malformed
* or direction is invalid.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& id,
const UnicodeString& rules,
UTransDirection direction,
UnicodeFilter* adoptedFilter,
UParseError& parseError,
UErrorCode& status) :
Transliterator(id, adoptedFilter) {
_construct(rules, direction,parseError,status);
}
/**
* Constructs a new transliterator from the given rules.
* @param id the id for the transliterator.
* @param rules rules, separated by ';'
* @param direction either FORWARD or REVERSE.
* @param adoptedFilter the filter for this transliterator.
* @param status Output param set to success/failure code.
* @exception IllegalArgumentException if rules are malformed
* or direction is invalid.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& id,
const UnicodeString& rules,
UTransDirection direction,
UnicodeFilter* adoptedFilter,
UErrorCode& status) :
Transliterator(id, adoptedFilter) {
UParseError parseError;
_construct(rules, direction,parseError, status);
}
/**
* Covenience constructor with no filter.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& id,
const UnicodeString& rules,
UTransDirection direction,
UErrorCode& status) :
Transliterator(id, 0) {
UParseError parseError;
_construct(rules, direction,parseError, status);
}
/**
* Covenience constructor with no filter and FORWARD direction.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& id,
const UnicodeString& rules,
UErrorCode& status) :
Transliterator(id, 0) {
UParseError parseError;
_construct(rules, UTRANS_FORWARD, parseError, status);
}
/**
* Covenience constructor with FORWARD direction.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& id,
const UnicodeString& rules,
UnicodeFilter* adoptedFilter,
UErrorCode& status) :
Transliterator(id, adoptedFilter) {
UParseError parseError;
_construct(rules, UTRANS_FORWARD,parseError, status);
}
U_NAMESPACE_END
#endif /* #if !UCONFIG_NO_TRANSLITERATION */
#endif