Each set of rules consists of two groups, one forward, and one reverse. * This is a convention that is not enforced; rules for one direction may be * omitted, with the result that translations in that direction will not modify * the source text. * *
Rule syntax * *
Rule statements take one of the following forms: *
alefmadda=\u0622{alefmadda}",
* will be replaced by the Unicode character U+0622.softvowel=[eiyEIY][abc] |
* The set containing the characters 'a', 'b', and 'c'. | *
[^abc] |
* The set of all characters except 'a', 'b', and 'c'. | *
[A-Z] |
* The set of all characters from 'A' to 'Z' in Unicode order. | *
[:Lu:] |
* The set of Unicode uppercase letters. See * www.unicode.org * for a complete list of categories and their two-letter codes. | *
[^a-z[:Lu:][:Ll:]] |
* The set of all characters except 'a' through 'z' and * uppercase or lowercase letters. | *
ai>{alefmadda}ai<{alefmadda}Forward and reverse translation rules consist of a match
* pattern and an output string. The match pattern consists
* of literal characters, optionally preceded by context, and optionally
* followed by context. Context characters, like literal pattern characters,
* must be matched in the text being transliterated. However, unlike literal
* pattern characters, they are not replaced by the output text. For example,
* the pattern "[abc]def" indicates the characters
* "def" must be preceded by "abc" for a successful
* match. If there is a successful match, "def" will be replaced,
* but not "abc". The initial '[' is optional, so
* "abc]def" is equivalent to "[abc]def". Another
* example is "123[456]" (or "123[456") in which the
* literal pattern "123" must be followed by "456".
*
*
The output string of a forward or reverse rule consists of characters to
* replace the literal pattern characters. If the output string contains the
* character '|', this is taken to indicate the location of the
* cursor after replacement. The cursor is the point in the text
* at which the next replacement, if any, will be applied.
*
*
Example * *
The following example rules illustrate many of the features of the rule * language. *
| Rule 1. | *abc]def>x|y |
| Rule 2. | *xyz>r |
| Rule 3. | *yz>q |
Applying these rules to the string "adefabcdefz" yields the
* following results:
*
*
|adefabcdefz |
* Initial state, no rules match. Advance cursor. |
a|defabcdefz |
* Still no match. Rule 1 does not match because the preceding * context is not present. |
ad|efabcdefz |
* Still no match. Keep advancing until there is a match... |
ade|fabcdefz |
* ... |
adef|abcdefz |
* ... |
adefa|bcdefz |
* ... |
adefab|cdefz |
* ... |
adefabc|defz |
* Rule 1 matches; replace "def" with "xy"
* and back up the cursor to before the 'y'. |
adefabcx|yz |
* Although "xyz" is present, rule 2 does not match
* because the cursor is before the 'y', not before the
* 'x'. Rule 3 does match. Replace "yz" with
* "q". |
adefabcxq| |
* The cursor is at the end; transliteration is complete. |
The order of rules is significant. If multiple rules may match at some * point, the first matching rule is applied. * *
Forward and reverse rules may have an empty output string. Otherwise, an * empty left or right hand side of any statement is a syntax error. * *
Single quotes are used to quote the special characters
* =><{}[]|. To specify a single quote itself, inside or
* outside of quotes, use two single quotes in a row. For example, the rule
* "'>'>o''clock" changes the string ">" to
* the string "o'clock".
*
*
Notes * *
While a RuleBasedTransliterator is being built, it checks that the rules * are added in proper order. For example, if the rule "a>x" is followed by the * rule "ab>y", then the second rule will throw an exception. The reason is * that the second rule can never be triggered, since the first rule always * matches anything it matches. In other words, the first rule masks * the second rule. There is a cost of O(n^2) to make this check; in real-world * tests it appears to approximately double build time. * *
One optimization that can be made is to add a pragma to the rule language,
* "#pragma order", that turns off ordering checking. This pragma can then be
* added to all of our resource-based rules (after we build these once and
* determine that there are no ordering errors). I haven't made this change yet
* in the interests of keeping the code from getting too byzantine.
*
* @author Alan Liu
*/
class U_I18N_API RuleBasedTransliterator : public Transliterator {
/**
* The data object is immutable, so we can freely share it with
* other instances of RBT, as long as we do NOT own this object.
*/
TransliterationRuleData* data;
/**
* If true, we own the data object and must delete it.
*/
bool_t isDataOwned;
public:
/**
* Direction constant passed to constructor to specify whether forward
* or reverse rules are parsed. The other rules are ignored.
*/
enum Direction {
/**
* Direction constant passed to constructor to create a transliterator
* using the forward rules.
*/
FORWARD,
/**
* Direction constant passed to constructor to create a transliterator
* using the reverse rules.
*/
REVERSE
};
/**
* Constructs a new transliterator from the given rules.
* @param rules rules, separated by '\n'
* @param direction either FORWARD or REVERSE.
* @exception IllegalArgumentException if rules are malformed
* or direction is invalid.
*/
RuleBasedTransliterator(const UnicodeString& ID,
const UnicodeString& rules,
Direction direction,
UnicodeFilter* adoptedFilter,
UErrorCode& status);
/**
* Covenience constructor with no filter.
*/
RuleBasedTransliterator(const UnicodeString& ID,
const UnicodeString& rules,
Direction direction,
UErrorCode& status);
/**
* Covenience constructor with no filter and FORWARD direction.
*/
RuleBasedTransliterator(const UnicodeString& ID,
const UnicodeString& rules,
UErrorCode& status);
/**
* Covenience constructor with FORWARD direction.
*/
RuleBasedTransliterator(const UnicodeString& ID,
const UnicodeString& rules,
UnicodeFilter* adoptedFilter,
UErrorCode& status);
RuleBasedTransliterator(const UnicodeString& ID,
const TransliterationRuleData* theData,
UnicodeFilter* adoptedFilter = 0);
RuleBasedTransliterator(const RuleBasedTransliterator&);
virtual ~RuleBasedTransliterator();
/**
* Implement Transliterator API.
*/
Transliterator* clone(void) const;
/**
* Transliterates a segment of a string. Transliterator API.
* @param text the string to be transliterated
* @param start the beginning index, inclusive; 0 <= start
* <= limit.
* @param limit the ending index, exclusive; start <= limit
* <= text.length().
* @param result buffer to receive the transliterated text; previous
* contents are discarded
*/
virtual void transliterate(const UnicodeString& text,
int32_t start, int32_t limit,
UnicodeString& result) const;
/**
* Transliterates a segment of a string. Transliterator API.
* @param text the string to be transliterated
* @param start the beginning index, inclusive; 0 <= start
* <= limit.
* @param limit the ending index, exclusive; start <= limit
* <= text.length().
* @return The new limit index
*/
virtual int32_t transliterate(Replaceable& text,
int32_t start, int32_t limit) const;
/**
* Implements {@link Transliterator#handleKeyboardTransliterate}.
*/
virtual void handleKeyboardTransliterate(Replaceable& text,
int32_t index[3]) const;
/**
* Returns the length of the longest context required by this transliterator.
* This is preceding context.
* @return Maximum number of preceding context characters this
* transliterator needs to examine
*/
virtual int32_t getMaximumContextLength(void) const;
private:
void _construct(const UnicodeString& rules,
Direction direction,
UErrorCode& status);
};
/**
* Constructs a new transliterator from the given rules.
* @param rules rules, separated by '\n'
* @param direction either FORWARD or REVERSE.
* @exception IllegalArgumentException if rules are malformed
* or direction is invalid.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& ID,
const UnicodeString& rules,
Direction direction,
UnicodeFilter* adoptedFilter,
UErrorCode& status) :
Transliterator(ID, adoptedFilter) {
_construct(rules, direction, status);
}
/**
* Covenience constructor with no filter.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& ID,
const UnicodeString& rules,
Direction direction,
UErrorCode& status) :
Transliterator(ID, 0) {
_construct(rules, direction, status);
}
/**
* Covenience constructor with no filter and FORWARD direction.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& ID,
const UnicodeString& rules,
UErrorCode& status) :
Transliterator(ID, 0) {
_construct(rules, FORWARD, status);
}
/**
* Covenience constructor with FORWARD direction.
*/
inline RuleBasedTransliterator::RuleBasedTransliterator(
const UnicodeString& ID,
const UnicodeString& rules,
UnicodeFilter* adoptedFilter,
UErrorCode& status) :
Transliterator(ID, adoptedFilter) {
_construct(rules, FORWARD, status);
}
#endif