 |
VOOZH | about |
|
VOOZH | about |
This document explains the TokenExtractor abstract class, which serves as the primary extension point for customizing lexical rules in the Query Translator system. Token extractors define the regular expression patterns that recognize different token types in input query strings and create appropriate Token objects from matched text.
This page covers the abstract TokenExtractor base class, the two provided implementations (Full and Text), and guidance for implementing custom extractors. For information about the token types themselves, see Token Types. For details on how extractors are used in the tokenization process, see Tokenization Process.
Sources: lib/Languages/Galach/TokenExtractor.php1-124
The TokenExtractor abstract class is located at lib/Languages/Galach/TokenExtractor.php and provides the foundation for all token extraction implementations. It defines the contract for extracting tokens from input strings and handles the common logic of regex matching and byte offset calculation.
Sources: lib/Languages/Galach/TokenExtractor.php14-124 lib/Languages/Galach/TokenExtractor/Full.php18 lib/Languages/Galach/TokenExtractor/Text.php17
The extract() method at lib/Languages/Galach/TokenExtractor.php26-49 orchestrates the token extraction process:
Sources: lib/Languages/Galach/TokenExtractor.php26-49
| Method | Visibility | Required | Purpose |
|---|---|---|---|
extract(string, int) | public final | Inherited | Main entry point - extracts token at given position |
getExpressionTypeMap() | protected abstract | Must override | Returns array mapping regex patterns to token types |
createTermToken(int, array) | protected abstract | Must override | Creates term-type tokens (Word, Phrase, Tag, User) |
createGroupBeginToken(int, array) | protected | Optional | Creates GroupBegin tokens, can be overridden |
getByteOffset(string, int) | private | Inherited | Converts character position to byte offset for preg_match() |
Sources: lib/Languages/Galach/TokenExtractor.php26-123
The getExpressionTypeMap() method must return an array where:
/Au flagsTokenizer class(?<lexeme>...) identifying the matched token textThe /Au flags are critical:
/A anchors matching at the current byte offset/u enables UTF-8 mode for proper multi-byte character handlingSources: lib/Languages/Galach/TokenExtractor.php51-60
The Full implementation at lib/Languages/Galach/TokenExtractor/Full.php supports all features of the Galach language, including domain prefixes, tags, and users.
The Full extractor defines expressions for all Galach tokens at lib/Languages/Galach/TokenExtractor/Full.php25-39:
Sources: lib/Languages/Galach/TokenExtractor/Full.php25-39
Full extractor recognizes optional domain prefixes in three contexts:
GROUP_BEGIN at lib/Languages/Galach/TokenExtractor/Full.php34:
/(?<lexeme>(?:(?<domain>[a-zA-Z_][a-zA-Z0-9_\-.]*):)?(?<delimiter>\())/Au
Matches: domain:( or just (
PHRASE at lib/Languages/Galach/TokenExtractor/Full.php37:
/(?<lexeme>(?:(?<domain>[a-zA-Z_][a-zA-Z0-9_\-.]*):)?(?<quote>(?<!\\\\)["])(?<phrase>.*?)(?:(?<!\\\\)(?P=quote)))/Aus
Matches: domain:"phrase" or "phrase"
WORD at lib/Languages/Galach/TokenExtractor/Full.php38:
/(?<lexeme>(?:(?<domain>[a-zA-Z_][a-zA-Z0-9_\-.]*):)?(?<word>(?:\\\\\\\\|\\\\ |\\\\\(|\\\\\)|\\\\"|[^"()\s])+?))/Au
Matches: domain:word or word
Sources: lib/Languages/Galach/TokenExtractor/Full.php34-38
The createTermToken() method at lib/Languages/Galach/TokenExtractor/Full.php46-87 creates four types of term tokens based on which named capture group is set:
Special character unescaping:
\ + - ! ( ) : # @ space at lib/Languages/Galach/TokenExtractor/Full.php57Sources: lib/Languages/Galach/TokenExtractor/Full.php46-87
The Text implementation at lib/Languages/Galach/TokenExtractor/Text.php provides a simplified subset of Galach features, supporting only basic text search functionality without domains, tags, or users.
The Text extractor defines a reduced set of expressions at lib/Languages/Galach/TokenExtractor/Text.php24-36:
| Priority | Token Type | Pattern | Description |
|---|---|---|---|
| 1 | WHITESPACE | /[\s]+/ | One or more whitespace characters |
| 2 | MANDATORY | /\+/ | Plus sign operator |
| 3 | PROHIBITED | /-/ | Minus sign operator |
| 4 | LOGICAL_NOT_2 | /!/ | Exclamation mark operator |
| 5 | GROUP_END | /\)/ | Closing parenthesis |
| 6 | LOGICAL_NOT | /NOT/ | NOT keyword |
| 7 | LOGICAL_AND | /AND|&&/ | AND keyword or && |
| 8 | LOGICAL_OR | /OR|||/ | OR keyword or || |
| 9 | GROUP_BEGIN | /\(/ | Opening parenthesis (no domain) |
| 10 | TERM (phrase) | /"..."/ | Quoted phrase (no domain) |
| 11 | TERM (word) | /[term]/ | Word term (no domain) |
Sources: lib/Languages/Galach/TokenExtractor/Text.php24-36
Key differences:
#identifier is not included@identifier is not included\ + - ! ( ) " space at lib/Languages/Galach/TokenExtractor/Text.php54Sources: lib/Languages/Galach/TokenExtractor/Text.php24-76 lib/Languages/Galach/TokenExtractor/Full.php25-88
| Feature | Full | Text | Notes |
|---|---|---|---|
| Whitespace | ✓ | ✓ | Identical pattern |
Mandatory (+) | ✓ | ✓ | Identical pattern |
Prohibited (-) | ✓ | ✓ | Identical pattern |
Logical NOT (!, NOT) | ✓ | ✓ | Identical patterns |
| Logical AND | ✓ | ✓ | Identical pattern |
| Logical OR | ✓ | ✓ | Identical pattern |
| Group Begin | ✓ | ✓ | Full supports domain prefix |
| Group End | ✓ | ✓ | Identical pattern |
| Word terms | ✓ | ✓ | Full supports domain prefix |
| Phrase terms | ✓ | ✓ | Full supports domain prefix |
Tag terms (#tag) | ✓ | ✗ | Full only |
User terms (@user) | ✓ | ✗ | Full only |
| Domain prefixes | ✓ | ✗ | Full only |
| Escaped characters | 10 chars | 7 chars | Full adds :, #, @ |
Use case guidance:
Sources: lib/Languages/Galach/TokenExtractor/Full.php25-88 lib/Languages/Galach/TokenExtractor/Text.php24-76
Custom token extractors enable extending or modifying the lexical rules of Galach or creating entirely new query languages.
To create a custom extractor, extend TokenExtractor and implement two abstract methods:
Sources: lib/Languages/Galach/TokenExtractor.php61-73
Here's a conceptual outline of a minimal custom extractor that only supports words and whitespace:
Step 1: Define the expression type map
Return an array mapping regex patterns to token types. Each regex must have a (?<lexeme>...) named capture group:
Step 2: Implement term token creation
Check which named capture groups are present and create the appropriate token:
Sources: lib/Languages/Galach/TokenExtractor/Full.php41-44 lib/Languages/Galach/TokenExtractor/Full.php46-87
When defining custom patterns:
Always use /Au flags:
/A anchors match at current position/u enables UTF-8 modeDefine (?<lexeme>...) capture group: Required for all patterns at lib/Languages/Galach/TokenExtractor.php56-57
Order matters: Expressions are tried in array order. More specific patterns should come before general ones.
Use negative lookbehind for escaping: Prevent matching escaped characters:
(?<!\\\\)["] // Match quote not preceded by backslash
Use lookahead/lookbehind for word boundaries: Prevent operators from matching inside words at lib/Languages/Galach/TokenExtractor/Full.php31-33
Handle multi-byte characters: The getByteOffset() method at lib/Languages/Galach/TokenExtractor.php120-123 converts character positions to byte offsets for proper UTF-8 handling
Sources: lib/Languages/Galach/TokenExtractor.php26-49 lib/Languages/Galach/TokenExtractor/Full.php25-39
The test at tests/Galach/Tokenizer/TokenExtractorTest.php17-36 demonstrates testing for PCRE errors. Custom extractors should test:
Sources: tests/Galach/Tokenizer/TokenExtractorTest.php1-75
The TokenExtractor is used by the Tokenizer class (see Tokenization Process) which:
extract() at each position in the input stringTokenSequenceThe extractor is passed to the Tokenizer constructor, making it easy to swap implementations or use custom extractors.
Refresh this wiki