Text analysis.
also: tokenizer · tokenization · stemming · stemmer · stop words · text analyzer · analysis pipelineTurning text into index terms: character filters, a tokenizer, then token filters (lowercasing, stop words, stemming).
the two-minute refresher
- Character filters strip HTML and normalise characters (café → cafe).
- The tokenizer splits on whitespace and punctuation.
- Token filters lowercase, drop stop words and stem (jumps → jump).
- Queries must go through the same analysis, or 'Running' won't match 'run'.