checks if word is excluded from indexing

(excluded-word? “and”) => true

split hyphenated words

(expand-hyphenated-words “hello-world”) => “hello-world” “hello” “world”

remove excluded words

classifies text into word stems

(stems “The lazy brown fox jumped over”) => [“lazi” “brown” “fox” “jump” “over”]

makes tokens from text

(tokenise “The lazy brown fox jumped over”) => [“lazy” “brown” “fox” “jumped” “over”]