Graphorin API reference v0.1.0
Graphorin API reference / @graphorin/tools / / tokenise
Function: tokenise()
ts
function tokenise(text, stopwords?): string[];Defined in: packages/tools/src/registry/bm25.ts:46
Tokenise a body — lowercase, alphanumeric runs only, drop stopwords.
Parameters
| Parameter | Type | Default value |
|---|---|---|
text | string | undefined |
stopwords | ReadonlySet<string> | DEFAULT_STOPWORDS |
Returns
string[]