tokenizationΒΆ

Classes

UFDTokenizer

The UFD Tokenizer class used for to generate token for the embedding model, derived from XLM Roberta Tokenizer class.