CsgTokenizer

class CsgTokenizer(*args, **kwargs)

build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

This implementation does not add special tokens and this method should be overridden in a subclass.

Parameters

• token_ids_0 (List[int]) – The first tokenized sequence.

• token_ids_1 (List[int], optional) – The second tokenized sequence.

Returns

The model input with special tokens.

Return type

List[int]

convert_tokens_to_string(tokens)

Converts a sequence of tokens (string) in a single string.