sgnlp.models.span_extraction.utils.load_examples

load_examples(examples: List[Dict[str, torch.Tensor]], tokenizer: sgnlp.models.span_extraction.tokenization.RecconSpanExtractionTokenizer, max_seq_length: int = 512, doc_stride: int = 512, max_query_length: int = 512, evaluate: bool = False, output_examples: bool = False) → torch.utils.data.dataset.TensorDataset[source]

Convert list of examples to TensorDataset

Parameters

  • examples (List[Dict[str, torch.Tensor]]) – train data

  • tokenizer (RecconSpanExtractionTokenizer) – RecconSpanExtractionTokenizer from sgnlp

  • max_seq_length (int, optional) – set max_seq_length. Defaults to 512.

  • doc_stride (int, optional) – set max_seq_length. Defaults to 512.

  • max_query_length (int, optional) – set max_seq_length. Defaults to 512.

  • evaluate (bool, optional) – option to use for evaluation. Defaults to False.

  • output_examples (bool, optional) – option to output examples. Defaults to False.

Returns

train data converted to TensorDataset

Return type

TensorDataset