TextPreprocessor constructor

TextPreprocessor({
  1. String vocab = defaultVocab,
  2. int maxLength = 64,
})

Creates a preprocessor with the given vocabulary and max sequence length.

vocab is a string where each character gets an index (starting at 1). Index 0 is reserved for padding, and vocab.length + 1 for unknown chars.

Implementation

TextPreprocessor({
  String vocab = defaultVocab,
  int maxLength = 64,
})  : _maxLength = maxLength,
      _padIndex = 0,
      _unknownIndex = vocab.length + 1,
      _charToIndex = {
        for (int i = 0; i < vocab.length; i++) vocab[i]: i + 1,
      };