Tokenization is a way of separating a piece of text into smaller units called tokens. Here, tokens can be either words, characters, or subwords. Hence, tokenization can be broadly classified into 3 types – word, character, and subword (n-gram characters) tokenization.
cat nlp count tensorflow tokenizer natural-language character sentence keras-classification-models subword nerual-network imdb-dataset deep-learning-architectures rnn-keras smaller-units tokenizer-nlp
-
Updated
Jun 30, 2021 - Jupyter Notebook