Questions tagged [tokenizer]
The tokenizer tag has no usage guidance.
Pset6 Similaritites Less Comfortable
Earlier, I was looking up questions about Pset6 to see if there are any answers that can help me with my code, however, it looks like I completely missed the memo for using set with similarities. My ...
Tokenizer adds space before text
When I tokenize my text, it is adding a space before it, and I am unable to strip it. tokenizer = nltk.tokenize.TweetTokenizer() tokens = tokenizer.tokenize(text) x = 0 for i in tokens: token = ...
Tokenizer changing text
When I try to tokenise some text to check its sentiment, the tokeniser automatically adds [''] around it. So if my text says good, when I tokenize it and print it, it says ['good']. Please can you ...
TweetTokenizer() vs split(), tokens
I don't understand, why we are using the TweetTokenizer() instead of just split()? What advantages and disadvantages does the TweetTokenizer() have? Also, I don't fully understand what are tokens? ...
How to check if certain token is in the file in analyzer.py
I have implementing (in my understanding) analyzer.py. I am confused after dividing a tweet into tokens how to check them if they are available in "negative" or "positive", my question is "Is there a ...