Questions tagged [tokenizer]
The tokenizer tag has no usage guidance.
5
questions
1
vote
2
answers
101
views
Pset6 Similaritites Less Comfortable
Earlier, I was looking up questions about Pset6 to see if there are any answers that can help me with my code, however, it looks like I completely missed the memo for using set with similarities. My ...
- 157
0
votes
1
answer
45
views
Tokenizer adds space before text
When I tokenize my text, it is adding a space before it, and I am unable to strip it.
tokenizer = nltk.tokenize.TweetTokenizer()
tokens = tokenizer.tokenize(text)
x = 0
for i in tokens:
token = ...
- 135
0
votes
1
answer
35
views
Tokenizer changing text
When I try to tokenise some text to check its sentiment, the tokeniser automatically adds [''] around it. So if my text says good, when I tokenize it and print it, it says ['good']. Please can you ...
- 135
0
votes
1
answer
165
views
TweetTokenizer() vs split(), tokens
I don't understand, why we are using the TweetTokenizer() instead of just split()?
What advantages and disadvantages does the TweetTokenizer() have?
Also, I don't fully understand what are tokens? ...
- 17
0
votes
1
answer
398
views
How to check if certain token is in the file in analyzer.py
I have implementing (in my understanding) analyzer.py.
I am confused after dividing a tweet into tokens how to check them if they are available in "negative" or "positive", my question is "Is there a ...
- 93