Questions tagged [tokenizer]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
2answers
92 views

Pset6 Similaritites Less Comfortable

Earlier, I was looking up questions about Pset6 to see if there are any answers that can help me with my code, however, it looks like I completely missed the memo for using set with similarities. My ...
0
votes
1answer
42 views

Tokenizer adds space before text

When I tokenize my text, it is adding a space before it, and I am unable to strip it. tokenizer = nltk.tokenize.TweetTokenizer() tokens = tokenizer.tokenize(text) x = 0 for i in tokens: token = ...
0
votes
1answer
28 views

Tokenizer changing text

When I try to tokenise some text to check its sentiment, the tokeniser automatically adds [''] around it. So if my text says good, when I tokenize it and print it, it says ['good']. Please can you ...
0
votes
1answer
153 views

TweetTokenizer() vs split(), tokens

I don't understand, why we are using the TweetTokenizer() instead of just split()? What advantages and disadvantages does the TweetTokenizer() have? Also, I don't fully understand what are tokens? ...
0
votes
1answer
393 views

How to check if certain token is in the file in analyzer.py

I have implementing (in my understanding) analyzer.py. I am confused after dividing a tweet into tokens how to check them if they are available in "negative" or "positive", my question is "Is there a ...