Questions tagged [tokenizer]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
2 answers

Pset6 Similaritites Less Comfortable

Earlier, I was looking up questions about Pset6 to see if there are any answers that can help me with my code, however, it looks like I completely missed the memo for using set with similarities. My ...
  • 157
0 votes
1 answer

Tokenizer adds space before text

When I tokenize my text, it is adding a space before it, and I am unable to strip it. tokenizer = nltk.tokenize.TweetTokenizer() tokens = tokenizer.tokenize(text) x = 0 for i in tokens: token = ...
  • 135
0 votes
1 answer

Tokenizer changing text

When I try to tokenise some text to check its sentiment, the tokeniser automatically adds [''] around it. So if my text says good, when I tokenize it and print it, it says ['good']. Please can you ...
  • 135
0 votes
1 answer

TweetTokenizer() vs split(), tokens

I don't understand, why we are using the TweetTokenizer() instead of just split()? What advantages and disadvantages does the TweetTokenizer() have? Also, I don't fully understand what are tokens? ...
  • 17
0 votes
1 answer

How to check if certain token is in the file in

I have implementing (in my understanding) I am confused after dividing a tweet into tokens how to check them if they are available in "negative" or "positive", my question is "Is there a ...