When I try to tokenise some text to check its sentiment, the tokeniser automatically adds [''] around it. So if my text says good, when I tokenize it and print it, it says ['good']. Please can you tell me why this is? Here is my code:

tokenizer = nltk.tokenize.TweetTokenizer()
token = tokenizer.tokenize(text)

print("", text)
print("", token)

1 Answer 1


From analyzer.py in the spec

a tokenizer that you can use to split a tweet (which is maximally a 140-character str object) into a list of words (i.e., shorter str objects).

The tokenizer returns a list, so you will need to iterate over it to get each individual word token from the text.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .