Tokenizer changing text

Question

When I try to tokenise some text to check its sentiment, the tokeniser automatically adds [''] around it. So if my text says good, when I tokenize it and print it, it says ['good']. Please can you tell me why this is? Here is my code:

tokenizer = nltk.tokenize.TweetTokenizer()
token = tokenizer.tokenize(text)

print("", text)
print("", token)

Answer 1

From analyzer.py in the spec

a tokenizer that you can use to split a tweet (which is maximally a 140-character str object) into a list of words (i.e., shorter str objects).

The tokenizer returns a list, so you will need to iterate over it to get each individual word token from the text.