Tokenizer adds space before text

Question

When I tokenize my text, it is adding a space before it, and I am unable to strip it.

tokenizer = nltk.tokenize.TweetTokenizer()
tokens = tokenizer.tokenize(text)
x = 0 

for i in tokens:
    token = tokens[x]
    token = token.strip(' ')

    print("token:",token)

Answer 1

Are you sure it's adding a space, and you're not just printing a space using print? Consider

print('token: "{}"'.format(token))

Added " so it's clear where the token begins and ends. Is there an extra space?