0

When I tokenize my text, it is adding a space before it, and I am unable to strip it.

tokenizer = nltk.tokenize.TweetTokenizer()
tokens = tokenizer.tokenize(text)
x = 0 

for i in tokens:
    token = tokens[x]
    token = token.strip(' ')

    print("token:",token)
1
  • Try using token.strip() instead, just in case the issue is because of tabs
    – sinister
    Nov 26 '17 at 17:31
1

Are you sure it's adding a space, and you're not just printing a space using print? Consider

print('token: "{}"'.format(token))

Added " so it's clear where the token begins and ends. Is there an extra space?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .