i ve been reading post because the tokens is not working in the analyzer and i don t understand how come in all post i ve red about tweets there are no tokens at all (but zamyla said to use the tokens ?). I understand that the tokens break the tweets in one tweet and than we use the token reprensenting the word in one tokens(one tweet). But now i am confused how to use it and where to use it.

FOR ANALYSER : Here is what Zamyla said to use :
tokens = self.tokeniser.tokenize(text)
for analyser which it makes sens because we use a text and no need for the tweettoken yet=> error i get is ''instance of Analyser has no tokenizer member''. If i switch it like seen in post =>

tokenizer = nltk.tokenize.TweetTokenizer()
    tokens = tokeniser.tokenize(text).

well i still have an error at the tokens'' undefined variable tokeniser ?? i still have import nltk.

For TWEETS, zamyla said now to use the tweettokenizer, 

tokenizer = nltk.tokenize.TweetTokenizer()
        tokens = tokeniser.tokenize()

but if i use it in tweet like said, i have the message error ''undefine variable tokenizer''.

Please can u please enlight me ? tks

1 Answer 1


I think it's just a typo. "Tokeniser" and "Tokenizer"

  • tks but how we explain which one to use where (analyzer and tweets ) ?
    – Genevie
    Sep 4, 2017 at 20:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .