i ve been reading post because the tokens is not working in the analyzer and i don t understand how come in all post i ve red about tweets there are no tokens at all (but zamyla said to use the tokens ?). I understand that the tokens break the tweets in one tweet and than we use the token reprensenting the word in one tokens(one tweet). But now i am confused how to use it and where to use it.
FOR ANALYSER : Here is what Zamyla said to use :
tokens = self.tokeniser.tokenize(text)
for analyser which it makes sens because we use a text and no need for the tweettoken yet=> error i get is ''instance of Analyser has no tokenizer member''. If i switch it like seen in post =>
tokenizer = nltk.tokenize.TweetTokenizer()
tokens = tokeniser.tokenize(text).
well i still have an error at the tokens'' undefined variable tokeniser ?? i still have import nltk.
For TWEETS, zamyla said now to use the tweettokenizer,
tokenizer = nltk.tokenize.TweetTokenizer()
tokens = tokeniser.tokenize()
but if i use it in tweet like said, i have the message error ''undefine variable tokenizer''.
Please can u please enlight me ? tks
