I am working on analyzer and smile. Except "a+", "2-faced" and "2-faces" my code works correctly.These words should give me ':)', ':(', ':('. But all of them give me ':|'. I am sure these words in memory because i tested them. The mistake isn't in set variable. So it must be in tokenizer. Here is my tokenizer code.
def analyze(self, text): """Analyze text for sentiment, returning its score.""" # Do tokenize # Tknzr # # lowers words' letters with preserve_case, # strips @name with strip_handles # reduces repeated letters(greater than 3) with reduce_len tknzr = nltk.tokenize.TweetTokenizer(preserve_case = False, strip_handles = True, reduce_len = True) tokens = tknzr.tokenize(text) # Analyze tokens score = 0 for token in tokens: if token in self.positives: score += 1 elif token in self.negatives: score -= 1 return score
Maybe i am wrong but I think tokenizer strips numbers and '+'. If that is true, how can i handle that? Apart from that do you see any other mistake is here?