I am working on analyzer and smile. Except "a+", "2-faced" and "2-faces" my code works correctly.These words should give me ':)', ':(', ':('. But all of them give me ':|'. I am sure these words in memory because i tested them. The mistake isn't in set variable. So it must be in tokenizer. Here is my tokenizer code.
def analyze(self, text):
"""Analyze text for sentiment, returning its score."""
# Do tokenize
# Tknzr #
# lowers words' letters with preserve_case,
# strips @name with strip_handles
# reduces repeated letters(greater than 3) with reduce_len
tknzr = nltk.tokenize.TweetTokenizer(preserve_case = False,
strip_handles = True,
reduce_len = True)
tokens = tknzr.tokenize(text)
# Analyze tokens
score = 0
for token in tokens:
if token in self.positives:
score += 1
elif token in self.negatives:
score -= 1
return score
Maybe i am wrong but I think tokenizer strips numbers and '+'. If that is true, how can i handle that? Apart from that do you see any other mistake is here?