TweetTokenizer don't process 3 specific words

Question

I am working on analyzer and smile. Except "a+", "2-faced" and "2-faces" my code works correctly.These words should give me ':)', ':(', ':('. But all of them give me ':|'. I am sure these words in memory because i tested them. The mistake isn't in set variable. So it must be in tokenizer. Here is my tokenizer code.

def analyze(self, text):
    """Analyze text for sentiment, returning its score."""

    # Do tokenize 
    # Tknzr # 
    # lowers words' letters with preserve_case, 
    # strips @name with strip_handles
    # reduces repeated letters(greater than 3) with reduce_len
    tknzr = nltk.tokenize.TweetTokenizer(preserve_case = False,
                                         strip_handles = True,  
                                         reduce_len = True)     
    tokens = tknzr.tokenize(text)

    # Analyze tokens
    score = 0
    for token in tokens:
        if token in self.positives:
            score += 1
        elif token in self.negatives:
            score -= 1

    return score

Maybe i am wrong but I think tokenizer strips numbers and '+'. If that is true, how can i handle that? Apart from that do you see any other mistake is here?

Answer 1

It's not a "mistake" in tokenizer, it is how tokenizer works. It creates separate tokens for "punctuation", as is evidenced by the examples in the doc:

>>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--"
>>> tknzr.tokenize(s0)
['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--']

You could add 3 tests to the code to handle those specific cases. Or you could write a fairly sophisticated parsing algorithm. IMO, those cases do not materially affect the outcome of the project (read: grade - there is no check50, and the staff's solution gives the same result). Another slightly subversive solution: remove them from the word lists and never think of them again! :)