0

For analyzer.py, I think my code works for defining the first part: instantiation of the analyzer, but not for defining the second: analyze. Please see the failing part of my code below.

Also, I have one question - the walkthrough for analyzer.py mentions tokenization. If, however, tokenizing an article of 1000 words, for instance, only means splitting up the article into 1000 separate words, why do we have to tokenize the user's input to this program, if the program is set up to prompt the user to enter one and only one word? Any insight is highly appreciated, thank you!

def analyze(self, text):
    """Analyze text for sentiment, returning its score."""
    tokenizer = nltk.tokenize.TweetTokenizer()
    tokensii = tokenizer.tokenize(text)
    tokens = str(tokensii).lower()
    print(tokens)
    score = 0
    for token in tokens:
        if token in self.positives:
            score = score + 1
        elif token in self.negatives:
            score = score - 1
    return score
Improve this question

1 Answer 1

Reset to default
0

Big problem here str(tokensii). tokenizer returns a list. str returns a string. for token in tokens iterates over each character in the string. It may look ok here print(tokens), but tokens is not a list anymore.

The smile program is set up to prompt the user for one "word". The analyze function may be useful for analyzing a list of words, perhaps in the tweets assignment?

smile actually is set up to prompt for one "input". You could run it with ./smile "Now is the time...." ie, a phrase in double quotes.

From TweetTokenizer doc:

When instantiating Tokenizer objects, there is a single option: preserve_case. By default, it is set to True. If it is set to False, then the tokenizer will downcase everything except for emoticons.

If you instantiate the tokenizer with preserve_case False (i.e. tokenizer = nltk.tokenize.TweetTokenizer(False)), all your tokens will be "downcased", obviating the need for lower()

Improve this answer
2
  • Thank you for your advice! My code works now, after placing the "lower" thing inside the for loop, although I still don't get why I can't place it outside of it, before entering the for loop.
    – Alex Chan
    Commented Mar 25, 2017 at 5:13
  • You can do it outside the loop, any number of ways. You cannot, however, cast the list as a str. Info added to answer.
    – DinoCoderSaurus
    Commented Mar 25, 2017 at 12:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .