!/usr/bin/env python3

import os import sys

from analyzer import Analyzer from termcolor import colored from helpers import get_user_timeline from nltk.tokenize import TweetTokenizer

def main():

# ensure proper usage
if len(sys.argv) != 2:
    sys.exit("Usage: ./smile tweeter_screenName")
# absolute paths to lists
positives = os.path.join(sys.path[0], "positive-words.txt")
negatives = os.path.join(sys.path[0], "negative-words.txt")

# instantiate analyzer
analyzer = Analyzer(positives, negatives)

# get tweets
tweets = get_user_timeline(sys.argv[1], 50)

if tweets == None:
    sys.exit ("Lack of proper tweets")

for tweet in tweets:
    tokenizer = TweetTokenizer()
    score = 0
    tokens = tokenizer.tokenize(tweet)
    for token in tokens:
        score += analyzer.analyze(token)
    if score > 0.0:
        print(colored(tweet, "green"))
    elif score < 0.0:
        print(colored(tweet, "red"))
        print(colored(tweet, "yellow"))

if __name__ == "__main__":

This code above works, but I don't understand why this works fine:

tokenizer = TweetTokenizer()
tokens = tokenizer.tokenize(tweet)

and this doesn't want to work:

tokens = TweetTokenizer.tokenize(tweet)

For me it is the same...

1 Answer 1


TweetTokenizer is a class in the nltk.tokenize.casual module.

When you use the name of the class as a function, like:

tokenizer = TweetTokenizer()

that function, in Object Oriented Languages, is called the "constructor" of the class, because it "constructs" an object of that class.

Classes can have static functions, which are functions you can call, without the need of creating an object first. For example, let's take a look at the following class, which has a constructor (the __init__() method), a class method (the baz() method), and a static method (the bar() method).

class Test(object):
    def __init__(self, foo):
        self.foo = foo

    def bar(s):
        print("bar() says {}".format(s))

    def baz(self, s):
        print("{} says {}".format(self.foo, s))

test1 = Test("test1")


See that in order to use the class method, (or member method), we have to first create an object of this class, which we will call test1. Then, we can call the baz() method on this object, by test1.baz("hello!").

In order to call the static method, we don't have to create an object first, and we can call it directly from within the class, by Test.bar("hello!").

In your question's case, tokenize() is member method, (it's not static) and it needs to be called on a object, so you have to first create that object. The reason for this, is because you can further customize what will happen during the tokenization, by passing some arguments in the constructor of the object. Check the link above to see all possible arguments.

Happy coding!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .