Tokenization problems in CS50ai Project 8 (Questions)

Question

I need help figuring out what's wrong with my code in the last project in CS50ai. I've failed like 5 times now my submissions and I'm so frustrated. lol. Obviously I haven't submitted the same code every time, which is why I'm asking for help since every submission works as asked in the specifications* (I think).

*As the examples shown in the projects website and the specifications, my submissions all worked as "expected"

Every submissions has had the same response "Your code does not handle all tokenization correctly, which can lead to parsing and interpretation errors. Do review the specification's requirements for appropriate tokenization: https://cs50.harvard.edu/ai/2020/projects/6/questions/#specification"

My tokenization function is this:

def tokenize(document):
    """
    Given a document (represented as a string), return a list of all of the
    words in that document, in order.

    Process document by coverting all words to lowercase, and removing any
    punctuation or English stopwords.
    """
    
    tokens = nltk.word_tokenize(document.lower())
    new_tokens = []

    for word in tokens:
        if word in nltk.corpus.stopwords.words("english"):
            continue
        if word in string.punctuation:
            continue
        new_tokens.append(word)

    return sorted(new_tokens)

I first had the idea that every punctuation had to be removed, from every word; solution which I submitted and failed. I also tried removing every word that's not "a word", like """" or -----. But searching and asking on Discord, people said that only words that are one punctuation character had to be removed, even words like """" should not be removed.

With this tokenize function I get the results expected using the example queries on the projects website, along with the rest of my code.

If anybody has been in the same situation or passed the project review, any hint or comment will be appreciated.

Answer 1

@Sebastian, you still working on this project? Your code looks like it should work (although the if logic needs clean-up/simplification). That said, I have one question: why did you use sorted() on the returned value?

The instructions and docstring comments say: "return a list of all of the words in that document, in order". It does not say sorted in alphabetical order.

I don't think this changes the function's behavior, but could trigger the response about "Your code does not handle all tokenization correctly"