Pset6 Similaritites Less Comfortable

Question

Earlier, I was looking up questions about Pset6 to see if there are any answers that can help me with my code, however, it looks like I completely missed the memo for using set with similarities. My new question is: do I have to use set for my code, and if not, how do I fix my errors such as "AttributeError: '_io.TextIOWrapper' object has no attribute 'sent_tokenize'".

My code is as follows:

from nltk.tokenize import sent_tokenize

def lines(a, b):
"""Return lines in both a and b"""
    a_file = open("a", "r+")
    b_file = open("b", "r+")
    a_list = a_file.readlines()
    b_list = b_file.readlines()
    matching_lines = []
    for i in range(len(a_list)):
        if a_list[i] == b_list[i]:
            if a_list[i] not in matching_lines:
                matching_lines.append(a_list[i])
    return matching_lines


def sentences(a, b):
"""Return sentences in both a and b"""
    a_file = open("a", "r+")
    b_file = open("b", "r+")
    a_list = a_file.readlines()
    b_list = b_file.readlines()
    matching_sent = []
    for i in range(len(sent_tokenize(a_list))):
        if sent_tokenize(a_list)[i] == sent_tokenize(b_list)[i]:
            if sent_tokenize(a_list)[i] not in matching_sent:
                matching_sent.append(sent_tokenize(a_list)[i])
    return matching_sent


def substrings(a, b, n):
"""Return substrings of length n in both a and b"""
    matching_subs = []
    a_file = open("a", "r+")
    b_file = open("b", "r+")
    for i in range(len(a_file)):
        a_list = a_file[i:n+i]
    for i in range(len(b_file)):
        b_list = b_file[i:n+i]
    if a_list[i] == b_list[i]:
        if a_list[i] not in matching_subs:
            matching_subs.append(a_list[i])
    return matching_subs

I realize that my code for sentences is very different compared to the rest of my functions, I was playing around with tokenize because I don't think I fully understand how it should be implemented. For instance, is it new_variable.sent_tokenize(a_list), a_list.sent_tokenize(), or neither? Or if I have to use set, does that even matter?

Thank you for your help!

Answer 1

You could indeed utilise the Python's build-in Set type, although not compulsory, together with its intersection method for the purpose of filtering within all three functions.

However, the use of sent_tokenize is imperative as the manual implantation of its equivalent would be bit more complex and is out of scope for the problem set.

The outlined exception AttributeError: '_io.TextIOWrapper is probably caused due to the misemploy of the function as a method. The following line of code illustrates an example usage which successfully yield the list ['Good muffins cost $3.88 in New York.', 'Could you please buy me two of them?', 'Thanks!']

sent_tokenize('Good muffins cost $3.88 in New York. Could you please buy me two of them? Thanks!')

It would be great if you could post the complete error message to abet a more successful diagnose of the issue.

Answer 2

the TL;DR: is that sent.tokenize() wants you to send it a string of English text, and it will spit back a list of the sentences contained in that long string (aka a list of strings). See here for an accessible overview of the tokenizing functions of NLTK.

so if I have a string of text

>>> string = "Python is cool. Python is fun. Python is great."

then I can do

>>> phrases = sent_tokenize(string)
>>> print(phrases)
['Python is cool.', 'Python is fun.', 'Python is great.']

In fact, we could've even done string = sent_tokenize(string) if we don't need to keep the original long string.

One important thing that I think should be pointed out about your code: In each function you're trying to open a and b, but if you look at the compare function of application.py you'll see that those files were already opened and stored in the variables that are passed to the functions you're working on.

So your functions are receiving long strings (the contents of the files to be compared) and you're trying to open those as though they were file addresses.

With regards to how to use set, I'd suggest that you use the terminal on your computer or in the IDE (rather than writing code and running it) to do little practice tests for learning how different functions and methods work. For example, here is a little example I could do in my terminal to learn more about set(), by creating a list with some duplicate numbers, converting it into a set, and then back into a list:

>>> nums = [1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]
>>> nums
[1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]

>>> nums = set(nums)
>>> nums
set([1, 2, 3, 4, 5, 6, 7, 8])

>>> nums = list(nums)
>>> nums = [1, 2, 3, 4, 5, 6, 7, 8]

bonus: you can also use dictionaries to remove duplicate values from a list. Since a dictionary is a series of key-value pairs where keys must be unique, you can take a list and create a dictionary using the values of the list as the keys of the dictionary which automatically trims the duplicate. Then you just convert back to a list. Here's what it would look like in the terminal:

>>> nums = [1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]
>>> nums
[1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]

>>> nums = dict.withkeys(nums)
>>> nums
{1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None}

>>> nums = list(nums)
[1, 2, 3, 4, 5, 6, 7, 8]