Earlier, I was looking up questions about Pset6 to see if there are any answers that can help me with my code, however, it looks like I completely missed the memo for using set with similarities. My new question is: do I have to use set for my code, and if not, how do I fix my errors such as "AttributeError: '_io.TextIOWrapper' object has no attribute 'sent_tokenize'".

My code is as follows:

from nltk.tokenize import sent_tokenize

def lines(a, b):
"""Return lines in both a and b"""
    a_file = open("a", "r+")
    b_file = open("b", "r+")
    a_list = a_file.readlines()
    b_list = b_file.readlines()
    matching_lines = []
    for i in range(len(a_list)):
        if a_list[i] == b_list[i]:
            if a_list[i] not in matching_lines:
    return matching_lines

def sentences(a, b):
"""Return sentences in both a and b"""
    a_file = open("a", "r+")
    b_file = open("b", "r+")
    a_list = a_file.readlines()
    b_list = b_file.readlines()
    matching_sent = []
    for i in range(len(sent_tokenize(a_list))):
        if sent_tokenize(a_list)[i] == sent_tokenize(b_list)[i]:
            if sent_tokenize(a_list)[i] not in matching_sent:
    return matching_sent

def substrings(a, b, n):
"""Return substrings of length n in both a and b"""
    matching_subs = []
    a_file = open("a", "r+")
    b_file = open("b", "r+")
    for i in range(len(a_file)):
        a_list = a_file[i:n+i]
    for i in range(len(b_file)):
        b_list = b_file[i:n+i]
    if a_list[i] == b_list[i]:
        if a_list[i] not in matching_subs:
    return matching_subs

I realize that my code for sentences is very different compared to the rest of my functions, I was playing around with tokenize because I don't think I fully understand how it should be implemented. For instance, is it new_variable.sent_tokenize(a_list), a_list.sent_tokenize(), or neither? Or if I have to use set, does that even matter?

Thank you for your help!


You could indeed utilise the Python's build-in Set type, although not compulsory, together with its intersection method for the purpose of filtering within all three functions.

However, the use of sent_tokenize is imperative as the manual implantation of its equivalent would be bit more complex and is out of scope for the problem set.

The outlined exception AttributeError: '_io.TextIOWrapper is probably caused due to the misemploy of the function as a method. The following line of code illustrates an example usage which successfully yield the list ['Good muffins cost $3.88 in New York.', 'Could you please buy me two of them?', 'Thanks!']

sent_tokenize('Good muffins cost $3.88 in New York. Could you please buy me two of them? Thanks!')

It would be great if you could post the complete error message to abet a more successful diagnose of the issue.

  • When I use tokenize now as sent_tokenize(insert_variable) I get a huge piece of text in my terminal like, "File "test.py", line 9, in <module> for i in range(len(sent_tokenize(a_list))): File "/opt/pyenv/versions/3.6.0/lib/python3.6/site-packages/nltk/tokenize/__init__.py", line 97, in sent_tokenize return tokenizer.tokenize(text) [etc]", what does this mean? I assume that's not how tokenize outputs, so is that something I need to fix? – Tenacity Apr 12 '19 at 15:28
  • The sent_tokenize expects a string as the argument but inferring from the the error extract len(sent_tokenize(a_list)) particularly the variable name a_list I believe its being parameterised a list instead. This is apparent if the last line of the error resembles an equivalent of TypeError: expected string or bytes-like object. If the last line of error differs from the preceding please comment the very last line of the error. – Ashen Gunaratne Apr 13 '19 at 13:58

the TL;DR: is that sent.tokenize() wants you to send it a string of English text, and it will spit back a list of the sentences contained in that long string (aka a list of strings). See here for an accessible overview of the tokenizing functions of NLTK.

so if I have a string of text

>>> string = "Python is cool. Python is fun. Python is great."

then I can do

>>> phrases = sent_tokenize(string)
>>> print(phrases)
['Python is cool.', 'Python is fun.', 'Python is great.']

In fact, we could've even done string = sent_tokenize(string) if we don't need to keep the original long string.

One important thing that I think should be pointed out about your code: In each function you're trying to open a and b, but if you look at the compare function of application.py you'll see that those files were already opened and stored in the variables that are passed to the functions you're working on.

So your functions are receiving long strings (the contents of the files to be compared) and you're trying to open those as though they were file addresses.

With regards to how to use set, I'd suggest that you use the terminal on your computer or in the IDE (rather than writing code and running it) to do little practice tests for learning how different functions and methods work. For example, here is a little example I could do in my terminal to learn more about set(), by creating a list with some duplicate numbers, converting it into a set, and then back into a list:

>>> nums = [1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]
>>> nums
[1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]

>>> nums = set(nums)
>>> nums
set([1, 2, 3, 4, 5, 6, 7, 8])

>>> nums = list(nums)
>>> nums = [1, 2, 3, 4, 5, 6, 7, 8]

bonus: you can also use dictionaries to remove duplicate values from a list. Since a dictionary is a series of key-value pairs where keys must be unique, you can take a list and create a dictionary using the values of the list as the keys of the dictionary which automatically trims the duplicate. Then you just convert back to a list. Here's what it would look like in the terminal:

>>> nums = [1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]
>>> nums
[1, 1, 2, 3, 4, 4, 5, 5, 5, 5, 6, 7, 8]

>>> nums = dict.withkeys(nums)
>>> nums
{1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None}

>>> nums = list(nums)
[1, 2, 3, 4, 5, 6, 7, 8]
  • This was really helpful, and I was able to fix my problems with sentences. However, substrings keeps returning with "IndexError: string index out of range" for the code if a_list[i] == b_list[i]:. Why is that an error and how can I fix it? – Tenacity Apr 23 '19 at 15:35
  • @Tenacity If you're getting a "String index out of range" error, that means you're trying to reference a spot of a string that doesn't exist. If a_list has 30 elements and b_list has 28 elements, then you would expect to get this error on the last two checks, when you try to access the 29th and 30th elements of b_list. I think that itself should suggest how to fix it, but let me know if you still want a pointer for that! – Gonzo Nieto Apr 24 '19 at 20:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .