2 votes

Pset6 Similaritites Less Comfortable

You could indeed utilise the Python's build-in Set type, although not compulsory, together with its intersection method for the purpose of filtering within all three functions. However, the use of ...
user avatar
1 vote

Pset6 Similaritites Less Comfortable

the TL;DR: is that sent.tokenize() wants you to send it a string of English text, and it will spit back a list of the sentences contained in that long string (aka a list of strings). See here for an ...
user avatar
1 vote
Accepted

Tokenizer adds space before text

Are you sure it's adding a space, and you're not just printing a space using print? Consider print('token: "{}"'.format(token)) Added " so it's clear where the token begins and ends. Is there an ...
user avatar
  • 20.7k
1 vote
Accepted

TweetTokenizer() vs split(), tokens

As far as I can tell, punctuation. "Hip Hip Hooray!".split() would return a list of 3 tokens (['Hip','Hip','Hooray!']). tokenize("Hip Hip Hooray!") returns a list of 4 tokens (['Hip', 'Hip', 'Hooray','...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible