scourgify Lesson6 Problem Set check50 errors

Question

Description

I have been working on this solution for a couple of weeks now. I wrote the code to satisfy all requirements described in the problem set in about three hours. The rest of the time has been spent trying to get check50 to pass, which is a challenge because we do not know what is being tested.

Steps Taken

1. I completed manual testing including with before.csv deleted down to one line in addition to the headers. All tests pass as expected.
2. The "scourgify.py creates new CSV file" appears as a random failure, even with no code changes e.g. logging out for the night and logging in the next day.
3. The "scourgify.py cleans short CSV file, scourgify.py does not produce CSV with specified format" test always fails. I have written code to both implicitly close the CSV files and to monitor sub-processes to confirm that exit code 0 is generated (not included in the code below).

Next steps

1. It would be useful to know what check50 is expecting e.g. as acceptance criteria.
2. I will try it on another computer in another location to see if that helps.
3. I was provided with the suggestion that the issue is possibly a buffering/flushing issue or perhaps a check50 timing error; neither of which I would have expected on an "Introduction to..." course.
4. If anyone can spot a glaring error in the code sample provided, that would be great! I am TRYING to learn.
5. I cannot find nothing in the tutorial video for File I/O that relates to small file issues or for "with" failing to close files. I presume the idea is to have folk trawl through the documentation. I have not found anything that relates to this though.
6. Are there other clean issues that we are supposed to be dealing with e.g. missing headers, other delimiters, non-utf data etc?

import sys
import csv


# scourgify.py to alter the sequence of strings in a .csv file
def main():
    new_format = []
    # Validate CLI input string
    if check_arguments_length(sys.argv):
        try:
            with open(sys.argv[1]) as before:
                reader = csv.DictReader(before)
                for row in reader:
                    last, first = row["name"].split(",")
                    new_format.append(
                        {
                            "first": first.strip(),
                            "last": last.strip(),
                            "house": row["house"].strip(),
                        }
                    )
        except FileNotFoundError:
            sys.exit("File does not exist")
        else:
            # Create and write to after.csv
            with open(sys.argv[2], "w") as after:
                writer = csv.DictWriter(after, fieldnames=["first", "last", "house"])
                # Add header names
                writer.writeheader()
                for row in sorted(
                    new_format, key=lambda new_format: new_format["first"]
                ):
                    writer.writerow(
                        {
                            "first": row["first"],
                            "last": row["last"],
                            "house": row["house"],
                        }
                    )


# Check that sys.argv is the right length
def check_arguments_length(args):
    if len(args) < 3:
        sys.exit("Too few command-line arguments")
    elif len(args) > 3:
        sys.exit("Too many command-line arguments")
    elif not args[1].endswith(".csv") or not args[2].endswith(".csv"):
        sys.exit("Not .csv files")
    else:
        return True


if __name__ == "__main__":
    main()

Answer 1

Success!

I managed to solve this in the end both by including the write loop inside of the read loop, also by including the delimiter and fieldnames parameters along with the if statement to check for fieldnames being recognised as data.

That way, there was no need to use an 'else:' clause.

Use of icecream for debugging was invaluable!

with open(sys.argv[1]) as before, open(sys.argv[2], "w") as after:
    reader = csv.DictReader(
        before, fieldnames=["name", "house"], delimiter=","
    )
    writer = csv.DictWriter(after, fieldnames=["first", "last", "house"])
    # This line writes the header row to the CSV file, based on the field names provided earlier
    writer.writeheader()
    for row in reader:
        # This 'if' statement checks for fieldnames being recognised as data
        if row == {"house": "house", "name": "name"}:
            continue
        last, first = row["name"].split(",")
        new_format = {
            first": first.lstrip(),
            "last": last,
            "house": row["house"],
        }
        # This line writes the content of the new_format dictionary as a row in the after CSV file
        writer.writerow(new_format)

Answer 2

This works for me! Turned out I need an lstrip()

def main():
try:
    if len(sys.argv) < 3:
        sys.exit("Too few command-line arguments")
    elif len(sys.argv) > 3:
        sys.exit("Too many command-line arguments")
    else:
        filename = sys.argv[1]
        file = read_file(filename)
        write_file(sys.argv[2], file)

except FileNotFoundError:
        sys.exit("Could not read invalid_file.csv")

def read_file(filename):
new_csv = []
with open(filename) as file:
    reader = csv.DictReader(file)
    for row in reader:
        last, first = row["name"].split(",")
        new_csv.append({"first": first.lstrip(), "last": last.lstrip(), "house": row["house"]})
return new_csv

def write_file(newfilename, data):
with open(newfilename, "w") as file:
    writer = csv.DictWriter(file, fieldnames = ["first","last", "house"])
    writer.writeheader()
    for row in data:
        writer.writerow(row)
return writer