Otter has two major use cases: grading on the instructor’s machine (local grading), and generating files to use Gradescope’s autograding infrastructure.
Collecting Student Submissions¶
The first major decision is how you’ll collect student submissions. You can collect these however you want, although otter has builtin compatibility with Gradescope and Canvas. If you choose either Gradescope or Canvas, just export the submissions and unzip that into some directory. If you are collecting another way, you’ll need to create a metadata file. You can use either JSON or YAML format, and the structure is pretty simple: each element needs to have a filename and a student identifier. A sample YAML metadata file would be:
- identifier: 0 filename: test-nb-0.ipynb - identifier: 1 filename: test-nb-1.ipynb - identifier: 2 filename: test-nb-2.ipynb - identifier: 3 filename: test-nb-3.ipynb - identifier: 4 filename: test-nb-4.ipynb - identifier: 5 filename: test-nb-5.ipynb ...
If you have any files that are needed by the notebooks (e.g. data files), put these into the directory that contains the notebooks. You should also have your directory of tests nearby. At this stage, your directory should probably look something like this:
| grading | - requirements.txt | submissions | - meta.yml | - nb0.ipynb | - nb1.ipynb | - nb2.ipynb | - nb3.ipynb | - nb4.ipynb | - nb5.ipynb | - data.csv | tests | - q1.py | - q2.py | - q3.py | - q4.py
Now that you’ve set up the directory, let’s get down to grading. Go to the terminal,
cd into your grading directory (
grading in the example above), and let’s build the
otter command. The first thing we need is our notebooks path or
-p argument. Otter assumes this is
./, but our notebooks are located in
./submissions, so we’ll need
-p submissions in our command. We also need a tests directory, which otter assumes is at
./tests; because this is where our tests are, we’re alright on this front, and don’t need a
Now we need to tell otter how we’ve structured our metadata. If you’re using a Gradescope or Canvas export, just pass the
-c flag, respectively, with no arguments. If you’re using a custom metadata file, as in our example, pass the
-y flag with the path to the metadata file as its argument; in our case, we will pass
At this point, we need to make a decision: do we want PDFs? If there are questions that need to be manually graded, it might be nice to generate a PDF of each submission so that it can be easily read; if you want to generate PDFs, pass one of the
--html-filter flags (cf. PDFs). For our example, let’s say that we do want PDFs.
Now that we’ve made all of these decisions, let’s put our command together. Our command is:
otter -p submissions -y meta.yml --pdf -v
Note that Otter automatically found our requirements file at
./requirements.txt. If it had been in a different location, we would have needed to pass the path to it to the
-r flag. Note also that we pass the
-v flag so that it prints verbose output. Once this command finishes running, you will end up with a new file and a new folder in your working directory:
| grading | - final_grades.csv | - requirements.txt | manual_submissions | - nb0.pdf | - nb1.pdf | - nb2.pdf | - nb3.pdf | - nb4.pdf | - nb5.pdf | submissions ... | tests ...
Otter created the
final_grades.csv file with the grades for each student, broken down by test, and the
manual_submissions directory to house the PDF that was generated of each notebook.
Congrats, you’re done! You can use the grades in the CSV file and the PDFs to complete grading however you want.
You can find more information about the command line utility here
To get started using otter with Gradescope, create some test cases and a requirements.txt file (if necessary). Once you have these pieces in place, put them into a directory along with any additional files that your notebook requires (e.g. data files), for example:
| gradescope | - data.csv | - requirements.txt | - utils.py | tests | - q1.py | - q2.py ...
To create the zipfile for Gradescope, use the
otter gen command after
cding into the directory you created. For the directory above, once I’ve
gradescope, I would run the following to generate the zipfile:
otter gen data.csv utils.py
As above, Otter automatically found our requirements file at
./requirements.txt. Notice also that we didn’t indicate the path to the tests directory; this is because the default argument of the
-t flag is
./tests, so otter found them automatically.
After this command finishes running, you should have a file called
autograder.zip in the current working directory:
| gradescope | - autograder.zip | - data.csv | - requirements.txt | - utils.py | tests | - q1.py | - q2.py ...
To use this zipfile, create a Programming Assignment on Gradescope and upload this zipfile on the Configure Autograder page of the assignment. Gradescope will then build a Docker image on which it will grade each student’s submission.
You can find more information about Gradescope usage here.