Grading Locally

The command line interface allows instructors to grade notebooks locally by launching Docker containers on the instructor’s machine that grade notebooks and return a CSV of grades and (optionally) PDF versions of student submissions for manually graded questions.

Configuration Files

Otter grades students submissions in individual Docker containers that are based on a Docker image generated through the use of a configuration zip file. Before grading assignments locally, an instructor should create such a zip file by using a tool such as Otter Assign or Otter Generate. This file will be used in the construction of a Docker image tagged otter-grader:{zip file hash}. This Docker image will then have containers spawned from it for each submission that is graded.

Otter’s Docker images can be pruned with otter grade --prune.

Metadata

Metadata files and flags are optional, but can be helpful when using the CSVs generated by Otter for grading. When organizing submissions for grading, Otter supports two major categories of exports:

  • 3rd party collection exports, or

  • instructor-made metadata files

These two categories are broken down and described below.

3rd Party Exports

If you are using a grading service like Gradescope or an LMS like Canvas to collect submissions, Otter can interpret the export format of these 3rd party services in place of a metadata file. To use a service like Gradescope or Canvas, download the submissions for an assignment, unzip the provided file, and then proceed as described in below using the metadata flag corresponding to your chosen service.

For example, if I had a Canvas export, I would unzip it, cd into the unzipped directory, copy my tests to ./tests, and run

otter grade -c

to grade the submissions (assuming no extra requirements, no PDF generation, and no support files required).

Metadata Files

If you are not using a service like Gradescope or Canvas to collect submissions, instructors can also create a simple JSON- or YAML-formatted metadata file for their students’ submissions and provide this to Otter.

The strucure of the metadata is quite simple: it is a list of 2-item dictionaries, where each dictionary has a student identifier stored as the identifier key and the filename of the student’s submission as filename. An example of each is provided below.

YAML format:

- identifier: 0
  filename: test-nb-0.ipynb
- identifier: 1
  filename: test-nb-1.ipynb
- identifier: 2
  filename: test-nb-2.ipynb
- identifier: 3
  filename: test-nb-3.ipynb
- identifier: 4
  filename: test-nb-4.ipynb

JSON format:

[
  {
    "identifier": 0,
    "filename": "test-nb-0.ipynb"
  },
  {
    "identifier": 1,
    "filename": "test-nb-1.ipynb"
  },
  {
    "identifier": 2,
    "filename": "test-nb-2.ipynb"
  },
  {
    "identifier": 3,
    "filename": "test-nb-3.ipynb"
  },
  {
    "identifier": 4,
    "filename": "test-nb-4.ipynb"
  }
]

A JSON- or YAML-formatted metadata file is specified to Otter using the -j or -y flag, respectively. Each flag requires a single argument that corresponds to the path to the metadata file.

Using the CLI

Before using the command line utility, you should have

  • written tests for the assignment,

  • generated a configuration zip file from those tests, and

  • downloaded submissions into a directory

The grading interface, encapsulated in the otter grade command, runs the local grading process and defines the options that instructors can set when grading. A comprehensive list of flags is provided below.

Basic Usage

The simplest usage of the Otter Grade is when we have a directory structure as below (and we have cded into grading in the command line) and we don’t require PDFs or additional requirements.

| grading
  | - autograder.zip
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

In the case above, our otter command would be, very simply,

otter grade -y meta.yml

Because the submissions are on the current working directory (grading), our configuration file is at ./autograder.zip, and we don’t mind output to ./, we can use the defualt values of the -a and -o flags, leaving the only necessary flag the metadata flag. Since we have a YAML metadata file, we specify -y and pass the path to the metadata file, ./meta.yml.

After grader, our directory will look like this:

| grading
  | - autograder.zip
  | - final_grades.csv
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

and the grades for each submission will be in final_grades.csv.

If we wanted to generate PDFs for manual grading, we would specify this when making the configuration file and add the --pdfs flag to tell Otter to copy the PDFs out of the containers:

otter grade -y meta.yml --pdfs

and at the end of grading we would have

| grading
  | - autograder.zip
  | - final_grades.csv
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | submission_pdfs
    | - nb0.pdf
    | - nb1.pdf
    | - nb2.pdf
    ...
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

Requirements

The Docker image used for grading will be built as described in the Otter Generatte section. If you require any packages not listed there, or among the dependencies of any packages above, you should create a requirements.txt file containing only those packages and use it when running your configuration generator.

Support Files

Some notebooks require support files to run (e.g. data files). If your notebooks require any such files, you should generate your configuration zip file with those files.

Intercell Seeding

Otter Grade also supports intercell seeding. This behavior should be configured as a part of your configuration zip file.