Grading Locally

The command line interface allows instructors to grade notebooks locally by launching Docker containers on the instructor’s machine that grade notebooks and return a CSV of grades and (optionally) PDF versions of student submissions for manually graded questions.

Metadata

Metadata files and flags are optional, but can be helpful when using the CSVs generated by Otter for grading. When organizing submissions for grading, Otter supports two major categories of exports:

  • 3rd party collection exports, or

  • instructor-made metadata files

These two categories are broken down and described below.

3rd Party Exports

If you are using a grading service like Gradescope or an LMS like Canvas to collect submissions, Otter can interpret the export format of these 3rd party services in place of a metadata file. To use a service like Gradescope or Canvas, download the submissions for an assignment, unzip the provided file, and then proceed as described in Grading Locally using the metadata flag corresponding to your chosen service.

For example, if I had a Canvas export, I would unzip it, cd into the unzipped directory, copy my tests to ./tests, and run

otter grade -c

to grade the submissions (assuming no extra requirements, no PDF generation, and no support files required).

Metadata Files

If you are not using a service like Gradescope or Canvas to collect submissions, instructors can also create a simple JSON- or YAML-formatted metadata file for their students’ submissions and provide this to Otter.

The strucure of the metadata is quite simple: it is a list of 2-item dictionaries, where each dictionary has a student identifier stored as the identifier key and the filename of the student’s submission as filename. An example of each is provided below.

YAML format:

- identifier: 0
  filename: test-nb-0.ipynb
- identifier: 1
  filename: test-nb-1.ipynb
- identifier: 2
  filename: test-nb-2.ipynb
- identifier: 3
  filename: test-nb-3.ipynb
- identifier: 4
  filename: test-nb-4.ipynb

JSON format:

[
  {
    "identifier": 0,
    "filename": "test-nb-0.ipynb"
  },
  {
    "identifier": 1,
    "filename": "test-nb-1.ipynb"
  },
  {
    "identifier": 2,
    "filename": "test-nb-2.ipynb"
  },
  {
    "identifier": 3,
    "filename": "test-nb-3.ipynb"
  },
  {
    "identifier": 4,
    "filename": "test-nb-4.ipynb"
  }
]

A JSON- or YAML-formatted metadata file is specified to Otter using the -j or -y flag, respectively. Each flag requires a single argument that corresponds to the path to the metadata file. See the “Basic Usage” section of Grading Locally for more information.

Using the CLI

Before using the command line utility, you should have

  • written tests for the assignment

  • downloaded submissions into a directory

The grading interface, encapsulated in the otter grade command, runs the local grading process and defines the options that instructors can set when grading. A comprehensive list of flags is provided below.

Flag Default Value Description
-h, --help Show help message and exit
-p, --path ./ Path to directory of submissions
-t, --tests-path ./tests Path to directory of tests
-o, --output-path ./ Path at which to write output (CSV and directory of PDFs)
-g, --gradescope Indicates a Gradescope export format for submissions
-c, --canvas Indicates a Canvas export format for submissions
-j, --json Path to JSON metadata file
-y, --yaml Path to YAML metadata file
-s, --scripts Indicates that Python scripts are being executed (not IPython notebooks)
-z, --zips Indicates that the submissions are zip files formatted by otter.Notebook.export
--pdf Generate unfiltered PDFs for manual grading
--tag-filter Generate PDFs filtered by cell tags for manual grading
--html-filter Generate PDFs filtered by HTML comments for manual grading
-f, --files Path to any support files needed for execution (e.g. data files)
-v, --verbose Write verbose output to console
--seed A random seed for intercell seeding
-r, --requirements ./requirements.txt Path to requirements.txt file
--containers 4 Number of parallel containers to launch; submissions will be divided evenly among them
--image ucbdsinfra/otter-grader Docker image on which to grade submissions
--no-kill Prevents containers from being killed after execution for debugging

Basic Usage

The simplest usage of the Otter Grade is when we have a directory structure as below (and we have cded into grading in the command line) and we don’t require PDFs or additional requirements.

| grading
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

In the case above, our otter command would be, very simply,

otter grade -y meta.yml

Because the submissions are on the current working directory (grading), our tests are at ./tests, and we don’t mind output to ./, we can use the defualt values of the -p, -t, and -o flags, leaving the only necessary flag the metadata flag. Since we have a YAML metadata file, we specify -y and pass the path to the metadata file, ./meta.yml.

After grader, our directory will look like this:

| grading
  | - final_grades.csv
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

and the grades for each submission will be in final_grades.csv.

If we wanted to generate PDFs for manual grading, we would specify one of the three PDF flags:

otter grade -y meta.yml --pdf

and at the end of grading we would have

| grading
  | - final_grades.csv
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | submission_pdfs
    | - nb0.pdf
    | - nb1.pdf
    | - nb2.pdf
    ...
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

Metadata Flags

The four metadata flags, -g, -c, -j, and -y, correspond to different export/metadata file formats, and are optional. Also note that the latter two require you to specify a path to the metadata file. You must specify a metadata flag every time you run Otter Grade, and you may not specify more than one. For more information about metadata and export formats, see above. If you don’t specify a metadata flag, the CSV file that Otter returns will be primary keyed on the filename of the submission.

Requirements

The ucbdsinfra/otter-grader Docker image comes preinstalled with the following Python packages and their dependencies:

  • datascience

  • jupyter_client

  • ipykernel

  • matplotlib

  • pandas

  • ipywidgets

  • scipy

  • tornado

  • nb2pdf

  • otter-grader

If you require any packages not listed above, or among the dependencies of any packages above, you should create a requirements.txt file containing only those packages. If this file is created in the working directory (i.e. ./requirements.txt), then Otter will automatically find this file and include it. If this file is not at ./requirements.txt, pass its path to the -r flag.

For example, continuining the example above with the package SymPy, I would create a requirements.txt

| grading
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | - requirements.txt
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

that lists only SymPy

$ cat requirements.txt
sympy

Now my call, using HTML comment filtered PDF generation this time, would become

otter grade --html-filter

Note the lack of the -r flag; since I created my requirements file in the working directory, Otter found it automatically. Also note that I am no longer including a metadata flag; the CSV file will have a file column instead of an identifier column, primary keying each submission on the filename rather than a provided identifier.

Grading Python Scripts

If I wanted to grade Python scripts instead of IPython notebooks, my call to Otter would only add the -s flag. Consider the directory structure below:

| grading
  | - meta.yml
  | - sub0.py
  | - sub1.py
  | - sub2.py
  ...
  | - requirements.txt
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

My call to grade these submissions would be

otter grade -sy meta.yml

Note the lack of a PDF flag, as it doesn’t make sense to convert Python files to PDFs. PDF flags only work when grading IPython Notebooks.

Grading otter.Notebook.export Zip Files

Otter Grade supports the grading the zip archives produced by otter.Notebook.export. To grade these, just add the -z flags to your otter grade command and run as normal. For example, for the following grading directory,

| grading
  | - meta.yml
  | - sub0.zip
  | - sub1.zip
  | - sub2.zip
  ...
  | - requirements.txt
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

I would grade with

otter grade -zy meta.yml

Support Files

Some notebooks require support files to run (e.g. data files). If your notebooks require any such files, there are two ways to get them into the container so that they are available to notebooks:

  • specifying paths to the files with the -f flag

  • putting them into the notebook path

Suppose that my notebooks in grading required data.csv in my ../data directory:

| data
  | - data.csv
| grading
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | - requirements.txt
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

I could pass this data into the container using the call

otter grade -y meta.yml -f ../data/data.csv

Or I could move (or copy) data.csv into grading:

mv ../data/data.csv ./
| data
| grading
  | - data.csv
  | - meta.yml
  | - nb0.ipynb
  | - nb1.ipynb
  | - nb2.ipynb
  ...
  | - requirements.txt
  | tests
    | - q1.py
    | - q2.py
    | - q3.py
    ...

and then just run Otter as normal:

otter grade

All non-notebook files in the notebooks path are copied into all of the containers, so data.csv will be made available to all notebooks.

Intercell Seeding

Otter Grader also supports intercell seeding via the --seed flag. In notebooks, NumPy and Python’s random library are both seeded between every pair of code cells, so that the deterministic output can be used in writing hidden tests. In scripts, NumPy and random are seeded before the script’s execution. As an example, I can pass a seed to Otter with the above directory structure with

otter grade --seed 42

Otter Grade Reference

Grade assignments locally using Docker containers

usage: otter grade [-h] [-p PATH] [-t TESTS_PATH] [-o OUTPUT_PATH] [-g] [-c]
                   [-j JSON] [-y YAML] [-s] [-z] [--pdfs [{unfiltered,html}]]
                   [-f FILES [FILES ...]] [-v] [--seed SEED] [-r REQUIREMENTS]
                   [--containers CONTAINERS] [--image IMAGE] [--no-kill]
                   [--debug]

Named Arguments

-p, --path

Path to directory of submissions

Default: “./”

-t, --tests-path

Path to directory of tests

Default: “./tests/”

-o, --output-path

Path to which to write output

Default: “./”

-g, --gradescope

Flag for Gradescope export

-c, --canvas

Flag for Canvas export

-j, --json

Flag for path to JSON metadata

Default: False

-y, --yaml

Flag for path to YAML metadata

Default: False

-s, --scripts

Flag to incidicate grading Python scripts

-z, --zips

Whether submissions are zip files from Notebook.export

--pdfs

Possible choices: unfiltered, html

Default: False

-f, --files

Specify support files needed to execute code (e.g. utils, data files)

-v, --verbose

Flag for verbose output

--seed

A random seed to be executed before each cell

-r, --requirements

Flag for Python requirements file path; ./requirements.txt automatically checked

Default: “requirements.txt”

--containers

Specify number of containers to run in parallel

--image

Custom docker image to run on

Default: “ucbdsinfra/otter-grader”

--no-kill

Do not kill containers after grading

--debug

Print stdout/stderr from grading for debugging