Command Line Utility¶
The command line interface allows instructors to grade notebooks locally by launching Docker containers on the instructor’s machine that grade notebooks and return a CSV of grades and (optionally) PDF versions of student submissions for manually graded questions.
Before using the command line utility, you should have
- written tests for the assignment
- downloaded submissions into a directory
- written a metadata file if not using a Gradescope or Canvas export
Using the CLI¶
The CLI, encapsulated in the
otter command, runs the local grading process and defines the options that instructors can set when grading. A comprehensive list of flags is provided below.
||Show help message and exit|
||Path to directory of submissions|
||Path to directory of tests|
||Path at which to write output (CSV and directory of PDFs)|
||Indicates a Gradescope export format for submissions|
||Indicates a Canvas export format for submissions|
||Path to JSON metadata file|
||Path to YAML metadata file|
||Indicates that Python scripts are being executed (not IPython notebooks)|
|Generate unfiltered PDFs for manual grading|
||Generate PDFs filtered by cell tags for manual grading|
||Generate PDFs filtered by HTML comments for manual grading|
||Path to any support files needed for execution (e.g. data files)|
||Write verbose output to console|
||Path to requirements.txt file|
||4||Number of parallel containers to launch; submissions will be divided evenly among them|
||ucbdsinfra/otter-grader||Docker image on which to grade submissions|
||Prevents containers from being killed after execution for debugging|
The simplest usage of the
otter command is when we have a directory structure as below (and we have
grading in the command line) and we don’t require PDFs or additional requirements.
| grading | - meta.yml | - nb0.ipynb | - nb1.ipynb | - nb2.ipynb ... | tests | - q1.py | - q2.py | - q3.py ...
In the case above, our otter command would be, very simply,
otter -y meta.yml
Because the submissions are on the current working directory (
grading), our tests are at
./tests, and we don’t mind output to
./, we can use the defualt values of the
-o flags, leaving the only necessary flag the metadata flag. Since we have a YAML metadata file, we specify
-y and pass the path to the metadata file,
After grader, our directory will look like this:
| grading | - final_grades.csv | - meta.yml | - nb0.ipynb | - nb1.ipynb | - nb2.ipynb ... | tests | - q1.py | - q2.py | - q3.py ...
and the grades for each submission will be in
If we wanted to generate PDFs for manual grading, we would specify one of the three PDF flags:
otter -y meta.yml --pdf
and at the end of grading we would have
| grading | - final_grades.csv | - meta.yml | - nb0.ipynb | - nb1.ipynb | - nb2.ipynb ... | manual_submissions | - nb0.pdf | - nb1.pdf | - nb2.pdf ... | tests | - q1.py | - q2.py | - q3.py ...
The four metadata flags,
-y, correspond to different export/metadata file formats. Also note that the latter two require you to specify a path to the metadata file. You must specify a metadata flag every time you run
otter, and you may not specify more than one. For more information about metadata and export formats, see this page.
ucbdsinfra/otter-grader Docker image comes preinstalled with the following Python packages and their dependencies:
If you require any packages not listed above, or among the dependencies of any packages above, you should create a requirements.txt file containing only those packages. If this file is created in the working directory (i.e.
./requirements.txt), then Otter will automatically find this file and include it. If this file is not at
./requirements.txt, pass its path to the
For example, continuining the example above with the package SymPy, I would create a requirements.txt
| grading | - meta.yml | - nb0.ipynb | - nb1.ipynb | - nb2.ipynb ... | - requirements.txt | tests | - q1.py | - q2.py | - q3.py ...
that lists only SymPy
$ cat requirements.txt sympy
otter call, using HTML comment filtered PDF generation this time, would become
otter -y meta.yml --html-filter
Note the lack of the
-r flag; since I created my requirements file in the working directory, Otter found it automatically.
Grading Python Scripts¶
If I wanted to grade Python scripts instead of IPython notebooks, my call to
otter would only add the
-s flag. Consider the directory structure below:
| grading | - meta.yml | - sub0.py | - sub1.py | - sub2.py ... | - requirements.txt | tests | - q1.py | - q2.py | - q3.py ...
My call to grade these submissions would be
otter -sy meta.yml
Note the lack of a PDF flag, as it doesn’t make sense to convert Python files to PDFs. PDF flags only work when grading IPython Notebooks.
Some notebooks require support files to run (e.g. data files). If your notebooks require any such files, there are two ways to get them into the container so that they are available to notebooks:
- specifying paths to the files with the
- putting them into the notebook path
Suppose that my notebooks in
data.csv in my
| data | - data.csv | grading | - meta.yml | - nb0.ipynb | - nb1.ipynb | - nb2.ipynb ... | - requirements.txt | tests | - q1.py | - q2.py | - q3.py ...
I could pass this data into the container using the
otter -y meta.yml -f ../data/data.csv
Or I could move (or copy)
$ mv ../data/data.csv ./
| data | grading | - data.csv | - meta.yml | - nb0.ipynb | - nb1.ipynb | - nb2.ipynb ... | - requirements.txt | tests | - q1.py | - q2.py | - q3.py ...
and then just run
otter as normal:
otter -y meta.yml
All non-notebook files in the notebooks path are copied into all of the containers, so
data.csv will be made available to all notebooks.