Grading Locally¶
The command line interface allows instructors to grade notebooks locally by launching Docker containers on the instructor’s machine that grade notebooks and return a CSV of grades and (optionally) PDF versions of student submissions for manually graded questions.
Configuration Files¶
Otter grades students submissions in individual Docker containers that are based on a Docker image
generated through the use of a configuration zip file. Before grading assignments locally, an
instructor should create such a zip file by using a tool such as Otter Assign or Otter Generate. This file will be
used in the construction of a Docker image tagged otter-grader:{zip file hash}
. This Docker
image will then have containers spawned from it for each submission that is graded.
Otter’s Docker images can be pruned with otter grade --prune
.
Using the CLI¶
Before using the command line utility, you should have
written tests for the assignment,
generated a configuration zip file from those tests, and
downloaded submissions into a directory
The grading interface, encapsulated in the otter grade
command, runs the local grading process
and defines the options that instructors can set when grading. A comprehensive list of flags is
provided in the CLI Reference.
Basic Usage¶
The simplest usage of the Otter Grade is when we have a directory structure as below (and we have
change directories into grading
in the command line) and we don’t require PDFs or additional
requirements.
grading
├── autograder.zip
├── nb0.ipynb
├── nb1.ipynb
├── nb2.ipynb # etc.
└── tests
├── q1.py
├── q2.py
└── q3.py # etc.
In the case above, our otter command would be, very simply,
otter grade
Because the submissions are on the current working directory (grading
), our configuration file
is at ./autograder.zip
, and we don’t mind output to ./
, we can use the defualt values of the
-a
and -o
flags.
After grader, our directory will look like this:
grading
├── autograder.zip
├── final_grades.csv
├── nb0.ipynb
├── nb1.ipynb
├── nb2.ipynb # etc.
└── tests
├── q1.py
├── q2.py
└── q3.py # etc.
and the grades for each submission will be in final_grades.csv
.
If we wanted to generate PDFs for manual grading, we would specify this when making the
configuration file and add the --pdfs
flag to tell Otter to copy the PDFs out of the containers:
otter grade --pdfs
and at the end of grading we would have
grading
├── autograder.zip
├── final_grades.csv
├── nb0.ipynb
├── nb1.ipynb
├── nb2.ipynb # etc.
├── submission_pdfs
│ ├── nb0.pdf
│ ├── nb1.pdf
│ └── nb2.pdf # etc.
└── tests
├── q1.py
├── q2.py
└── q3.py # etc.
To grade a single notebook(as opposed to all the notebooks in a directory), you can pass the path to the
notebook file using the flag -p
. For example, using the directory structure above, the call looks
like this:
otter grade -p "./nb0.ipynb"
The usual information for this graded notebook is written to final_grades.csv
but the percent correct is returned
to the command-line from the call as well.
To grade submissions that aren’t notebook files, use the --ext
flag, which accepts the file
extension to search for submissions with. For example, if we had the same example as above but with
Rmd files:
otter grade --ext Rmd
If you’re grading submission export zip files (those generated by otter.Notebook.export
or
ottr::export
), your otter_config.json
should have its zips
key set to true, and you
should pass the -z
flag to Otter Grade. You also won’t need to use the --ext
flag regardless
of the type of file being graded inside the zip file.
Requirements¶
The Docker image used for grading will be built as described in the Otter Generatte section. If you require any packages not listed there, or among the dependencies of any packages above, you should create a requirements.txt file containing only those packages and use it when running your configuration generator.
Support Files¶
Some notebooks require support files to run (e.g. data files). If your notebooks require any such files, you should generate your configuration zip file with those files.
Intercell Seeding¶
Otter Grade also supports intercell seeding. This behavior should be configured as a part of your configuration zip file.