Student Usage
Contents
Student Usage#
Otter provides an IPython API and a command line tool that allow students to run checks and export notebooks within the assignment environment.
The Notebook
API#
Otter supports in-notebook checks so that students can check their progress when working through
assignments via the otter.Notebook
class. The Notebook
takes one optional parameter that
corresponds to the path from the current working directory to the directory of tests; the default
for this path is ./tests
.
import otter
grader = otter.Notebook()
If my tests were in ./hw00-tests
, then I would instantiate with
grader = otter.Notebook("hw00-tests")
Students can run tests in the test directory using Notebook.check
which takes in a question
identifier (the file name without the .py
extension). For example,
grader.check("q1")
will run the test q1.py
in the tests directory. If a test passes, then the cell displays “All
tests passed!” If the test fails, then the details of the first failing test are printed out,
including the test code, expected output, and actual output:

Students can also run all tests in the tests directory at once using Notebook.check_all
:
grader.check_all()
This will rerun all tests against the current global environment and display the results for each tests concatenated into a single HTML output. It is recommended that this cell is put at the end of a notebook for students to run before they submit so that students can ensure that there are no variable name collisions, propagating errors, or other things that would cause the autograder to fail a test they should be passing.
Exporting Submissions#
Students can also use the Notebook
class to generate their own PDFs for manual grading using the
method Notebook.export
. This function takes an optional argument of the path to the notebook; if
unspecified, it will infer the path by trying to read the config file (if present), using the path
of the only notebook in the working directory if there is only one, or it will raise an error
telling you to provide the path. This method creates a submission zip file that includes the
notebook file, the log, and, optionally, a PDF of the notebook (set pdf=False
to disable this
last).
As an example, if I wanted to export hw01.ipynb
with cell filtering, my call would be
grader.export("hw01.ipynb")
as filtering is by defult on. If I instead wanted no filtering, I would use
grader.export("hw01.ipynb", filtering=False)
To generate just a PDF of the notebook, use Notebook.to_pdf
.
Running on Non-standard Python Environments#
When running on non-standard Python notebook environments (which use their own interpreters, such as
Colab or Jupyterlite), some Otter features are disabled due differences in file system access, the
unavailability of compatible versions of packages, etc. When you instantiate a Notebook
, Otter
automatically tries to determine if you’re running on one of these environments, but you can
manually indicate which you’re running on by setting either the colab
or jupyterlite
argument to True
.
Command Line Script Checker#
Otter also features a command line tool that allows students to run checks on Python files from the
command line. otter check
takes one required argument, the path to the file that is being
checked, and three optional flags:
-t
is the path to the directory of tests. If left unspecified, it is assumed to be./tests
-q
is the identifier of a specific question to check (the file name without the.py
extension). If left unspecified, all tests in the tests directory are run.--seed
is an optional random seed for execution seeding
The recommended file structure for using the checker is something like the one below:
hw00
├── hw00.py
└── tests
├── q1.py
└── q2.py # etc.
After a cd
into hw00
, if I wanted to run the test q2.py, I would run
$ otter check hw00.py -q q2
All tests passed!
In the example above, I passed all of the tests. If I had failed any of them, I would get an output like that below:
$ otter check hw00.py -q q2
1 of 2 tests passed
Tests passed:
possible
Tests failed:
*********************************************************************
Line 2, in tests/q2.py 0
Failed example:
1 == 1
Expected:
False
Got:
True
To run all tests at once, I would run
$ otter check hw00.py
Tests passed:
q1 q3 q4 q5
Tests failed:
*********************************************************************
Line 2, in tests/q2.py 0
Failed example:
1 == 1
Expected:
False
Got:
True
As you can see, I passed for of the five tests above, and filed q2.py.
If I instead had the directory structure below (note the new tests directory name)
hw00
├── hw00.py
└── hw00-tests
├── q1.py
└── q2.py # etc.
then all of my commands would be changed by adding -t hw00-tests
to each call. As an example,
let’s rerun all of the tests again:
$ otter check hw00.py -t hw00-tests
Tests passed:
q1 q3 q4 q5
Tests failed:
*********************************************************************
Line 2, in hw00-tests/q2.py 0
Failed example:
1 == 1
Expected:
False
Got:
True
otter.Notebook
Reference#
- class otter.check.notebook.Notebook(nb_path=None, tests_dir='./tests', tests_url_prefix=None, colab=None, jupyterlite=None)#
Notebook class for in-notebook autograding
- Parameters
nb_path (
str
, optional) – path to the notebook being runtests_dir (
str
, optional) – path to tests directorycolab (
bool
, optional) – whether this notebook is being run on Google Colab; ifNone
, this information is automatically parsed from IPython on creationjupyterlite (
bool
, optional) – whether this notebook is being run on JupyterLite; ifNone
, this information is automatically parsed from IPython on creation
- add_plugin_files(plugin_name, *args, nb_path=None, **kwargs)#
Runs the
notebook_export
event of the pluginplugin_name
and tracks the file paths it returns to be included when callingNotebook.export
.- Parameters
plugin_name (
str
) – importable name of an Otter plugin that implements thefrom_notebook
hook*args – arguments to be passed to the plugin
nb_path (
str
, optional) – path to the notebook**kwargs – keyword arguments to be passed to the plugin
- check(question, global_env=None)#
Runs tests for a specific question against a global environment. If no global environment is provided, the test is run against the calling frame’s environment.
- Parameters
question (
str
) – name of question being gradedglobal_env (
dict
, optional) – global environment resulting from execution of a single notebook
- Returns
the grade for the question
- Return type
otter.test_files.abstract_test.TestFile
- check_all()#
Runs all tests on this notebook. Tests are run against the current global environment, so any tests with variable name collisions will fail.
- export(nb_path=None, export_path=None, pdf=True, filtering=True, pagebreaks=True, files=[], display_link=True, force_save=False, run_tests=False)#
Exports a submission to a zip file. Creates a submission zipfile from a notebook at
nb_path
, optionally including a PDF export of the notebook and any files infiles
.- Parameters
nb_path (
str
, optional) – path to the notebook we want to export; will attempt to infer if not providedexport_path (
str
, optional) – path at which to write zipfile; defaults to notebook’s name +.zip
pdf (
bool
, optional) – whether a PDF should be includedfiltering (
bool
, optional) – whether the PDF should be filtered; ignored ifpdf
isFalse
pagebreaks (
bool
, optional) – whether pagebreaks should be included between questions in the PDFfiles (
list
ofstr
, optional) – paths to other files to include in the zip filedisplay_link (
bool
, optional) – whether or not to display a download linkforce_save (
bool
, optional) – whether or not to display JavaScript that force-saves the notebook (only works in Jupyter Notebook classic, not JupyterLab)
- run_plugin(plugin_name, *args, nb_path=None, **kwargs)#
Runs the plugin
plugin_name
with the specified arguments. Usenb_path
if the path to the notebook is not configured.- Parameters
plugin_name (
str
) – importable name of an Otter plugin that implements thefrom_notebook
hook*args – arguments to be passed to the plugin
nb_path (
str
, optional) – path to the notebook**kwargs – keyword arguments to be passed to the plugin
- to_pdf(nb_path=None, filtering=True, pagebreaks=True, display_link=True, force_save=False)#
Exports a notebook to a PDF using Otter Export
- Parameters
nb_path (
str
, optional) – path to the notebook we want to export; will attempt to infer if not providedfiltering (
bool
, optional) – set true if only exporting a subset of notebook cells to PDFpagebreaks (
bool
, optional) – if true, pagebreaks are included between questionsdisplay_link (
bool
, optional) – whether or not to display a download linkforce_save (
bool
, optional) – whether or not to display JavaScript that force-saves the notebook (only works in Jupyter Notebook classic, not JupyterLab)