Student Usage#

Otter provides an IPython API and a command line tool that allow students to run checks and export notebooks within the assignment environment.

The `Notebook` API#

Otter supports in-notebook checks so that students can check their progress when working through assignments via the otter.Notebook class. The Notebook takes one optional parameter that corresponds to the path from the current working directory to the directory of tests; the default for this path is ./tests.

import otter
grader = otter.Notebook()

If my tests were in ./hw00-tests, then I would instantiate with

grader = otter.Notebook("hw00-tests")

Students can run tests in the test directory using Notebook.check which takes in a question identifier (the file name without the .py extension). For example,

grader.check("q1")

will run the test q1.py in the tests directory. If a test passes, then the cell displays “All tests passed!” If the test fails, then the details of the first failing test are printed out, including the test code, expected output, and actual output:

Students can also run all tests in the tests directory at once using Notebook.check_all:

grader.check_all()

This will rerun all tests against the current global environment and display the results for each tests concatenated into a single HTML output. It is recommended that this cell is put at the end of a notebook for students to run before they submit so that students can ensure that there are no variable name collisions, propagating errors, or other things that would cause the autograder to fail a test they should be passing.

Exporting Submissions#

Students can also use the Notebook class to generate a zip file containing all of their work for submission with the method Notebook.export. This function takes an optional argument of the path to the notebook; if unspecified, it will infer the path by trying to read the config file (if present), using the path of the only notebook in the working directory if there is only one, or it will raise an error telling you to provide the path. This method creates a submission zip file that includes the notebook file, the log, and, optionally, a PDF of the notebook (set pdf=False to disable this last).

As an example, if I wanted to export hw01.ipynb with cell filtering, my call would be

grader.export("hw01.ipynb")

as filtering is by defult on. If I instead wanted no filtering, I would use

grader.export("hw01.ipynb", filtering=False)

To generate just a PDF of the notebook, use Notebook.to_pdf.

Both of these methods support force-saving the notebook before exporting it, so that any unsaved changes a student has made to their notebook will be reflected in the exported version. In Python, this works by using ipylab to communicate with the JupyterLab frontend. To use it, set force_save=True:

grader.export("hw01.ipynb", force_save=True)

In R, you must install the ottr_force_save_labextension Python package. This JupyterLab extension exposes a hook that ottr::export uses by running JavaScript to save the notebook.

ottr::export("hw01.ipynb", force_save=TRUE)

Force saving is not supported for Rmd files, and the argument is ignored if used when not running on Jupyter.

Running on Non-standard Python Environments#

When running on non-standard Python notebook environments (which use their own interpreters, such as Colab or Jupyterlite), some Otter features are disabled due differences in file system access, the unavailability of compatible versions of packages, etc. When you instantiate a Notebook, Otter automatically tries to determine if you’re running on one of these environments, but you can manually indicate which you’re running on by setting either the colab or jupyterlite argument to True.

Command Line Script Checker#

Otter also features a command line tool that allows students to run checks on Python files from the command line. otter check takes one required argument, the path to the file that is being checked, and three optional flags:

-t is the path to the directory of tests. If left unspecified, it is assumed to be ./tests
-q is the identifier of a specific question to check (the file name without the .py extension). If left unspecified, all tests in the tests directory are run.
--seed is an optional random seed for execution seeding

The recommended file structure for using the checker is something like the one below:

hw00
├── hw00.py
└── tests
    ├── q1.py
    └── q2.py  # etc.

After a cd into hw00, if I wanted to run the test q2.py, I would run

$ otter check hw00.py -q q2
All tests passed!

In the example above, I passed all of the tests. If I had failed any of them, I would get an output like that below:

$ otter check hw00.py -q q2
1 of 2 tests passed

Tests passed:
    possible


Tests failed:
*********************************************************************
Line 2, in tests/q2.py 0
Failed example:
    1 == 1
Expected:
    False
Got:
    True

To run all tests at once, I would run

$ otter check hw00.py
Tests passed:
    q1  q3  q4  q5


Tests failed:
*********************************************************************
Line 2, in tests/q2.py 0
Failed example:
    1 == 1
Expected:
    False
Got:
    True

As you can see, I passed for of the five tests above, and filed q2.py.

If I instead had the directory structure below (note the new tests directory name)

hw00
├── hw00.py
└── hw00-tests
    ├── q1.py
    └── q2.py  # etc.

then all of my commands would be changed by adding -t hw00-tests to each call. As an example, let’s rerun all of the tests again:

$ otter check hw00.py -t hw00-tests
Tests passed:
    q1  q3  q4  q5


Tests failed:
*********************************************************************
Line 2, in hw00-tests/q2.py 0
Failed example:
    1 == 1
Expected:
    False
Got:
    True

`otter.Notebook` Reference#

class otter.check.notebook.Notebook(nb_path=None, tests_dir='./tests', tests_url_prefix=None, colab=None, jupyterlite=None)#

Notebook class for in-notebook autograding

Parameters:

nb_path (str | None) – path to the notebook being run
tests_dir (str) – path to tests directory
tests_url_prefix (str | None) – a URL prefix to use to download test files
colab (bool | None) – whether this notebook is being run on Google Colab; if None, this information is automatically parsed from IPython on creation
jupyterlite (bool | None) – whether this notebook is being run on JupyterLite; if None, this information is automatically parsed from IPython on creation

add_plugin_files(plugin_name, *args, nb_path=None, **kwargs)#

Runs the notebook_export event of the plugin plugin_name and tracks the file paths it returns to be included when calling Notebook.export.

Parameters:

plugin_name (str) – importable name of an Otter plugin that implements the from_notebook hook
*args – arguments to be passed to the plugin
nb_path (str | None) – path to the notebook
**kwargs – keyword arguments to be passed to the plugin

check(question, global_env=None)#

Runs tests for a specific question against a global environment. If no global environment is provided, the test is run against the calling frame’s environment.

Parameters:

question (str) – name of question being graded
global_env (dict[str, object] | None) – a global environment in which to run the tests

Returns:

the grade for the question

Return type:

otter.test_files.abstract_test.TestFile

check_all()#: Runs all tests on this notebook. Tests are run against the current global environment, so any tests with variable name collisions will fail.

export(nb_path=None, export_path=None, pdf=True, filtering=True, pagebreaks=True, files=None, display_link=True, force_save=False, run_tests=False)#

Export a submission zip file.

Creates a submission zipfile from a notebook at nb_path, optionally including a PDF export of the notebook and any files in files.

If run_tests is true, the zip is validated by running it against local test cases in a separate Python process. The results of this run are printed to stdout.

Parameters:

nb_path (str | None) – path to the notebook we want to export; will attempt to infer if not provided
export_path (str | None) – path at which to write zipfile; defaults to {notebook name}_{timestamp}.zip
pdf (bool) – whether a PDF should be included
filtering (bool) – whether the PDF should be filtered
pagebreaks (bool) – whether pagebreaks should be included between questions in the PDF
files (list[str] | None) – paths to other files to include in the zip file
display_link (bool) – whether or not to display a download link
force_save (bool) – whether or not to display JavaScript that force-saves the notebook (only works in Jupyter Notebook classic, not JupyterLab)
run_tests (bool) – whether to validating the resulting submission zip against local test cases

run_plugin(plugin_name, *args, nb_path=None, **kwargs)#

Runs the plugin plugin_name with the specified arguments. Use nb_path if the path to the notebook is not configured.

Parameters:

plugin_name (str) – importable name of an Otter plugin that implements the from_notebook hook
*args – arguments to be passed to the plugin
nb_path (str, optional) – path to the notebook
**kwargs – keyword arguments to be passed to the plugin

to_pdf(nb_path=None, filtering=True, pagebreaks=True, display_link=True, force_save=False)#

Exports a notebook to a PDF using Otter Export

Parameters:

nb_path (str | None) – path to the notebook we want to export; will attempt to infer if not provided
filtering (bool) – set true if only exporting a subset of notebook cells to PDF
pagebreaks (bool) – if true, pagebreaks are included between questions
display_link (bool) – whether or not to display a download link
force_save (bool) – whether or not to display JavaScript that force-saves the notebook (only works in Jupyter Notebook classic, not JupyterLab)

Student Usage

Contents

Student Usage#

The Notebook API#

Exporting Submissions#

Running on Non-standard Python Environments#

Command Line Script Checker#

otter.Notebook Reference#

The `Notebook` API#

`otter.Notebook` Reference#