Non-containerized Grading#

Otter supports programmatic or command-line grading of assignments without requiring the use of Docker as an intermediary. This functionality is designed to allow Otter to run in environments that do not support containerization, such as on a user’s JupyterHub account. If Docker is available, it is recommended that Otter Grade is used instead, as non-containerized grading is less secure.

To grade locally, Otter exposes the otter run command for the command line or the module otter.api for running Otter programmatically. The use of both is described in this section. Before using Otter Run, you should have generated an autograder configuration zip file.

Otter Run works by creating a temporary grading directory using the tempfile library and replicating the autograder tree structure in that folder. It then runs the autograder there as normal. Note that Otter Run does not run environment setup files (e.g. setup.sh) or install requirements, so any requirements should be available in the environment being used for grading.

Grading from the Command Line#

To grade a single submission from the command line, use the otter run utility. This has one required argument, the path to the submission to be graded, and will run Otter in a separate directory structure created using tempfile. Use the optional -a flag to specify a path to your configuration zip file if it is not at the default path ./autograder.zip. Otter Run will write a JSON file, the results of grading, at {output_path}/results.json (output_path can be configured with the -o flag, and defaults to ./).

If I wanted to use Otter Run on hw00.ipynb, I would run

otter run hw00.ipynb

If my autograder configuration file was at ../autograder.zip, I would run

otter run -a ../autograder.zip hw00.ipynb

Either of the above will produce the results file at ./results.json.

For more information on the command-line interface for Otter Run, see the CLI Reference.

Grading Programmatically#

Otter includes an API through which users can grade assignments from inside a Python session, encapsulated in the submodule otter.api. The main method of the API is otter.api.grade_submission, which takes in an autograder configuration file path and a submission path and grades the submission, returning the GradingResults object that was produced during grading.

For example, to grade hw00.ipynb with an autograder configuration file in autograder.zip, I would run

from otter.api import grade_submission
grade_submission("hw00.ipynb", "autograder.zip")

grade_submission has an optional argument quiet which will suppress anything printed to the console by the grading process during execution when set to True (default False).

For more information about grading programmatically, see the API reference.

Grading Results#

This section describes the object that Otter uses to store and manage test case scores when grading.

class otter.test_files.GradingResults(test_files, notebook=None)#

Stores and wrangles test result objects

Initialize with a list of otter.test_files.abstract_test.TestFile subclass objects and this class will store the results as named tuples so that they can be accessed/manipulated easily. Also contains methods to put the results into a nice dict format or into the correct format for Gradescope.

Parameters

results (list of TestFile) – the list of test file objects summarized in this grade

results#

maps test names to GradingTestCaseResult named tuples containing the test result information

Type

dict

output#

a string to include in the output field for Gradescope

Type

str

all_hidden#

whether all results should be hidden from the student on Gradescope

Type

bool

tests#

list of test names according to the keys of results

Type

list of str

clear_results()#

Empties the dictionary of results

classmethod from_ottr_json(ottr_output)#

Creates a GradingResults object from the JSON output of Ottr.

Parameters

ottr_output (str) – the JSON output of Ottr as a string

Returns

the Ottr grading results

Return type

GradingResults

get_plugin_data(plugin_name, default=None)#

Retrieves data for plugin plugin_name in the results

This method uses dict.get to retrive the data, so a KeyError is never raised if plugin_name is not found; rather, it returns None.

Parameters

  • plugin_name (str) – the importable name of a plugin

  • default (any, optional) – a default value to return if plugin_name is not found

Returns

the data stored for plugin_name if found

Return type

any

get_result(test_name)#

Returns the TestFile corresponding to the test with name test_name

Parameters

test_name (str) – the name of the desired test

Returns

the graded test file object

Return type

TestFile

get_score(test_name)#

Returns the score of a test tracked by these results

Parameters

test_name (str) – the name of the test

Returns

the score

Return type

int or float

has_catastrophic_failure()#

Returns whether these results contain a catastrophic error (i.e. an error that prevented submission results from being generated or read).

Returns

whether there is such an error

Return type

bool

hide_everything()#

Indicates that all results should be hidden from students on Gradescope

property passed_all_public#

whether all public tests in these results passed

Type

bool

property possible#

the total points possible

Type

int or float

set_output(output)#

Updates the output field of the results JSON with text relevant to the entire submission. See https://gradescope-autograders.readthedocs.io/en/latest/specs/ for more information.

Parameters

output (str) – the output text

set_pdf_error(error: Exception)#

Set a PDF generation error to be displayed as a failed (0-point) test on Gradescope.

Parameters

error (Exception) – the error thrown

set_plugin_data(plugin_name, data)#

Stores plugin data for plugin plugin_name in the results. data must be picklable.

Parameters

  • plugin_name (str) – the importable name of a plugin

  • data (any) – the data to store; must be serializable with pickle

summary(public_only=False)#

Generate a summary of these results and return it as a string.

Parameters

public_only (bool, optional) – whether only public test cases should be included

Returns

the summary of results

Return type

str

property test_files#

the names of all test files tracked in these grading results

Type

list[TestFile]

to_dict()#

Converts these results into a dictinary, extending the fields of the named tuples in results into key, value pairs in a dict.

Returns

the results in dictionary form

Return type

dict

to_gradescope_dict(ag_config)#

Converts these results into a dictionary formatted for Gradescope’s autograder. Requires a dictionary of configurations for the Gradescope assignment generated using Otter Generate.

Parameters

ag_config (otter.run.run_autograder.autograder_config.AutograderConfig) – the autograder config

Returns

the results formatted for Gradescope

Return type

dict

to_report_str()#

Returns these results as a report string generated using the __repr__ of the TestFile class.

Returns

the report

Return type

str

property total#

the total points earned

Type

int or float

update_score(test_name, new_score)#

Updates the values in the GradingTestCaseResult object stored in self.results[test_name] with the key-value pairs in kwargs.

Parameters

  • test_name (str) – the name of the test

  • new_score (int or float) – the new score

verify_against_log(log, ignore_hidden=True) → List[str]#

Verifies these scores against the results stored in this log using the results returned by Log.get_results for comparison. A discrepancy occurs if the scores differ by more than the default tolerance of math.isclose. If ignore_hidden is True, hidden tests are ignored when verifying scores.

Parameters

  • log (otter.check.logs.Log) – the log to verify against

  • ignore_hidden (bool, optional) – whether to ignore hidden tests during verification

Returns

a list of error messages for discrepancies; if none were found, the list

is empty

Return type

list[str]

classmethod without_results(e)#

Creates an empty results object that represents an execution failure during autograding.

The returned results object will alert students and instructors to this failure, providing the error message and traceback to instructors, and report a score of 0 on Gradescope.

Parameters

e (Exception) – the error that was thrown

Returns

the results object

Return type

GradingResults