This section of the documentation describes the process by which submissions are executed in the autograder. Regardless of the method by which Otter is used, the autograding internals are all the same, although the execution differs slightly based on the file type of the submission.
All Python submissions, regardless of type, are converted to notebooks and graded with nbformat’s
ExecutePreprocessor. For Python scripts, they are converted by creating a notebook with a
single code cell that contains the script’s contents.
The notebooks are executed as follows:
before_executionplugin event is run.
Cells tagged with
truein the metadata are removed from the notebook.
(If grading from an Otter log) all lines of code which are not import statements are removed from the submission and replaced with code to unpack the serialized environments from the log.
Additional checks indicated in the cell metadata (
otter[tests]) are added those cells.
(If intercell seeding is being used) a cell importing
randomis added to the top of the notebook and each code cell is prepended with the seeding code.
The current working directory is added to
Cells to initialize Otter and export grading results are added at the beginning and end of the notebook, respectively.
The notebook is executed with the
ExecutePreprocessor. If running in debug mode, errors are not ignored.
The results exported by Otter from the notebook are loaded.
after_gradingplugin event is run.