Logging

In order to assist with debugging students’ checks, Otter automatically logs events when called from otter.Notebook or otter check. The events are logged in the following methods of otter.Notebook:

  • __init__

  • _auth

  • check

  • check_all

  • export

  • submit

  • to_pdf

The events are stored as otter.logs.LogEntry objects which are pickled and appended to a file called .OTTER_LOG. To interact with a log, the otter.logs.Log class is provided; logs can be read in using the class method Log.from_file:

from otter.logs import Log
log = Log.from_file(".OTTER_LOG")

The log order defaults to chronological (the order in which they were appended to the file) but this can be reversed by setting the ascending argument to False. To get the most recent results for a specific question, use Log.get_results:

log.get_results("q1")

Note that the otter.logs.Log class does not support editing the log file, only reading and interacting with it.

Logging Environments

Whenever a student runs a check cell, Otter can store their current global environment as a part of the log. The purpose of this is twofold: 1) to allow the grading of assignments to occur based on variables whose creation requires access to resources not possessed by the grading environment, and 2) to allow instructors to debug students’ assignments by inspecting their global environment at the time of the check. This behavior must be preconfigured with an Otter configuration file that has its save_environment key set to true.

Shelving is accomplished by using the dill library to pickle (almost) everything in the global environment, with the notable exception of modules (so libraries will need to be reimported in the instructor’s environment). The environment (a dictionary) is pickled and the resulting file is then stored as a byte string in one of the fields of the log entry.

Environments can be saved to a log entry by passing the environment (as a dictionary) to LogEntry.shelve. Any variables that can’t be shelved (or are ignored) are added to the unshelved attribute of the entry.

from otter.logs import LogEntry
entry = LogEntry()
entry.shelve(globals())

The shelve method also optionally takes a parameter variables that is a dictionary mapping variable names to fully-qualified type strings. If passed, only variables whose names are keys in this dictionary and whose types match their corresponding values will be stored in the environment. This helps from serializing unnecessary objects and prevents students from injecting malicious code into the autograder. To get the type string, use the function otter.utils.get_variable_type. As an example, the type string for a pandas DataFrame is "pandas.core.frame.DataFrame":

>>> import pandas as pd
>>> from otter.utils import get_variable_type
>>> df = pd.DataFrame()
>>> get_variable_type(df)
'pandas.core.frame.DataFrame'

With this, we can tell the log entry to only shelve dataframes named df:

from otter.logs import LogEntry
variables = {"df": "pandas.core.frame.DataFrame"}
entry = LogEntry()
entry.shelve(globals(), variables=variables)

If you are grading from the log and are utilizing variables, you must include this dictionary as a JSON string in your configuration, otherwise the autograder will deserialize anything that the student submits. This configuration is set in two places: in the Otter configuration file that you distribute with your notebook and in the autograder. Both of these are handled for you if you use Otter Assign to generate your distribution files.

To retrieve a shelved environment from an entry, use the LogEntry.unshelve method. During the process of unshelving, all functions have their __globals__ updated to include everything in the unshelved environment and, optionally, anything in the environment passed to global_env.

>>> env = entry.unshelve() # this will have everything in the shelf in it -- but not factorial
>>> from math import factorial
>>> env_with_factorial = entry.unshelve({"factorial": factorial}) # add factorial to all fn __globals__
>>> "factorial" in env_with_factorial["some_fn"].__globals__
True
>>> factorial is env_with_factorial["some_fn"].__globals__["factorial"]
True

See the reference below for more information about the arguments to LogEntry.shelve and LogEntry.unshelve.

Debugging with the Log

The log is useful to help students debug tests that they are repeatedly failing. Log entries store any errors thrown by the process tracked by that entry and, if the log is a call to otter.Notebook.check, also the test results. Any errors held by the log entry can be re-thrown by calling LogEntry.raise_error:

from otter.logs import Log
log = Log.from_file(".OTTER_LOG")
entry = log.entries[0]
entry.raise_error()

The test results of an entry can be returned using LogEntry.get_results:

entry.get_results()

Grading from the Log

As noted earlier, the environments stored in logs can be used to grade students’ assignments. If the grading environment does not have the dependencies necessary to run all code, the environment saved in the log entries will be used to run tests against. For example, if the execution hub has access to a large SQL server that cannot be accessed by a Gradescope grading container, these questions can still be graded using the log of checks run by the students and the environments pickled therein.

To configure these pregraded questions, include an Otter configuration file in the assignment directory that defines the notebook name and that the saving of environments should be turned on:

{
    "notebook": "hw00.ipynb",
    "save_environment": true
}

If you are restricting the variables serialized during checks, also set the variables or ignore_modules parameters. If you are grading on Gradescope, you must also tell the autograder to grade from the log using the --grade-from-log flag when running or the grade_from_log subkey of generate if using Otter Assign.

Otter Logs Reference

otter.logs.Log

class otter.logs.Log(entries, ascending=True)

A class for reading and interacting with a log. Allows you to iterate over the entries in the log and supports integer indexing. Does not support editing the log file.

Parameters
  • entries (list of LogEntry) – the list of entries for this log

  • ascending (bool, optional) – whether the log is sorted in ascending (chronological) order; default True

entries

the list of log entries in this log

Type

list of LogEntry

ascending

chronological order

Type

bool

classmethod from_file(filename, ascending=True)

Loads and sorts a log from a file.

Parameters
  • filename (str) – the path to the log

  • ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order; default True

Returns

Log: the Log instance created from the file

get_question_entry(question)

Gets the most recent entry corresponding to the question question

Parameters

question (str) – the question to get

Returns

LogEntry: the most recent log entry for question

Raises

QuestionNotInLogException – if the question is not in the log

get_questions()

Returns a sorted list of all question names that have entries in this log.

Returns

the questions in this log

Return type

list of str

get_results(question)

Gets the most recent grading result for a specified question from this log

Parameters

question (str) – the question name to look up

Returns

the most recent result for the question

Return type

otter.test_files.abstract_test.TestCollectionResults

Raises

QuestionNotInLogException – if the question is not found

question_iterator()

Returns an iterator over the most recent entries for each question.

Returns

the iterator

Return type

QuestionLogIterator

sort(ascending=True)

Sorts this logs entries by timestmap using LogEntry.sort_log.

Parameters

ascending (bool, optional) –

otter.logs.LogEntry

class otter.logs.LogEntry(event_type, shelf=None, unshelved=[], results=[], question=None, success=True, error=None)

An entry in Otter’s log. Tracks event type, grading results, success of operation, and errors thrown.

Parameters
  • event_type (EventType) – the type of event for this entry

  • results (list of otter.test_files.abstract_test.TestCollectionResults, optional) – the results of grading if this is an EventType.CHECK record

  • question (str, optional) –

  • success (bool, optional) – whether the operation was successful

  • error (Exception, optional) – an error thrown by the process being logged if any

event_type

the entry type

Type

EventType

shelf

a pickled environment stored as a bytes string

Type

bytes

unshelved

a list of variable names that were unable to be pickled during shelving

Type

list of str

results

grading results if this is an EventType.CHECK entry

Type

list of otter.test_files.abstract_test.TestCollectionResults

question

question name if this is a check entry

Type

str

success

whether the operation tracked by this entry was successful

Type

bool

error

an error thrown by the tracked process if applicable

Type

Exception

timestamp

timestamp of event in UTC

Type

datetime.datetime

flush_to_file(filename)

Appends this log entry (pickled) to a file

Parameters

filename (str) – the path to the file to append this entry

get_results()

Get the results stored in this log entry

Returns

the results at this

entry if this is an EventType.CHECK record

Return type

list of otter.test_files.abstract_test.TestCollectionResults

get_score_perc()

Returns the percentage score for the results of this entry

Returns

the percentage score

Return type

float

static log_from_file(filename, ascending=True)

Reads a log file and returns a sorted list of the log entries pickled in that file

Parameters
  • filename (str) – the path to the log

  • ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order; default True

Returns

the sorted log

Return type

list of LogEntry

raise_error()

Raises the error stored in this entry

Raises

Exception – the error stored at this entry, if present

shelve(env, delete=False, filename=None, ignore_modules=[], variables=None)

Stores an environment env in this log entry using dill as a bytes object in this entry as the shelf attribute. Writes names of any variables in env that are not stored to the unshelved attribute.

If delete is True, old environments in the log at filename for this question are cleared before writing env. Any module names in ignore_modules will have their functions ignored during pickling.

Parameters
  • env (dict) – the environment to pickle

  • delete (bool, optional) – whether to delete old environments

  • filename (str, optional) –

  • ignore_modules (list of str, optional) – module names to ignore during pickling

  • variables (dict, optional) – map of variable name to type string indicating only variables to include (all variables not in this dictionary will be ignored)

Returns

this entry

Return type

LogEntry

static shelve_environment(env, variables=None, ignore_modules=[])

Pickles an environment env using dill, ignoring any functions whose module is listed in ignore_modules. Returns the pickle file contents as a bytes object and a list of variable names that were unable to be shelved/ignored during shelving.

Parameters
  • env (dict) – the environment to shelve

  • variables (dict or list, optional) – a map of variable name to type string indicating only variables to include (all variables not in this dictionary will be ignored) or a list of variable names to include regardless of tpye

  • ignore_modules (list of str, optional) – the module names to igonre

Returns

the pickled environment and list of unshelved

variable names.

Return type

tuple of (bytes, list of str)

static sort_log(log, ascending=True)

Sorts a list of log entries by timestamp

Parameters
  • log (list of LogEntry) – the log to sort

  • ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order; default True

Returns

the sorted log

Return type

list of LogEntry

unshelve(global_env={})

Parses a bytes object stored in the shelf attribute and unpickles the object stored there using dill. Updates the __globals__ of any functions in shelf to include elements in the shelf. Optionally includes the env passed in as global_env.

Parameters

global_env (dict, optional) – a global env to include in unpickled function globals

Returns

the shelved environment

Return type

dict

otter.logs.EventType

class otter.logs.EventType(value)

Enum of event types for log entries

AUTH

an auth event

BEGIN_CHECK_ALL

beginning of a check-all call

BEGIN_EXPORT

beginning of an assignment export

CHECK

a check of a single question

END_CHECK_ALL

ending of a check-all call

END_EXPORT

ending of an assignment export

INIT

initialization of an otter.Notebook object

SUBMIT

submission of an assignment

TO_PDF

PDF export of a notebook