Logging#

In order to assist with debugging students’ checks, Otter automatically logs events when called from otter.Notebook or otter check. The events are logged in the following methods of otter.Notebook:

  • __init__

  • _auth

  • check

  • check_all

  • export

  • submit

  • to_pdf

The events are stored as otter.logs.LogEntry objects which are pickled and appended to a file called .OTTER_LOG. To interact with a log, the otter.logs.Log class is provided; logs can be read in using the class method Log.from_file:

from otter.check.logs import Log
log = Log.from_file(".OTTER_LOG")

The log order defaults to chronological (the order in which they were appended to the file) but this can be reversed by setting the ascending argument to False. To get the most recent results for a specific question, use Log.get_results:

log.get_results("q1")

Note that the otter.logs.Log class does not support editing the log file, only reading and interacting with it.

Logging Environments#

Whenever a student runs a check cell, Otter can store their current global environment as a part of the log. The purpose of this is twofold: 1) to allow the grading of assignments to occur based on variables whose creation requires access to resources not possessed by the grading environment, and 2) to allow instructors to debug students’ assignments by inspecting their global environment at the time of the check. This behavior must be preconfigured with an Otter configuration file that has its save_environment key set to true.

Shelving is accomplished by using the dill library to pickle (almost) everything in the global environment, with the notable exception of modules (so libraries will need to be reimported in the instructor’s environment). The environment (a dictionary) is pickled and the resulting file is then stored as a byte string in one of the fields of the log entry.

Environments can be saved to a log entry by passing the environment (as a dictionary) to LogEntry.shelve. Any variables that can’t be shelved (or are ignored) are added to the unshelved attribute of the entry.

from otter.logs import LogEntry
entry = LogEntry()
entry.shelve(globals())

The shelve method also optionally takes a parameter variables that is a dictionary mapping variable names to fully-qualified type strings. If passed, only variables whose names are keys in this dictionary and whose types match their corresponding values will be stored in the environment. This helps from serializing unnecessary objects and prevents students from injecting malicious code into the autograder. To get the type string, use the function otter.utils.get_variable_type. As an example, the type string for a pandas DataFrame is "pandas.core.frame.DataFrame":

>>> import pandas as pd
>>> from otter.utils import get_variable_type
>>> df = pd.DataFrame()
>>> get_variable_type(df)
'pandas.core.frame.DataFrame'

With this, we can tell the log entry to only shelve dataframes named df:

from otter.logs import LogEntry
variables = {"df": "pandas.core.frame.DataFrame"}
entry = LogEntry()
entry.shelve(globals(), variables=variables)

If you are grading from the log and are utilizing variables, you must include this dictionary as a JSON string in your configuration, otherwise the autograder will deserialize anything that the student submits. This configuration is set in two places: in the Otter configuration file that you distribute with your notebook and in the autograder. Both of these are handled for you if you use Otter Assign to generate your distribution files.

To retrieve a shelved environment from an entry, use the LogEntry.unshelve method. During the process of unshelving, all functions have their __globals__ updated to include everything in the unshelved environment and, optionally, anything in the environment passed to global_env.

>>> env = entry.unshelve() # this will have everything in the shelf in it -- but not factorial
>>> from math import factorial
>>> env_with_factorial = entry.unshelve({"factorial": factorial}) # add factorial to all fn __globals__
>>> "factorial" in env_with_factorial["some_fn"].__globals__
True
>>> factorial is env_with_factorial["some_fn"].__globals__["factorial"]
True

See the reference below for more information about the arguments to LogEntry.shelve and LogEntry.unshelve.

Debugging with the Log#

The log is useful to help students debug tests that they are repeatedly failing. Log entries store any errors thrown by the process tracked by that entry and, if the log is a call to otter.Notebook.check, also the test results. Any errors held by the log entry can be re-thrown by calling LogEntry.raise_error:

from otter.logs import Log
log = Log.from_file(".OTTER_LOG")
entry = log.entries[0]
entry.raise_error()

The test results of an entry can be returned using LogEntry.get_results:

entry.get_results()

Grading from the Log#

As noted earlier, the environments stored in logs can be used to grade students’ assignments. If the grading environment does not have the dependencies necessary to run all code, the environment saved in the log entries will be used to run tests against. For example, if the execution hub has access to a large SQL server that cannot be accessed by a Gradescope grading container, these questions can still be graded using the log of checks run by the students and the environments pickled therein.

To configure these pregraded questions, include an Otter configuration file in the assignment directory that defines the notebook name and that the saving of environments should be turned on:

{
    "notebook": "hw00.ipynb",
    "save_environment": true
}

If you are restricting the variables serialized during checks, also set the variables or ignore_modules parameters. If you are grading on Gradescope, you must also tell the autograder to grade from the log using the --grade-from-log flag when running or the grade_from_log subkey of generate if using Otter Assign.

Otter Logs Reference#

otter.logs.Log#

class otter.logs.Log(entries, ascending=True)#

A class for reading and interacting with a log. Allows you to iterate over the entries in the log and supports integer indexing. Does not support editing the log file.

Parameters:
  • entries (list of LogEntry) – the list of entries for this log

  • ascending (bool, optional) – whether the log is sorted in ascending (chronological) order; default True

entries#

the list of log entries in this log

Type:

list of LogEntry

ascending#

whether entries is sorted chronologically; False indicates reverse- chronological order

Type:

bool

classmethod from_file(filename, ascending=True)#

Loads and sorts a log from a file.

Parameters:
  • filename (str) – the path to the log

  • ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order; default True

Returns:

the Log instance created from the file

Return type:

Log

get_question_entry(question)#

Gets the most recent entry corresponding to the question question

Parameters:

question (str) – the question to get

Returns:

the most recent log entry for question

Return type:

LogEntry

Raises:

QuestionNotInLogException – if the question is not in the log

get_questions()#

Returns a sorted list of all question names that have entries in this log.

Returns:

the questions in this log

Return type:

list of str

get_results(question)#

Gets the most recent grading result for a specified question from this log

Parameters:

question (str) – the question name to look up

Returns:

the most recent result for the question

Return type:

otter.test_files.abstract_test.TestCollectionResults

Raises:

QuestionNotInLogException – if the question is not found

question_iterator()#

Returns an iterator over the most recent entries for each question.

Returns:

the iterator

Return type:

QuestionLogIterator

sort(ascending=True)#

Sorts this logs entries by timestmap using LogEntry.sort_log.

Parameters:

ascending (bool, optional) – whether to sort the log chronologically; defaults to True

otter.logs.LogEntry#

class otter.logs.LogEntry(event_type: EventType, shelf: bytes | None = None, results: GradingResults | None = None, question: str | None = None, success: bool = True, error: Exception | None = None)#

An entry in Otter’s log. Tracks event type, grading results, success of operation, and errors thrown.

Parameters:
  • event_type (EventType) – the type of event for this entry

  • results (otter.test_files.TestFile | otter.test_files.GradingResults | None) – the results of grading if this is an EventType.CHECK or EventType.END_CHECK_ALL record

  • question (str) – the question name for an EventType.CHECK record

  • success (bool) – whether the operation was successful

  • error (Exception) – an error thrown by the process being logged if any

error: Exception | None#

an error thrown by the tracked process if applicable

event_type: EventType#

the entry type

flush_to_file(filename)#

Appends this log entry (pickled) to a file

Parameters:

filename (str) – the path to the file to append this entry

get_results()#

Get the results stored in this log entry

Returns:

the results at this

entry if this is an EventType.CHECK record

Return type:

list of otter.test_files.abstract_test.TestCollectionResults

get_score_perc()#

Returns the percentage score for the results of this entry

Returns:

the percentage score

Return type:

float

static log_from_file(filename, ascending=True)#

Reads a log file and returns a sorted list of the log entries pickled in that file

Parameters:
  • filename (str) – the path to the log

  • ascending (bool) – whether the log should be sorted in ascending (chronological) order

Returns:

the sorted log

Return type:

list[LogEntry]

not_shelved: List[str]#

a list of variable names that were not added to the shelf

question: str | None#

question name if this is a check entry

raise_error()#

Raises the error stored in this entry

Raises:

Exception – the error stored at this entry, if present

results: GradingResults | None#

grading results if this is an EventType.CHECK entry

shelf: bytes | None#

a pickled environment stored as a bytes string

shelve(env, delete=False, filename=None, ignore_modules=[], variables=None)#

Stores an environment env in this log entry using dill as a bytes object in this entry as the shelf attribute. Writes names of any variables in env that are not stored to the not_shelved attribute.

If delete is True, old environments in the log at filename for this question are cleared before writing env. Any module names in ignore_modules will have their functions ignored during pickling.

Parameters:
  • env (dict) – the environment to pickle

  • delete (bool, optional) – whether to delete old environments

  • filename (str, optional) – path to log file; ignored if delete is False

  • ignore_modules (list of str, optional) – module names to ignore during pickling

  • variables (dict, optional) – map of variable name to type string indicating only variables to include (all variables not in this dictionary will be ignored)

Returns:

this entry

Return type:

LogEntry

static shelve_environment(env, variables=None, ignore_modules=[])#

Pickles an environment env using dill, ignoring any functions whose module is listed in ignore_modules. Returns the pickle file contents as a bytes object and a list of variable names that were unable to be shelved/ignored during shelving.

Parameters:
  • env (dict) – the environment to shelve

  • variables (dict or list, optional) – a map of variable name to type string indicating only variables to include (all variables not in this dictionary will be ignored) or a list of variable names to include regardless of tpye

  • ignore_modules (list of str, optional) – the module names to igonre

Returns:

the pickled environment and list of variable names that were

not shelved

Return type:

tuple[bytes, list[str]

static sort_log(log, ascending=True)#

Sorts a list of log entries by timestamp

Parameters:
  • log (list of LogEntry) – the log to sort

  • ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order

Returns:

the sorted log

Return type:

list[LogEntry]

success: bool#

whether the operation tracked by this entry was successful

timestamp: datetime#

timestamp of event in UTC

unshelve(global_env={})#

Parses a bytes object stored in the shelf attribute and unpickles the object stored there using dill. Updates the __globals__ of any functions in shelf to include elements in the shelf. Optionally includes the env passed in as global_env.

Parameters:

global_env (dict, optional) – a global env to include in unpickled function globals

Returns:

the shelved environment

Return type:

dict

otter.logs.EventType#

class otter.logs.EventType(value, names=None, *values, module=None, qualname=None, type=None, start=1, boundary=None)#

Enum of event types for log entries

AUTH = 1#

an auth event

BEGIN_CHECK_ALL = 2#

beginning of a check-all cell

BEGIN_EXPORT = 3#

beginning of an assignment export

CHECK = 4#

a check of a single question

END_CHECK_ALL = 5#

ending of a check-all cell

END_EXPORT = 6#

ending of an assignment export

INIT = 7#

initialization of an otter.check.notebook.Notebook object

SUBMIT = 8#

submission of an assignment (unused since Otter Service was removed)

TO_PDF = 9#

PDF export of a notebook (not used during a submission export)