Logging#

In order to assist with debugging students’ checks, Otter automatically logs events when called from otter.Notebook or otter check. The events are logged in the following methods of otter.Notebook:

__init__
_auth
check
check_all
export
submit
to_pdf

The events are stored as otter.logs.LogEntry objects which are pickled and appended to a file called .OTTER_LOG. To interact with a log, the otter.logs.Log class is provided; logs can be read in using the class method Log.from_file:

from otter.check.logs import Log
log = Log.from_file(".OTTER_LOG")

The log order defaults to chronological (the order in which they were appended to the file) but this can be reversed by setting the ascending argument to False. To get the most recent results for a specific question, use Log.get_results:

log.get_results("q1")

Note that the otter.logs.Log class does not support editing the log file, only reading and interacting with it.

Logging Environments#

Whenever a student runs a check cell, Otter can store their current global environment as a part of the log. The purpose of this is twofold: 1) to allow the grading of assignments to occur based on variables whose creation requires access to resources not possessed by the grading environment, and 2) to allow instructors to debug students’ assignments by inspecting their global environment at the time of the check. This behavior must be preconfigured with an Otter configuration file that has its save_environment key set to true.

Shelving is accomplished by using the dill library to pickle (almost) everything in the global environment, with the notable exception of modules (so libraries will need to be reimported in the instructor’s environment). The environment (a dictionary) is pickled and the resulting file is then stored as a byte string in one of the fields of the log entry.

Environments can be saved to a log entry by passing the environment (as a dictionary) to LogEntry.shelve. Any variables that can’t be shelved (or are ignored) are added to the unshelved attribute of the entry.

from otter.logs import LogEntry
entry = LogEntry()
entry.shelve(globals())

The shelve method also optionally takes a parameter variables that is a dictionary mapping variable names to fully-qualified type strings. If passed, only variables whose names are keys in this dictionary and whose types match their corresponding values will be stored in the environment. This helps from serializing unnecessary objects and prevents students from injecting malicious code into the autograder. To get the type string, use the function otter.utils.get_variable_type. As an example, the type string for a pandas DataFrame is "pandas.core.frame.DataFrame":

>>> import pandas as pd
>>> from otter.utils import get_variable_type
>>> df = pd.DataFrame()
>>> get_variable_type(df)
'pandas.core.frame.DataFrame'

With this, we can tell the log entry to only shelve dataframes named df:

from otter.logs import LogEntry
variables = {"df": "pandas.core.frame.DataFrame"}
entry = LogEntry()
entry.shelve(globals(), variables=variables)

If you are grading from the log and are utilizing variables, you must include this dictionary as a JSON string in your configuration, otherwise the autograder will deserialize anything that the student submits. This configuration is set in two places: in the Otter configuration file that you distribute with your notebook and in the autograder. Both of these are handled for you if you use Otter Assign to generate your distribution files.

To retrieve a shelved environment from an entry, use the LogEntry.unshelve method. During the process of unshelving, all functions have their __globals__ updated to include everything in the unshelved environment and, optionally, anything in the environment passed to global_env.

>>> env = entry.unshelve() # this will have everything in the shelf in it -- but not factorial
>>> from math import factorial
>>> env_with_factorial = entry.unshelve({"factorial": factorial}) # add factorial to all fn __globals__
>>> "factorial" in env_with_factorial["some_fn"].__globals__
True
>>> factorial is env_with_factorial["some_fn"].__globals__["factorial"]
True

See the reference below for more information about the arguments to LogEntry.shelve and LogEntry.unshelve.

Debugging with the Log#

The log is useful to help students debug tests that they are repeatedly failing. Log entries store any errors thrown by the process tracked by that entry and, if the log is a call to otter.Notebook.check, also the test results. Any errors held by the log entry can be re-thrown by calling LogEntry.raise_error:

from otter.logs import Log
log = Log.from_file(".OTTER_LOG")
entry = log.entries[0]
entry.raise_error()

The test results of an entry can be returned using LogEntry.get_results:

entry.get_results()

Grading from the Log#

As noted earlier, the environments stored in logs can be used to grade students’ assignments. If the grading environment does not have the dependencies necessary to run all code, the environment saved in the log entries will be used to run tests against. For example, if the execution hub has access to a large SQL server that cannot be accessed by a Gradescope grading container, these questions can still be graded using the log of checks run by the students and the environments pickled therein.

To configure these pregraded questions, include an Otter configuration file in the assignment directory that defines the notebook name and that the saving of environments should be turned on:

{
    "notebook": "hw00.ipynb",
    "save_environment": true
}

If you are restricting the variables serialized during checks, also set the variables or ignore_modules parameters. If you are grading on Gradescope, you must also tell the autograder to grade from the log using the --grade-from-log flag when running or the grade_from_log subkey of generate if using Otter Assign.

Otter Logs Reference#

`otter.logs.Log`#

class otter.logs.Log(entries, ascending=True)#

A class for reading and interacting with a log. Allows you to iterate over the entries in the log and supports integer indexing. Does not support editing the log file.

Parameters

entries (list of LogEntry) – the list of entries for this log
ascending (bool, optional) – whether the log is sorted in ascending (chronological) order; default True

entries#

the list of log entries in this log

Type: list of LogEntry

ascending#

whether entries is sorted chronologically; False indicates reverse- chronological order

Type: bool

classmethod from_file(filename, ascending=True)#

Loads and sorts a log from a file.

Parameters

filename (str) – the path to the log
ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order; default True

Returns

the Log instance created from the file

Return type

Log

get_question_entry(question)#

Gets the most recent entry corresponding to the question question

Parameters: question (str) – the question to get
Returns: the most recent log entry for question
Return type: LogEntry
Raises: QuestionNotInLogException – if the question is not in the log

get_questions()#

Returns a sorted list of all question names that have entries in this log.

Returns: the questions in this log
Return type: list of str

get_results(question)#

Gets the most recent grading result for a specified question from this log

Parameters: question (str) – the question name to look up
Returns: the most recent result for the question
Return type: otter.test_files.abstract_test.TestCollectionResults
Raises: QuestionNotInLogException – if the question is not found

question_iterator()#

Returns an iterator over the most recent entries for each question.

Returns: the iterator
Return type: QuestionLogIterator

sort(ascending=True)#

Sorts this logs entries by timestmap using LogEntry.sort_log.

Parameters: ascending (bool, optional) – whether to sort the log chronologically; defaults to True

`otter.logs.LogEntry`#

class otter.logs.LogEntry(event_type, shelf=None, unshelved=[], results=[], question=None, success=True, error=None)#

An entry in Otter’s log. Tracks event type, grading results, success of operation, and errors thrown.

Parameters

event_type (EventType) – the type of event for this entry
results (list of otter.test_files.abstract_test.TestCollectionResults, optional) – the results of grading if this is an EventType.CHECK record
question (str, optional) – the question name for an EventType.CHECK record
success (bool, optional) – whether the operation was successful
error (Exception, optional) – an error thrown by the process being logged if any

event_type#

the entry type

Type: EventType

shelf#

a pickled environment stored as a bytes string

Type: bytes

unshelved#

a list of variable names that were unable to be pickled during shelving

Type: list of str

results#

grading results if this is an EventType.CHECK entry

Type: list of otter.test_files.abstract_test.TestCollectionResults

question#

question name if this is a check entry

Type: str

success#

whether the operation tracked by this entry was successful

Type: bool

error#

an error thrown by the tracked process if applicable

Type: Exception

timestamp#

timestamp of event in UTC

Type: datetime.datetime

flush_to_file(filename)#

Appends this log entry (pickled) to a file

Parameters: filename (str) – the path to the file to append this entry

get_results()#

Get the results stored in this log entry

Returns

the results at this: entry if this is an EventType.CHECK record

Return type

list of otter.test_files.abstract_test.TestCollectionResults

get_score_perc()#

Returns the percentage score for the results of this entry

Returns: the percentage score
Return type: float

static log_from_file(filename, ascending=True)#

Reads a log file and returns a sorted list of the log entries pickled in that file

Parameters

filename (str) – the path to the log
ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order; default True

Returns

the sorted log

Return type

list of LogEntry

raise_error()#

Raises the error stored in this entry

Raises: Exception – the error stored at this entry, if present

shelve(env, delete=False, filename=None, ignore_modules=[], variables=None)#

Stores an environment env in this log entry using dill as a bytes object in this entry as the shelf attribute. Writes names of any variables in env that are not stored to the unshelved attribute.

If delete is True, old environments in the log at filename for this question are cleared before writing env. Any module names in ignore_modules will have their functions ignored during pickling.

Parameters

env (dict) – the environment to pickle
delete (bool, optional) – whether to delete old environments
filename (str, optional) – path to log file; ignored if delete is False
ignore_modules (list of str, optional) – module names to ignore during pickling
variables (dict, optional) – map of variable name to type string indicating only variables to include (all variables not in this dictionary will be ignored)

Returns

this entry

Return type

LogEntry

static shelve_environment(env, variables=None, ignore_modules=[])#

Pickles an environment env using dill, ignoring any functions whose module is listed in ignore_modules. Returns the pickle file contents as a bytes object and a list of variable names that were unable to be shelved/ignored during shelving.

Parameters

env (dict) – the environment to shelve
variables (dict or list, optional) – a map of variable name to type string indicating only variables to include (all variables not in this dictionary will be ignored) or a list of variable names to include regardless of tpye
ignore_modules (list of str, optional) – the module names to igonre

Returns

the pickled environment and list of unshelved: variable names.

Return type

tuple of (bytes, list of str)

static sort_log(log, ascending=True)#

Sorts a list of log entries by timestamp

Parameters

log (list of LogEntry) – the log to sort
ascending (bool, optional) – whether the log should be sorted in ascending (chronological) order; default True

Returns

the sorted log

Return type

list of LogEntry

unshelve(global_env={})#

Parses a bytes object stored in the shelf attribute and unpickles the object stored there using dill. Updates the __globals__ of any functions in shelf to include elements in the shelf. Optionally includes the env passed in as global_env.

Parameters: global_env (dict, optional) – a global env to include in unpickled function globals
Returns: the shelved environment
Return type: dict

`otter.logs.EventType`#

class otter.logs.EventType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)#

Enum of event types for log entries

AUTH#: an auth event

BEGIN_CHECK_ALL#: beginning of a check-all call

BEGIN_EXPORT#: beginning of an assignment export

CHECK#: a check of a single question

END_CHECK_ALL#: ending of a check-all call

END_EXPORT#: ending of an assignment export

INIT#: initialization of an otter.Notebook object

SUBMIT#: submission of an assignment

TO_PDF#: PDF export of a notebook

Otter-Grader documentation

Logging

Contents

Logging#

Logging Environments#

Debugging with the Log#

Grading from the Log#

Otter Logs Reference#

`otter.logs.Log`#

`otter.logs.LogEntry`#

`otter.logs.EventType`#

Otter-Grader documentation

Logging

Contents

Logging#

Logging Environments#

Debugging with the Log#

Grading from the Log#

Otter Logs Reference#

otter.logs.Log#

otter.logs.LogEntry#

otter.logs.EventType#

`otter.logs.Log`#

`otter.logs.LogEntry`#

`otter.logs.EventType`#