Built-In Plugins¶
Otter comes with a few built-in plugins to optionally extend its own functionality. These plugins are documented here. Note: To use these plugins, specify the importable names exactly as seen here (i.e. otter.plugins.builtin.GoogleSheetsGradeOverride
, not otter.plugins.builtin.grade_override.GoogleSheetsGradeOverride
).
Google Sheets Grade Override¶
This plugin allows you to override test case scores during grading by specifying the new score in a Google Sheet that is pulled in. To use this plugin, you must set up a Google Cloud project and obtain credentials for the Google Sheets API. This plugin relies on gspread
to interact with the Google Sheets API and its documentation contains instructions on how to obtain credentials. Once you have created credentials, download them as a JSON file.
This plugin requires two configurations: the path to the credentials JSON file and the URL of the Google Sheet. The former should be entered with the key credentials_json_path
and the latter sheet_url
; for example, in Otter Assign:
plugins:
- otter.plugins.builtin.GoogleSheetsGradeOverride:
credentials_json_path: /path/to/google/credentials.json
sheet_url: https://docs.google.com/some/google/sheet
The first tab in the sheet is assumed to be the override information.
During Otter Generate, the plugin will read in this JSON file and store the relevant data in the otter_config.json
for use during grading. The Google Sheet should have the following format:
Assignment ID | Test Case | Points | ||
---|---|---|---|---|
123456 | student@univ.edu | q1a - 1 | 1 | false |
Assignment ID
should be the ID of the assignment on Gradescope (to allow one sheet to be used for multiple assignments)Email
should be the student’s email address on GradescopeTest Case
should be the name of the test case as a string (e.g. if you have a test fileq1a.py
with 3 test cases, overriding the second case would beq1a - 2
)Points
should be the point value to assign the student for that test casePDF
should be whether or not a PDF should be re-submitted (to prevent losing manual grades on Gradescope for regrade requests)
Note that the use of this plugin requires specifying gpsread
in your requirements, as it is not included by default in Otter’s container image.
The actions taken by at hook for this plugin are detailed below.
otter.plugins.builtin.GoogleSheetsGradeOverride
Reference¶
-
class
otter.plugins.builtin.
GoogleSheetsGradeOverride
(submission_path, submission_metadata, plugin_config)¶ Otter plugin for overriding test case scores with values in a Google Sheet on Gradescope. Uses provided Google Service Account credentials to pull in the spreadsheet as a dataframe and edits test case scores by matching on the Gradescope assignment ID, student email, and test case name.
Implements the
during_generate
,before_grading
, andafter_grading
events. Make sure to list this plugin as ``otter.plugins.builtin.GoogleSheetsGradeOverride``, otherwise the ``during_generate`` event of this plugin will not work.The google sheet should have the following format:
Assignment ID
Email
Test Case
Points
PDF
123456
q1a - 1
1
false
Assignment ID
should be the ID of the assignment on Gradescope,Email
should be the email address corresponding to the student’s Gradescope account,Test Case
should be the name of the test case, andPoints
should be the number of points that the student should be assigned.PDF
should befalse
if the student’s PDF should not be regenerated during this run of the autograder.-
after_grading
(results)¶ Modifies the results of grading by pulling in the Google Sheet as a dataframe and updating the scores for the test cases found.
- Parameters
results (
otter.test_files.GradingResults
) – the results of grading
-
before_grading
(options)¶ Controls whether or not a PDF is generated by setting the
token
key of the grading options toNone
if any of the rows in the dataframe matching this (email, assignment ID) tuple indicate that a PDF should not be generated.- Parameters
options (
dict
) – the grading configurations
-
property
df
¶ The grade override information dataframe
-
during_generate
(otter_config)¶ Takes a path to Google Service Account credentials stored in this plugin’s config as key
credentials_json_path
and extracts the data from that file into the plugin’s config as keyservice_account_credentials
.- Parameters
otter_config (
dict
) – the parsed Otter configuration JSON file
-
Rate Limiting¶
This plugin allows instructors to limit the number of times students can submit to the autograder in any given window to prevent students from using the AG as an “oracle”. This plugin has a required configuration allowed_submissions
, the number of submissions allowed during the window, and accepts optional configurations weeks
, days
, hours
, minutes
, seconds
, milliseconds
, and microseconds
(all defaulting to 0
) to configure the size of the window. For example, to specify 5 allowed submissions per-day in your otter_config.json
:
{
"plugins": [
{
"otter.plugins.builtin.RateLimiting": {
"allowed_submissions": 5,
"days": 1
}
}
]
}
When a student submits, the window is caculated as a datetime.timedelta
object and if the student has at least allowed_submissions
in that window, the submission’s results are hidden and the student is only shown a message; from the example above:
You have exceeded the rate limit for the autograder. Students are allowed 5 submissions every 1 days.
The results of submission execution are still visible to instructors.
If the student’s submission is allowed based on the rate limit, the plugin outputs a message in the plugin report; from the example above:
Students are allowed 5 submissions every 1 days. You have {number of previous submissions} submissions in that period.
The actions taken by at hook for this plugin are detailed below.
otter.plugins.builtin.RateLimiting
Reference¶
-
class
otter.plugins.builtin.
RateLimiting
(submission_path, submission_metadata, plugin_config)¶ Implements submission rate limiting by restricting the number of submissions a student is allowed during a specified window
This plugin is configured by defining the duration of the window using the optional keys
weeks
,days
,hours
,minutes
,seconds
,milliseconds
, andmicroseconds
(all of which default to 0). The number of submissions allowed during this window is configured withallowed_submissions
. For example, to allow only 2 submissions every hour-and-a-half:{ "plugins": [ { "otter.plugins.builtin.RateLimiting": { "hours": 1, "minutes": 30, "allowed_submissions": 2 } } ] }
When a student submits, the window is caculated as a datetime.timedelta object and if the student has at least allowed_submissions in that window, the submission’s results are hidden and the student is only shown a message; from the example above:
You have exceeded the rate limit for the autograder. Students are allowed 2 submissions every 1 hours 30 minutes.
The results of submission execution are still visible to instructors.
If the student’s submission is allowed based on the rate limit, the plugin outputs a message in the plugin report; from the example above:
Students are allowed 2 submissions every 1 hours 30 minutes. You have {number of previous submissions} submissions in that period.
-
after_grading
(results)¶ Determines whether a student has exceeded the rate limit; if yes, all results are hidden from the student using
otter.test_files.GradingResults.hide_everything
and the output of the grader is set to the message.- Parameters
results (
otter.test_files.GradingResults.hide_everything
) – the results of grading
-
generate_report
()¶ Returns the message regarding whether the rate limit was exceeded for inclusion in the plugin report.
- Returns
the message
- Return type
str
-