Rate Limiting

This plugin allows instructors to limit the number of times students can submit to the autograder in any given window to prevent students from using the AG as an “oracle”. This plugin has a required configuration allowed_submissions, the number of submissions allowed during the window, and accepts optional configurations weeks, days, hours, minutes, seconds, milliseconds, and microseconds (all defaulting to 0) to configure the size of the window. For example, to specify 5 allowed submissions per-day in your otter_config.json:

{
    "plugins": [
        {
            "otter.plugins.builtin.RateLimiting": {
                "allowed_submissions": 5,
                "days": 1
            }
        }
    ]
}

When a student submits, the window is caculated as a datetime.timedelta object and if the student has at least allowed_submissions in that window, the submission’s results are hidden and the student is only shown a message; from the example above:

You have exceeded the rate limit for the autograder. Students are allowed 5 submissions every 1 days.

The results of submission execution are still visible to instructors.

If the student’s submission is allowed based on the rate limit, the plugin outputs a message in the plugin report; from the example above:

Students are allowed 5 submissions every 1 days. You have {number of previous submissions} submissions in that period.

The actions taken by at hook for this plugin are detailed below.

otter.plugins.builtin.RateLimiting Reference

class otter.plugins.builtin.RateLimiting(submission_path, submission_metadata, plugin_config)

Implements submission rate limiting by restricting the number of submissions a student is allowed during a specified window

This plugin is configured by defining the duration of the window using the optional keys weeks, days, hours, minutes, seconds, milliseconds, and microseconds (all of which default to 0). The number of submissions allowed during this window is configured with allowed_submissions. For example, to allow only 2 submissions every hour-and-a-half:

{
    "plugins": [
        {
            "otter.plugins.builtin.RateLimiting": {
                "hours": 1,
                "minutes": 30,
                "allowed_submissions": 2
            }
        }
    ]
}

When a student submits, the window is caculated as a datetime.timedelta object and if the student has at least allowed_submissions in that window, the submission’s results are hidden and the student is only shown a message; from the example above:

You have exceeded the rate limit for the autograder. Students are allowed 2 submissions every 1 hours 30 minutes.

The results of submission execution are still visible to instructors.

If the student’s submission is allowed based on the rate limit, the plugin outputs a message in the plugin report; from the example above:

Students are allowed 2 submissions every 1 hours 30 minutes. You have {number of previous submissions} submissions in that period.
after_grading(results)

Determines whether a student has exceeded the rate limit; if yes, all results are hidden from the student using otter.test_files.GradingResults.hide_everything and the output of the grader is set to the message.

Parameters

results (otter.test_files.GradingResults.hide_everything) – the results of grading

generate_report()

Returns the message regarding whether the rate limit was exceeded for inclusion in the plugin report.

Returns

the message

Return type

str