Grading on Gradescope¶
This section details how the autograder runs and how results are displayed to students and instructors on Gradescope. When a student submits to Gradescpe, the autograder does the following:
Copies the tests and support files from the autograder source
Globs the first IPYNB file and assumes this to be the submission to be graded
Reads in the log from the submission if it exists
Grades the notebook, globbing all tests and grading from the log if specified
Looks for discrepancies between the logged scores and the autograder scores and warngs about these if present
If indicated, exports the notebook as a PDF and POSTs this notebook to the other Gradescope assignment
Generates the JSON object for Gradescope’s results
Makes adjustments to the scores and visibility based on the configurations
Writes the JSON to the results file
Prints the results as a dataframe to stdout
Writing Tests for Gradescope¶
The tests that are used by the Gradescope autograder are the same as those used in other uses of Otter, but there is one important field that is relevant to Gradescope that is not pertinent to any other uses.
As noted in the second bullet here, the
key of each test case indicates the visibility of that specific test case.
Once a student’s submission has been autograded, the Autograder Results page will show the stdout of the grading process in the “Autograder Output” box and the student’s score in the side bar to the right of the output. The stdout includes information from verifying the student’s scores against the logged scores, a dataframe that contains the student’s score breakdown by question, and a summary of the information about test output visibility:
show_stdout was true in your
otter_config.json, then the autograder
output will be shown to students after grades are published on Gradescope. Students
will not be able to see the results of hidden tests nor the tests themselves, but they will see
that they failed some hidden test in the printed DataFrame from the stdout.
Below the autograder output, each test case is broken down into boxes. The first box is called “Public Tests” and summarizes the results of the public test cases only. This box is visible to students. If a student fails a hidden test but no public tests, then this box will show “All tests passed” for that question, even though the student failed a public test.
If the student fails a public test, then the output of the failed public test will be displayed here,
but no information about hidden tests will be included. In the example below, the student failed a
hidden test case in
q1 and a public test case in
Below the “Public Tests” box will be boxes for each question. These are hidden from the student
show_hidden was true in your
otter_config.json; if that is the case, then they will
become visible to students after grades are published. These boxes show the results for each question,
including the results of hidden test cases.
If all tests cases are passed, these boxes will indicate so:
If the studen fails any test cases, that information will be presented here. In the example below,
q1 - 4 and
q1 - 5 were hidden test cases; this screenshot corresponds to the second “Public
Tests” section screenshot above.