Grading on Gradescope

This section details how the autograder runs and how results are displayed to students and instructors on Gradescope. When a student submits to Gradescpe, the autograder does the following:

  1. Copies the tests and support files from the autograder source

  2. Globs the first IPYNB file and assumes this to be the submission to be graded

  3. Reads in the log from the submission if it exists

  4. Grades the notebook, globbing all tests and grading from the log if specified

  5. Looks for discrepancies between the logged scores and the autograder scores and warngs about these if present

  6. If indicated, exports the notebook as a PDF and POSTs this notebook to the other Gradescope assignment

  7. Generates the JSON object for Gradescope’s results

  8. Makes adjustments to the scores and visibility based on the configurations

  9. Writes the JSON to the results file

  10. Prints the results as a dataframe to stdout

Writing Tests for Gradescope

The tests that are used by the Gradescope autograder are the same as those used in other uses of Otter, but there is one important field that is relevant to Gradescope that is not pertinent to any other uses.

As noted in the second bullet here, the "hidden" key of each test case indicates the visibility of that specific test case.

Results

Once a student’s submission has been autograded, the Autograder Results page will show the stdout of the grading process in the “Autograder Output” box and the student’s score in the side bar to the right of the output. The stdout includes information from verifying the student’s scores against the logged scores, a dataframe that contains the student’s score breakdown by question, and a summary of the information about test output visibility:

If show_stdout was true in your otter_config.json, then the autograder output will be shown to students after grades are published on Gradescope. Students will not be able to see the results of hidden tests nor the tests themselves, but they will see that they failed some hidden test in the printed DataFrame from the stdout.

Below the autograder output, each test case is broken down into boxes. The first box is called “Public Tests” and summarizes the results of the public test cases only. This box is visible to students. If a student fails a hidden test but no public tests, then this box will show “All tests passed” for that question, even though the student failed a public test.

If the student fails a public test, then the output of the failed public test will be displayed here, but no information about hidden tests will be included. In the example below, the student failed a hidden test case in q1 and a public test case in q4.

Below the “Public Tests” box will be boxes for each question. These are hidden from the student unless show_hidden was true in your otter_config.json; if that is the case, then they will become visible to students after grades are published. These boxes show the results for each question, including the results of hidden test cases.

If all tests cases are passed, these boxes will indicate so:

If the studen fails any test cases, that information will be presented here. In the example below, q1 - 4 and q1 - 5 were hidden test cases; this screenshot corresponds to the second “Public Tests” section screenshot above.