.. role:: inline-html(raw)
    :format: html


.. _workflow_executing_submissions_gradescope:

Grading on Gradescope
=====================

This section details how the autograder runs and how results are displayed to students and 
instructors on Gradescope. When a student submits to Gradescpe, the autograder does the following:

#. Copies the tests and support files from the autograder source
#. Globs the first IPYNB file and assumes this to be the submission to be graded
#. Reads in the log from the submission if it exists
#. Grades the notebook, globbing all tests and grading from the log if specified
#. Looks for discrepancies between the logged scores and the autograder scores and warngs about 
   these if present
#. If indicated, exports the notebook as a PDF and POSTs this notebook to the other Gradescope 
   assignment
#. Generates the JSON object for Gradescope's results
#. Makes adjustments to the scores and visibility based on the configurations
#. Writes the JSON to the results file
#. Prints the results as a dataframe to stdout


Writing Tests for Gradescope
----------------------------

The tests that are used by the Gradescope autograder are the same as those used in other uses of 
Otter, but there is one important field that is relevant to Gradescope that is not pertinent to any 
other uses.

As noted in the second bullet :ref:`here <test_files_ok_format_caveats>`, the ``"hidden"`` 
key of each test case indicates the visibility of that specific test case.


Results
-------

Once a student's submission has been autograded, the Autograder Results page will show the stdout of 
the grading process in the "Autograder Output" box and the student's score in the side bar to the 
right of the output. The stdout includes information from verifying the student's scores against the 
logged scores, a dataframe that contains the student's score breakdown by question, and a summary of 
the information about test output visibility:


.. image:: images/gradescope_autograder_output.png
    :target: images/gradescope_autograder_output.png
    :alt: 

If ``show_stdout`` was true in your ``otter_config.json``, then the autograder 
output will be shown to students *after grades are published on Gradescope*. Students 
will **not** be able to see the results of hidden tests nor the tests themselves, but they will see 
that they failed some hidden test in the printed DataFrame from the stdout.

Below the autograder output, each test case is broken down into boxes. The first box is called
"Public Tests" and summarizes the results of the *public* test cases only. This box is visible to 
students. If a student fails a hidden test but no public tests, then this box will show "All tests
passed" for that question, even though the student failed a public test.

.. image:: images/gradescope_public_passed.png
    :target: images/gradescope_public_passed.png
    :alt: 

If the student fails a public test, then the output of the failed public test will be displayed here,
but no information about hidden tests will be included. In the example below, the student failed a
hidden test case in ``q1`` and a public test case in ``q4``.

.. image:: images/gradescope_public_failed.png
    :target: images/gradescope_public_failed.png
    :alt: 

Below the "Public Tests" box will be boxes for each question. These are hidden from the student
unless ``show_hidden`` was true in your ``otter_config.json``; if that is the case, then they will
become visible to students after grades are published. These boxes show the results for each question,
*including the results of hidden test cases*.

If all tests cases are passed, these boxes will indicate so:

.. image:: images/gradescope_questions_passed.png
    :target: images/gradescope_questions_passed.png
    :alt: 

If the studen fails any test cases, that information will be presented here. In the example below,
``q1 - 4`` and ``q1 - 5`` were hidden test cases; this screenshot corresponds to the second "Public
Tests" section screenshot above.

.. image:: images/gradescope_question_failed.png
    :target: images/gradescope_question_failed.png
    :alt: