Grading Container Image¶

When you use Otter Generate to create the configuration zip file for Gradescope, Otter includes the following software and packages for installation. The zip archive, when unzipped, has the following contents:

autograder/
| - environment.yml
| - otter_config.json
| - requirements.R
| - requirements.txt
| - run_autograder
| - run_otter.py
| - setup.sh
| files/
  | - ...
| tests/
  | - ...

Note that for pure-Python assignments, requirements.R is not included and all of the R-pertinent portions of setup.sh are removed. Below are descriptions of each of the items listed above and the Jinja2 templates used to create them (if applicable).

`setup.sh`¶

This file, required by Gradescope, performs the installation of necessary software for the autograder to run. The script template looks like this:

#!/usr/bin/env bash

# apt-get clean
# apt-get update
# apt-get install -y python3.7 python3-pip python3.7-dev

apt-get clean
apt-get update
apt-get install -y pandoc
apt-get install -y texlive-xetex texlive-fonts-recommended texlive-generic-recommended

# install wkhtmltopdf
wget --quiet -O /tmp/wkhtmltopdf.deb https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6-1/wkhtmltox_0.12.6-1.bionic_amd64.deb
apt-get install -y /tmp/wkhtmltopdf.deb

# update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1

# try to set up R
apt-get clean
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/'

add-apt-repository ppa:cran/libgit2
apt-get update
apt-get install -y build-essential libcurl4-gnutls-dev libxml2-dev libssl-dev libgit2-dev

# install conda
wget -nv -O {{ autograder_dir }}/source/miniconda_install.sh "{{ miniconda_install_url }}"
chmod +x {{ autograder_dir }}/source/miniconda_install.sh
{{ autograder_dir }}/source/miniconda_install.sh -b
echo "export PATH=/root/miniconda3/bin:\$PATH" >> /root/.bashrc

export PATH=/root/miniconda3/bin:$PATH
export TAR="/bin/tar"

# # install R dependencies
# conda install --yes r-base r-essentials 
# conda install --yes r-devtools -c conda-forge

# # install requirements
# pip3 install -r {{ autograder_dir }}/source/requirements.txt
# pip install -r {{ autograder_dir }}/source/requirements.txt
# Rscript {{ autograder_dir }}/source/requirements.r

# install dependencies with conda
conda env create -f {{ autograder_dir }}/source/environment.yml
conda run -n {{ otter_env_name }} Rscript {{ autograder_dir }}/source/requirements.r

# set conda shell
conda init --all

# install ottr; not sure why it needs to happen twice but whatever
git clone --single-branch -b {{ ottr_branch }} https://github.com/ucbds-infra/ottr.git {{ autograder_dir }}/source/ottr
cd {{ autograder_dir }}/source/ottr 
conda run -n {{ otter_env_name }} Rscript -e "devtools::install\\(\\)"
conda run -n {{ otter_env_name }} Rscript -e "devtools::install\\(\\)"

This script does the following:

Install Python 3.7 and pip
Install pandoc and LaTeX
Install wkhtmltopdf
Set Python 3.7 to the default python version
Install apt-get dependencies for R
Install Miniconda
Install R, r-essentials, and r-devtools from conda
Install Python requirements from requirements.txt (this step installs Otter itself)
Install R requires from requirements.R
Install Otter’s R autograding package ottr

Currently this script is not customizable, but you can unzip the provided zip file to edit it manually.

`environment.yml`¶

This file specifies the conda environment that Otter creates in setup.sh.

{% if not other_environment %}name: {{ otter_env_name }}
channels:
    - defaults
    - conda-forge
dependencies:
    - python=3.7
    - pip
    - nb_conda_kernels
    - r-base 
    - r-essentials 
    - r-devtools
    - libgit2
    - pip:
        - -r file:requirements.txt{% else %}{{ other_environment }}{% endif %}

`requirements.txt`¶

This file contains the Python dependencies required to execute submissions. It is also the file that your requirements are appended to when you provided your own requirements file for a Python assignment (unless --overwrite-requirements is passed). The template requirements for Python are:

{% if not overwrite_requirements %}datascience
jupyter_client
ipykernel
matplotlib
pandas
ipywidgets
scipy
seaborn
sklearn
jinja2
nbconvert
nbformat
dill
jupytext
numpy
otter-grader==2.1.6
{% endif %}{% if other_requirements %}
{{ other_requirements }}{% endif %}

If you are using R, there will be no additional requirements sent to this template, and rpy2 will be added. Note that this installs the exact version of Otter that you are working from. This is important to ensure that the configurations you set locally are correctly processed and processable by the version on the Gradescope container. If you choose to overwrite the requirements, make sure to do the same.

`requirements.R`¶

This file uses R functions like install.packages and devtools::install_github to install R packages. If you are creating an R assignment, it is this file (rather than requirements.txt) that your requirements and overwrite-requirements options are applied to. The template file contains:

`run_autograder`¶

This is the file that Gradescope uses to actually run the autograder when a student submits. Otter provides this file as an executable that activates the conda environment and then calls /autograder/source/run_otter.py:

#!/usr/bin/env bash

export PATH="/root/miniconda3/bin:$PATH"
source /root/miniconda3/etc/profile.d/conda.sh
conda activate {{ otter_env_name }}
python {{ autograder_dir }}/source/run_otter.py

`run_otter.py`¶

This file contains the logic to start the grading process by importing and running otter.run.run_autograder.main:

"""
Runs Otter on Gradescope
"""

import os
import subprocess

from otter.run.run_autograder import main as run_autograder

if __name__ == "__main__":
    run_autograder('{{ autograder_dir }}')

`otter_config.json`¶

This file contains any user configurations for grading. It has no template but is populated with the default values and any updates to those values that a user specifies. When debugging grading via SSH on Gradescope, a helpful tip is to set the debug key of this JSON file to true; this will stop the autograding from ignoring errors when running students’ code, and can be helpful in debugging specific submission issues.

`tests`¶

This is a directory containing the test files that you provide. All .py (or .R) files in the tests directory path that you provide are copied into this directory and are made available to submissions when the autograder runs.

`files`¶

This directory, not present in all autograder zip files, contains any support files that you provide to be made available to submissions.

Otter-Grader documentation

Grading Container Image¶

setup.sh¶

environment.yml¶

requirements.txt¶

requirements.R¶

run_autograder¶

run_otter.py¶

otter_config.json¶

tests¶

files¶