Grade manual answers automatically / Post-autograding hook

[Paul2708]

Is your feature request related to a problem? Please describe.
Some lectures want to use external and authenticated services to check manual answer cells, which is not possible right now, at least as far as I know.

Describe the solution you'd like
It would be cool to have a hook that takes a cell with context information and returns the points for that cell.
This way, instructors can define their own grading logic that can use external services.

Describe alternatives you've considered
As an alternative, this kind of hook can also be placed in the grader service configuration. Although the responsibility shifts from the instructor to the administrator, it might be easier to implement it.

More generally, it can be implemented using a "post-autograding" hook that takes the entire notebook, including answers and points, and returns a new "graded version" to update the results accordingly.

What do you think about it?

[nadjajovancevic]

Hi Paul,

the feature you described seems like a good idea, we would just need to discuss in greater detail how to implement it.
For that we would also like to know what type of configuration you wish for. To us the most reasonable set up is to use this feature either assignment-wide or lecture-wide.

[Paul2708]

I was thinking about a hook that looks like follows:

# grader_service_config.py
def post_grade_hook(submission: Submission, graded_submission: GradeBook) -> GradeBook:
    pass

c.LocalAutogradeExecutor.post_grade_hook = post_grade_hook

# local_grader.py
def start(self):
    try:
        self._clean_up_input_and_output_dirs()
        self.git_manager.pull_submission(self.input_path)

        autograding_start = datetime.now()
        self._write_gradebook(self._put_grades_in_assignment_properties())
        self._run()
        autograding_finished = datetime.now()

        # Apply hook
        updated_gradebook = self.post_grade_hook(self.submission, "get gradebook from autograde")
        "write updated gradebook to file so it is correctly picked up by self._set_properties"

        files_to_commit = self._get_whitelisted_files()
        self.git_manager.push_results(files_to_commit, self.output_path)
        self._set_properties()
        self._set_db_state()
    except Exception as e:
        # ...

I don't know if GradeBook is the correct data structure holding the autograded points and comments, but I was thinking of something like that.

So if lecturers want custom grading logic, they have to contact the administrators to write/paste the necessary code. If you find a way to move this configuration from the grader_service_config.py to the instructor, this would be even better.