The Rubric Lab allows you to build, test, and publish a rubric that can be used to reliably and consistently grade a free response problem. It uses a "human-in-the-loop" AI workflow that can generate a complete rubric from as little as the problem statement and a model solution. The Lab drafts criteria, creates sample student answers that correspond to various quality levels, and then validates the rubric by grading those samples repeatedly — iterating on the criteria until they produce scores that are accurate and consistent. You stay in control throughout: reviewing, editing, and guiding the AI at each step so the final rubric reflects your standards and produces defensible grades across the full range of student work.

To produce a rubric, the Lab walks you through an iterative workflow.

Open the Rubric Lab.
Build an initial rubric based on the problem and solution.
Build a variety of sample answers that will be used to validate the rubric, ensuring coverage across all critera.
Auto-grade the sample answers using the rubric, measuring consistency and accuracy.
Troubleshoot any scoring inconsistencies, adjusting the rubric and/or sample answers as needed until it converges on a rubric that passes all validation tests.
Publish the rubric.
Use the rubric for auto-grading within your assignment.

Step 1: Opening the Rubric Lab

To open the Rubric Lab:

Go to the problem, and edit it.
Select an eligible part for which you want to add rubric-based grading to (essay, show your work, or file upload).
Under the How should this part be graded? section, select Open Rubric Lab. This will open the Rubric Lab.

Step 2: Build rubric

To build a rubric:

Provide an optional rubric or guidance.
Select Auto-generate to generate an initial rubric. The rubric will appear in the right summary panel. It may undergo several iterations as it is reviewed for coverage and quality before complete.
Once the rubric has been generated, review the rubric in the summary panel.
If you'd like changes, select Propose changes, describe what to adjust, and the Lab will regenerate. You can also select Edit rubric to make manual edits. See Getting started with free response grading and rubrics for more information on rubric criteria types.
When you are ready, select Build sample answers to continue.

Step 3: Build sample answers

Sample answers of varying quality provide a means to test whether a rubric can be used by the auto-grader to (1) ensure the grader's score matches an expected score, and (2) ensure that the grader consistently grades the same answer across multiple iterations.

To build sample answers:

Select the Build sample answers tab.
Optionally provide guidance on the sample answer generation.
Select Auto-generate. The sample answers will appear in the right summary panel. They may undergo several iterations as they are reviewed for coverage and quality before complete.
Once the sample answers has been generated, review them in the summary panel.
If you'd like changes, select Propose changes, describe what to adjust, and the Lab will regenerate.
You can select use Add sample to manually add a sample answer. Samples that you create manually are pinned and preserved if sample answers are later regenerated.
You can select a sample and then select Edit to edit a given sample answer.
When you are ready, select Score sample answers to continue.

Step 4: Score sample answers

By scoring sample answers against a rubric, you ensure that the auto-grader accurately and consistently grades using the rubric. Validation measures two metrics: whether the rubric produces the right scores, and whether it produces them reliably.

Accuracy - compares the graded score to the expected score across all samples. The mean absolute error must stay below 10%.
Consistency - measures how often repeated grading runs select the same criterion levels. Selection variance must also stay below 10%. Beyond the overall score, validation checks each criterion individually: criteria accuracy requires at least 90% agreement on every criterion across runs.

To score sample answers:

Select the Score sample answers tab, then select Score all. Each sample answer will be graded multiple times, measuring accuracy and consistency across runs.
If all samples pass, the rubric is ready to publish. Go to Step 5.
If one or more samples produces inconsistent or inaccurate results, you can select the sample row to expand it and see how the auto-grader scores against the rubric vs what was expected. Go to Step 4 to troubleshoot any issues.

Step 5: Troubleshoot scoring inconsistencies

Rubric and sample generation can sometimes produce a rubric which does not auto-grade consistently. The Lab workflow is designed to iteratively revise the rubric until all sample tests pass. To troubleshoot scoring issues:

Select the Score sample answers tab, and then select Auto-analyze issues. This will start a triage process that determines what changes are needed to the rubric and/or the sample answers.
When the process completes, select View issues. One or more rubric or sample answer edits will be suggested. After looking at each suggestion, select Auto-revise or Leave as-is as desired. You can also manually edit either the rubric or the sample answers at any time.
When the revisions are complete, go back to the Score sample answers tab to rescore the sample answers (Step 3).
Repeat as necessary until scoring produces consistent and accurate results and the rubric is ready for publishing.
Note: In the event that the Lab fails to converge on a successful rubric, you can start the entire workflow over by selecting the Publish rubric tab, and then selecting Start over.

Step 6: Publish the rubric

To publish a draft rubric for use by a problem part:

Select the Publish rubric tab. This tab provides workflow status to let you know what steps are needed into order to complete your rubric.
If all sample tests have passed, the rubric is ready for publication. Select Publish rubric to publish the draft and make it available for auto-grading in your assignment.
If there are outstanding issues in your workflow, resolve before continuing. You can optionally publish an incomplete or manually created draft rubric by selecting Publish Anyway.

Step 7: Use the rubric for auto-grading within your assignment

Enabling rubric-based auto-grading for a problem part — enable or disable auto-grading on a part that has a published rubric
Reviewing auto-graded student work — view rubric-based scores and per-criterion feedback on student submissions
Capturing student work to use as sample answers — use real student responses to improve rubric accuracy over time