Onderwijslogistiek Support

How do I set up an Assignment with multiple assessors? (e.g. four-eyes principle)

Aangepast op

Some assignments (Assignments) involve multiple assessors, for example in the following situations:

  • Assessment according to the four-eye principle (e.g. at graduation).
    Here, the first and second assessor first grade independently and then come to a joint evaluation.
  • The criteria of a rubric to be graded are distributed among several assessors (e.g. by different areas of expertise)

In this article we show possible approaches in Brightspace for both situations.

This article is complementary to articles on creating assignments and on creating rubrics

 

Set an assignment for assessment according to the four-eye principle

For some assignments it is necessary to work according to the four-eye principle, where two assessors independently provide feedback on the submitted work and then jointly arrive at one final grade including feedback.

Below, we describe two ways you can grade according to the four-eye principle:

The first method is recommended in principle, because it gives the least chance of errors during the grading process. The second method is somewhat simpler, but also more sensitive to error.

Applying the four-eye principle through special options for assigning assessors

In this method, the rubric associated with the assignment is displayed on multiple tabs:

  • one tab for the first assessor
  • a tab for the second assessor
  • and a tab for the joint evaluation.

Each assessor can complete only their own tab as well as the collaborative tab. An assessor can view the assessment of the other assessor(s), but cannot change it.

  1. Create an Assignment the way you are used to (see Creating Assignments), also linking the grading rubric.
  2. Under the linked rubric you will see under the heading 'Evaluators' how many assessors are available in the environment.

If you want to use this option, you must click Select Evaluators when creating the assignment. All teachers in the course are then linked to the assignment. You can then later make a division as to who assesses what. Once something has been submitted for the assignment, you can no longer click this option.

  1. Click Select Evaluators to link the assessors who may grade this assignment.

Also if you want to link all reviewers you must perform this step otherwise the other settings for this functionality will not be visible.

  1. Click Done to commit your selection.

If you have a pool of assessors, pair specific assessors for each student now or at a later time.

Note: Each assessor assigned to a student must actually perform the assessment, otherwise a joint assessment cannot be created. You can continue to change the assigned assessors for each student as long as that student's assessment has not yet been published.

Assigning assessors to students is as follows:

  1. In the Assignment under Evaluation & Feedback, click Manage Allocations. You will then see a table with one row per student and one column per selected assessor.
  2. Determine how you want to assign:
    1. If you want to assign assessors automatically, under Default Allocation Method, select how you want to assign assessors to students and click Apply Allocation Method:
      1. Assign all assessors to all students (e.g., if there are only two teachers in the environment).
      2. 'At random' assign two or three assessors per student.
    2. If you want to manually assign assessors, click Clear Table Selections at the top right of the table.

      You can now manually select two or three tutors per student.  

When manually assigning assessors to students, it can be helpful if the students in the course environment have already been assigned to groups for each assessor. Selecting a group will filter the list of students, allowing you to easily assign the entire list to one or two assessors at once via the 'n of n learners' checkbox at the top of the column.

  1. Click Done at the bottom to save the allocations. You can keep changing the allocations per student until that students' grade is published. Even after that, you could also make changes provided that you first 'retract' the grade (via Retract).
  2. Select the option 'Multiple individual evaluations' to indicate that multiple assessors will each complete their own rubric and then a joint rubric (aggregated).
  3. Under Publishers, you can set who may publish the grades. There are three options:
    • all evaluators may publish all grades;
    • assessors may only publish grades they have given themselves;
    • you select only some persons who may publish grades.

Note: To complete the 'Aggregated' rubric, the assessor must be able to publish. So make sure at least one of the two assessors has rights to publish grades.

Applying the four-eye principle by using three rubrics

Another way to apply the four-eye principle when grading an Assignment is to link three rubrics to one Assignment. This is as follows:

  1. Create the rubric you want to use for the first assessor and make two copies of it so that you have a total of three identical rubrics:
    • Name rubric - first assessor
    • Name rubric - second assessor
    • Name rubric - joint evaluation

Tip: the order in which you create the rubrics is also the order in which they will appear in the assessment form. Hence, start with creating the rubric for the first assessor.

  1. For the rubrics for the first and second reviewer, set them to be hidden from students, as well as the scores:
  1. Create the Assignment to be graded
  2. Link all three Rubrics to this assignment, setting the joint rubric as the default rubric for the grade:

Set an Assignment where assessors grade different criteria from the same rubric

In this case, you have a rubric with multiple criteria, some of which are graded by one teacher and some by one or more other teachers.

This can be done in Brightspace in two ways:

In principle, the first option is recommended because it gives the least chance of error during the assessment process. The second method is easier to set up, but also more sensitive to error.

* Currently, there is a bug in the first option that causes more sensitivity to error: Namely, if you actually grade the same student with another assessor at the same time, the total score only includes the score of the assessor who last saved the rubric that was filled-in. This is not automatically corrected when reopening, saving or publishing. All feedback and scores per criterion from both assessors are recorded though.

There is a workaround: If you open the filled-in rubric again, uncheck one of the criteria scores and then check it again, the total score is corrected. However, this can easily be forgotten and then the wrong score is published to the student. If the student or the assessor discovers the error AFTER publication it can still be corrected in the same way and then clicking Update.

To avoid errors as much as possible you could also choose for now, to coordinate who checks which students and when.

Set an Assignment where assessors simultaneously grade different criteria from the same rubric

  1. Create a normal assignment and link a rubric to it that contains all the criteria on which the assignment is to be graded.
  2. If necessary, use Criteria Groups that you give a name corresponding to the different areas of expertise of the assessors involved.
  3. Look under the linked rubric under the heading 'Evaluators'. There you can see how many assessors are available in the environment. Click Select Evaluators to link two or more to the Assignment.
  4. Only then will you see the options below.
  1. Under 'Coordinate multiple evaluators', select One shared evaluation.
  2. By default, all selected assessors can grade all students. With 'One shared evaluation' it doesn't matter if all selected assessors have actually evaluated. Still it is possible to assign specific assessors per (group of) student(s) via Manage Allocations. (see explanation above)

    However, you cannot indicate which assessor grades which criteria or criteria group. The assessors have to coordinate and follow up among themselves, because there is no security to prevent you from grading the wrong criteria. However, because of the different areas of expertise involved, it makes sense that each assessor will simply grade the criteria that fit their own expertise.
  3. Under 'Publishers', select who may publish the grades. There are three options here:
    • All assessors may publish all grades
    • Assessors may only publish grades they have given themselves
    • You select only some persons who may publish grades
  4. Click Save and Close at the bottom to save the changes.

With multiple assessors simultaneously grading different criteria of the same rubric

Set an assignment where assessors rate different criteria from the same rubric one after another

  1. Create a normal assignment.
  2. Link a rubric to the assignment that contains all the criteria on which the assignment is to be graded.

    Optionally you can work with multiple Criteria Groups in the rubric, giving each Criteria Group a name corresponding to the different areas of expertise of the assessors involved. Read about this in Creating Rubrics.

Nothing else needs to be configured.

NOTE: Assessors should coordinate among themselves when to assess which students, to avoid assessing the same student at the same time.

After one assessor graded, the rubric is saved as a Draft and then the next assessor can proceed with it. Publishing the result to the student is done in the same way as other assignments.

Vorige Artikel How do I recover a deleted Assignment?
Volgende Artikel How do I create a resit in my course?
Meer hulp nodig? Mail je vragen over Brightspace naar [email protected] of bel naar 088-0191020, doorkiesnummer 1. Bereikbaar op werkdagen tussen 9.00 en 17.00 uur.