OSCAR - QC Setup: Adding a Qualification Set

Back to QC Overview 

Qualification Sets Overview

From the QC Setup page, Qualification sets can be created to use when a item's Qualification settings are enabled. A Qualification set is a number of responses that are either Escalated from Scoring or the Auditing page or are loaded directly into the Qualification set from outside of OSCAR. Qualification responses are saved to a set and their true scores configured using the Manage Responses feature. Each set must have a scoring Rule and a Pass Percent configured that will determine if raters successfully pass the set. Qualification sets will not be be fed to scorers until the set is made Active from the Set Options. Qualification sets will follow a set of rules and processes when used or a Qualification Set Workflow.

Rules

The set rule will determine how results are calculated for meeting the Pass Percent. There are five rule settings available in the Qualification set:

1. Exact Agreement, By Response - the rater's score on each response must have the exact same overall score as the true score for the response to be counted as correct
2. Exact Agreement, By Trait - the rater's score on each trait must have the exact same scores as the true score to be counted as correct
3. Exact/Adjacent, By Response - the rater's overall score on each response must be within 1 point of the overall true score for the response to be counted as correct
4. Exact/Adjacent, By Trait - the rater's score on each trait must be within 1 point of the true score to be counted as correct
5. Omaha- customize the maximum overall difference in score allowed (Max Score Diff) and the number of non/adjacent trait scores (Max Non-adjacent) to determine if the response is counted as correct

Overall Set Percentages

For each set, determine what percentage (correct/total) of responses must be correct for a rater to pass. Pass Percent can be set when Exact Agreement, By Response and Exact/Adjacent, By Response rules are selected.

When trait rules are selected, the passing percentage will be determined at the trait level rather than by the response (correct trait scores/total trait scores). Along with a pass threshold, a minimum threshold can also be set. Both settings will apply to all traits on the rubric. Percentage of Scores Overall and Minimum for each Trait can only be set when Exact Agreement, By Trait and Exact/Adjacent, By Trait rules are selected.

When the Omaha rule is selected, the passing percentage will be determined by the number of correct responses (correct/total), according to the customized rules created. Max Non-adjacent is the number of non-adjacent trait scores allowed on a response and Max Score Diff is the total difference allowed between the overall rater score and the overall true score. If enabled, the Resolve Min/Max checkbox will count each score difference greater than one as a separate non-adjacent towards the Max. For example, if a rater gives a trait score of 1 and the true score is 4, this would count as 2 non-adjacents.

Set Options

Along with making a set active or inactive, there are two additional toggle options for Qualification sets. The Reviewable option will allow raters to view the response true scores and any feedback/annotations saved to the responses after they've finished scoring all responses in the set. Normally, responses in a set will be fed to raters in the order determined when saving the set responses in Manage Responses. Using the Randomize Delivery option will create a random order for each rater when scoring the set, though the results in QC Reporting will remain in set order. 

Templates

A template can be created that will save the rule and overall setting combinations. This template can then be used when creating new sets in any section or item. 

Qualification Set Workflow

Along with the options described above, there are a few other processes to note regarding how Qualification sets behave in OSCAR.

• Qualification sets are accessed by raters from the scoring page by using the Scoring button. When a Qualification set is active, the rater will receive a notification that they are about to begin a Qualification set when they use the Score button.
• While scoring a Qualification set, the rater will be able to navigate backwards and forwards through the set, with the ability to adjust any score until they submit the last score in the set. Then, all of their scores will be submitted and calculated for pass/fail requirements. They will also be provided a link to review their performance on the responses IF the set has been made Reviewable.
• When more than one set is active, each set will always be completed by raters in the vertical order (top to bottom) the sets appear on the QC Setup page. 
  • To complete Qualification, raters will need to score the number of sets configured as "Must See" under the item's Qualification Settings.
  •  If a rater fails Qualification, they will automatically receive the next active Qualification set when hitting the score button again. In this way, raters can attempt to qualify multiple times should the project business rules allow it. Note, raters will not be able to subsequently attempt to qualify if there are no additional sets active or the user is inactive.
• For supervisors, qualification set results can be viewed from the QC Reporting page or the User Portfolio page. Use the QC Reporting page to also manage user results (reset, delete, unlock raters). Raters can view their own results directly after scoring or from their Personal Dashboard using the Qualification tab (if enabled for their Role). 
• After a Qualification set has been used, changes to the set settings can still be made. However, these changes will only affect the qualification status and set behavior on future results.