OSCAR- ITEM SETUP- Qualification Settings

Qualification is a setting in OSCAR that will require users to score one more qualification sets before they are able to access live scoring for an item. Qualification settings are customizable to allow multiple different pass requirements as well as requiring the rater to see and/or pass multiple qualification sets. For any qualification settings to take effect, the item must be Saved. Also, before turning on Qualification for an item, one or more qualification sets should be created and active from the QC Setup page. QC Setup is where the pass threshold for an individual set is configured. There are two pages where Qualification results can be viewed, QC Reporting and User Portfolio. 

Enabling Qualification

Once Qualification is turned on for an item and while it remains enabled, no raters will be able to score responses in the item without passing the qualification requirements. Qualification can be turned off again at any time (any qualification settings will remained saved) and raters will be able to score without restriction, regardless of their previous qualification results on the item. 

Must See and Must Pass

The first options to consider for Qualification are the "Must See" and "Must Pass" input boxes. Here, input the number of qualification sets raters must score (Must See) and the number of individual set requirements each rater must pass (Must Pass). The number of sets required to pass must be equal to or less than the number of sets that raters must see. Also, note that there must be at least as many qualification sets active as the number input for "Must See" or no rater will be able to meet the qualification requirement.  

Lock on Fail

Decide what behavior will happen when/if a rater does not meet the passing requirements for qualification. By default, the rater will be locked out of scoring and made inactive if they fail the passing requirements. Uncheck this box if there should be no change in system access after failing. Note, users that are inactive are still able to log into OSCAR and view other pages they have access to; they will not be able to score or take another qualification set until they are made active again. Make a user active again from the Users page or from QC Reporting while reviewing their failed result.

Cross-set Requirements

If all passing requirements are only individual set requirements, no additional settings need to be added and qualification can begin. However, it is also possible to add "Cross-set Requirements" that will create rules for passing that are checked against all qualification sets in addition to the individual set requirements. Use the button to add a cross-set requirement by choosing an option from the dropdown, completing any additional input, and pressing the button to save the requirement. Additional Cross-set Requirements can continue to be added in this way until all needed requirements have been added. Any Cross-set Requirements can be edited or deleted as needed.

Cross-set Requirements- Options

There are two basic types of Cross-set Requirements- those that apply to the overall scores given on on a response and those that apply to scores given on specific traits. When using trait-specific requirements, these can be applied to an individual trait by choosing the specific rubric trait or to all traits on the rubric. Both overall and trait-specific requirements will be calculated against all sets the rater scores, defined as the number of Must See sets. 

• Exact Agreement- Set a threshold percentage for raters to meet by matching the true score for all traits on qualification set responses exactly.
• Trait-Specific Exact Agreement- Set a threshold percentage for raters to meet by matching the true score on a specific trait exactly. 
• Exact/Adjacent Agreement- The threshold percentage set will be met by achieving an exact or adjacent match to the overall true score on each response.
• Trait-Specific Exact/Adjacent Agreement- The threshold percentage set will be met by achieving an exact or adjacent match to the specific trait's true score on each response.
• Max Discrepancies- The total number of non-adjacent overall scores given on each response by the rater that will result in failing qualification. 
• Max Discrepancies Per Trait- The total number of non-adjacent specific trait scores given on each response by the rater that will result in failing qualification. 
• Drop a Set- Remove a set that the rater has seen from the final calculation of all Cross-set Requirements. Choose an option for how the set dropped will be determined. Each option is based on the scores for an individual set. 

Qualification Results

After qualification has begun, results can be viewed from the Qualification Reporting page or the User Portfolio page. The Qualification Reporting page will display results on individual sets as well as overall results on all sets. Use the User Portfolio page to view user qualification progress against Cross-set Requirements. Below is an example report from the User Portfolio page where the rater passed qualification. The item qualification settings were 50% exact agreement on each set, and the Cross-set Requirements are displayed on the page.