If an instrument requires some professional judgments in scoring, then the manual should also include information on:
interrater reliability.
If an instrument requires professional judgments in scoring, it is crucial for the manual to include information on several key aspects that can guide the person scoring the instrument. Here are some important elements that should be included in the manual:
1. Scoring Criteria: The manual should clearly define the scoring criteria for each component or item of the instrument. It should outline what constitutes a low, medium, or high score, and provide examples or descriptions to illustrate each level. This helps ensure consistent and accurate scoring.
2. Scoring Rules and Guidelines: The manual should provide clear instructions on how to apply the scoring criteria correctly. It should cover any special rules, guidelines, or considerations that need to be taken into account during the scoring process. For example, it may provide instructions on how to handle missing or ambiguous responses, or how to deal with rare or complex situations.
3. Rater Training Procedures: If professional judgments are required for scoring, it is crucial to include information about rater training procedures in the manual. This may involve explaining the training process, providing sample scenarios for practice, or offering resources for additional learning. Rater calibration methods, such as inter-rater reliability checks, may also be outlined.
4. Decision-Making Processes: In cases where professional judgments are needed to make scoring decisions, the manual should describe the decision-making process. This may include providing clear guidelines on how to weigh different factors, resolve ambiguities, or handle conflicting information. It should help the scorer make consistent and defensible judgments.
5. Rater Qualifications and Background: The manual may also include information on the qualifications and background needed for individuals who will be scoring the instrument. This ensures that the scorers possess the necessary expertise or experience to make professional judgments accurately.
Overall, the inclusion of comprehensive information in the manual helps to ensure that scorers have the necessary guidance, knowledge, and resources to score the instrument accurately and consistently.