January 2009 – The National Commission for the Certification of Crane Operators (NCCCO) has been commended by the American National Standards Institute (ANSI) for the development of its practical examinations. Specifically, the commendation was awarded for NCCCO’s development of “inter-rater reliability measures” for its practical exams.
Inter-rater reliability refers to the likelihood that two examiners will independently record a candidate’s performance in the same way. Although much of the opportunity for subjectivity on CCO Practical Exams has been removed by means of its sophisticated design (including site lay-out and scoring system), a high measure of inter-rater reliability is essential for the practical exams to measure an operator’s skill fairly and accurately.
“NCCCO has not only provided Inter-Rater Reliability (IRR) of 94.75 . . . it has also analyzed the percentage of rater inconsistencies which impact pass/fail scores,” ANSI’s commendation states. “This IRR analysis, and the articulation of the data which support it, are exemplary of the best practices in performance testing.”
ANSI’s audit of NCCCO occurs annually as a requirement of its accreditation to ISO 17024, with an onsite inspection by ANSI auditors every other year. NCCCO previously received two commendations from ANSI during its main audit in 2007, both for its practical exam procedures.