Abstracts

Accepted Abstract and presented as Poster September 2004 - NAEMSE Symposium, Los Angeles, California: OSPE Phase 1

OSPE Phase 1- 2003 Abstract

Title: Online Summative Paramedic Exam (OSPE) –Phase 1: Feasibility of using web based testing tools to develop and administer a final summative exam to paramedic students.

Principal Investigator: David Page MS, NREMT-P

Co-Investigators: Marti Breiter BSN, NREMT-P; Gordon Kokx BS, NREMT-P; James Manson, NREMT-P.

Thesis/Hypothesis: It is feasible to use web based tools to develop and administer a summative evaluation for paramedic students.

Introduction: Web based testing tools allow multiple institutions to work collaboratively in the development, validation and administration of exams. No research is available regarding the feasibility of using web-based testing tools during paramedic programs.
Methods: Emergency Medical Services educators and medical directors from seven geographically diverse paramedic training institutions used an online instructional management system, WebCT, to develop, review and administer a summative paramedic exam to current students. The 200 item exam was divided into six topical categories: Airway, Cardiology, ECG interpretation, Trauma, Medical, Obstetrics-Pediatrics, and EMS operations. Each exam item was reviewed by at least four subject matter experts. A modification of the Angoff method was used to determine the exam cut score. The proctored exam was administered to 95 graduating paramedic students. A brief questionnaire and consent form were administered upon completion of all sections of the exam.

Results: Of the ninety-five students tested, eighty students (91%) consented to sharing their anonymous data for the study. Sixty-four students (80%; 95% CI: 70% - 88%) reported taking the exam in a proctored environment. Forty students (50%; 95% CI: 39% - 61%) indicated that they would have had equal or less opportunity to cheat as they would have if they had taken the exam on paper. Four students (5%; 95% CI: 1% - 12%) indicated they had more opportunity to cheat than if the exam was taken on paper. Seven students (9%; 95% CI: 4% - 17%) reported that it would have been possible to cheat on this exam. Sixty-two students (78%; 95% CI: 68% - 86%) indicated that they would recommend using computerized exams in the future. Eighteen students (23%; 95% CI: 14% - 33%) indicated they did not recommend using computerized exams in the future.
Conclusions: It is feasible to develop and administer a summative paramedic exam using web based tools. Online delivery allowed multiple geographically diverse institutions to collaborate. The majority of students recommended computerized exams. Relatively low numbers indicated that this computerized format would provide more opportunity to cheat.

 

OSPE Phase 2-abstract has been accepted for a Poster at the September 2005 conference.

Title: Predicting pass rates on the National Registry paramedic written exam using an online summative exam (OSPE Phase 2: Validity).

Principal Investigator: David I Page MS, NREMT-P
Co-Investigators: James P Manson, NREMT-P, Gordon Kokx BS, NREMT-P;

Thesis/Hypothesis: Students who pass the online summative paramedic exam (OSPE) will pass the National Registry paramedic written exam (NRPWE) on the first attempt.

Introduction: Anecdotal evidence suggests that many paramedic instructors lack access to valid multiple-choice exams, and that students experience significant anxiety around taking the NRPWE. Phase 1 of the OSPE project demonstrated that it is feasible to use web based tools to develop and administer a summative evaluation for paramedic students across multiple institutions (Page et al. 2004). Phase 2 seeks to demonstrate that such an exam is valid and reliable by comparing it to the NRPWE.

Methods: Emergency Medical Services educators and medical directors from eleven geographically diverse paramedic training institutions used an online instructional management system, WebCT, to develop, review and administer a summative paramedic exam to current students. The 200 item exam was divided into six topic areas similar to the NRPWE. Exam items were reviewed by at least four subject matter experts. A modification of the Angoff method was used to determine an exam cut score of 73%. The proctored exam was administered to 257 graduating paramedic students. A brief questionnaire and consent form were administered upon completion of all sections of the exam. Instructors reported NRPWE scores back to the authors.

Results: Of the 257 students tested, 252 (98%) consented to sharing their anonymous data for the study. Of these, 22 students did not take the NRWPE, scores were obtained for 230 (91%), representing eleven geographically diverse paramedic training programs. 113 (49%) passed the OSPE. Of these 110 (97%) passed the NRWPE on the first attempt. A student scoring less than 65% on the OSPE had an 80% likelihood of failing the NRWPE.

Conclusions: The hypothesis was proven. Students who pass the OSPE will pass the NRPWE. The OSPE is a valid and reliable evaluation. It is possible to use online tools to validate and administer an exam that will accurately predict a student’s performance on a national certifying exam.