Early Exposure to Valid Test Items Improves EMT Student Performance on Summative Exams

Ron Lawler, BUS, NRP; Elias Frick, BA, NRP; Ben Getsug, AAS; Brian Hendrickson, MS, EMT-P; Sara Houston, BS, NREMT-P; Evelyn Hunter, BS, MSW, LSW; Andy Lovell, NREMT-P; Luke Stanke, PhDc; Hanorah Vanni, BA; Mark Volker; Rachel Wasilewski, BS; Virginia Wilson, BSBA, BS, EMT-P

Introduction
There are several avenues for formative testing available to EMT programs, however most are not validated. Research in a variety of healthcare educational disciplines shows that valid formative examinations improve student outcomes on summative exams. Existing EMS-specific research is scarce on this subject.

Hypothesis
Exposure to valid, formative, secure EMT exams is predictive of success on valid, summative EMT exams.

Methods
EMT Readiness Exam 2 (ERE2) and EMT Unit Exam (EUE) results were retrospectively collected from Fisdap, an online EMS education data collection system from December 2010 through December 2014. Inclusion criteria consisted of student consent to research. Data points for the ERE2 and each EUE included scores and program ID. Data was fit to a hierarchical linear model (HLM) where level-1 contains information about the number of unit tests taken by a student and level-2 intercept for program affiliation (Table 1).

Table 1

The EMT Readiness Exam 2 (ERE2) and EMT Unit Exams (EUE) are valid assessments that adhere to the National EMS Practice Analysis developed by the NREMT and current American Heart Association Guidelines. The exams were constructed using classical test theory and pilot-tested with geographically diverse EMT students. The six EMT Unit Exams cover Airway, Cardiology, Medical, Obstetrics/Pediatrics, Operations, and Trauma. Each exam is designed to test knowledge and critical thinking ability in a given subject area. After pilot, the ERE2 was found to be predictive of first-time NREMT success: 97% of students who passed the ERE2 at the recommended Angoff cut score (72%) passed the NREMT exam on their first attempt (Graph 1).
 
Graph 1
 

Limitations
There are a number of unknown variables that this study does not control for including: context of EUE use (as pre-tests, module end exams, or pre-final exam prep), if exams were included in grade calculation and if there was any student exposure to other valid formative exams prior to the ERE2.

Results
The mean score of those students who took no EUEs was 67% (n = 22,359) while those who took all six exams was 71% (n = 1,557). Using the HLM to control for students in the same program, students who took all 6 unit exams still scored 1.8% higher on the ERE2 (Table 2).

Table 2

Conclusion
Improvement in EMT student scores on the ERE2 was correlated with the previous exposure to all six EUEs. Given the predictive value of the ERE2, this small increase in average score would result in more students achieving success on the NREMT certification exam. Based on ERE2 pilot statistics, at the average score without any unit exams (67%), about 74% of students could be expected to achieve first-time NREMT success. At the average score with formative exams (71%), that percentage increases to 92%. Given this result, EMT training program directors should consider using validated formative exams.

Next Steps
There is plenty of opportunity for further research in this area. Possible topics of research include: an examination of other valid test items used in place of/in addition to Fisdap Unit Exams, whether the length of program also has an effect on ERE success, whether Unit Exam exposure predicts success at the paramedic level, whether the timing of formative exam administration has an effect on the summative exam results, whether a student’s score on the first formative exam is predictive of attrition, and whether the administration of a valid entrance examination would positively impact summative exam results. There’s also an opportunity to examine whether exposure to formative exams specifically impacts performance on the critical thinking portion of the summative exams.