Is passing the exam enough?
It is generally acknowledged that dental students should be allowed to progress either in their programmes of study, or into their profession, by their ability to exceed pre-defined thresholds of learning outcomes.
With this criterion-referencing the performance of peers does not influence the success of any student, there is no restriction on how many individuals exceed the threshold. This is perfectly sensible as patients expect their dentist to be able to do things.
However, is focus upon criterion-referencing too simplistic? Is it enough to jump the hurdle once? How about thinking about a different summative assessment.
The driving test in England, Scotland and Wales is highly structured with a pass rate of just under 50%. In 2014 about 75% of the adult population of Great Britain held a driving licence implying that the skills and knowledge required are attainable by most people, albeit after a few attempts. This makes me think that a driver who needs 33 attempts to pass is an outlier, lacks innate aptitude, and could be assumed to be far less capable than most. Is it then reasonable to do norm-referencing and judge students in relation to their peers? As a patient, would you like to be treated by the worst dentist in their year? Let's look beyond the UK system.
The National Board Dental Examinations are sat US-wide and comprise two parts. Part 1 is a multiple choice test of biomedical sciences and dental anatomy with just under ten thousand students sitting it in 2016. The responses to the examination are collated into a standard score between 49 and 99 with a passing score of 75. It is important to note that the score is not a percentage as the equalising process is non-linear.
The standard setting process is highly documented and the organisers stress the importance of criterion-referencing in making the process fair.
So, the 'Reference Group' help to set the standard? Indeed, it makes sense that to monitor performance over successive sittings of the examination that the examiners look at the scores of students who have have undergone similar education to each other. Nonetheless, it does imply that students are indirectly judged against their peers.
Notably the average score for the reference group is 82 so students scoring between 75 and 82 will pass but might be considered to be 'below average' even though the examination has not been designed to discriminate between passing and excellent students. For this very reason, the scores of passing students are not released!
Returning to the driving test, only around 20% of learners pass the first time they take the driving test. Surprisingly it has been suggested that such first-time successful drivers are more at risk of being involved in an accident than those who pass after 2 or three attempts. This might be explained through applying models such as The Four Stages of Competence or the Dunning-Kruger effect which merely suggest that the learner who passes easily is too naive to recognise their inability, despite being able to exceed the threshold once.
We should also be concerned about testing time. All tests are snapshots of students' ability and it stands to reason that the more a student is observed the more can be learnt about them. The driving test, mentioned above, only lasts 40 minutes, less than the time most people spend commuting to work. Is it really possible to gain a true view of a driver's ability from what amounts to less than a single typical journey? Perhaps random events on test day play a greater effect than the learner driver's skill, irrespective of whether it is the 1st or 33rd attempt? This explains why the NBDE Part 1 contains a massive 400 questions taken over 8.5 hours.
Recent work on longitudinal assessment provides a structure for students to demonstrate their aptitude multiple times. This increases testing time, without needing a single huge test, and also, very importantly generates information to help students learn before their next assignment, this is often described as feed-forward. Repeated, voluntary assessment in now being piloted in some US specialist recertification programmes as a tool that is effective and readily accessible. A form of this approach is in use at Peninsula Dental School to assess student's development and promote learning and although effective it was reported to be labour intensive.
As dental educators strive to encourage lifelong learning and professionalism in students perhaps it is time to move away from milestone examinations and spend time guiding the outlying, underperforming students towards their peers.
 |
| From the GDC's Standards for Education 2015 |
With this criterion-referencing the performance of peers does not influence the success of any student, there is no restriction on how many individuals exceed the threshold. This is perfectly sensible as patients expect their dentist to be able to do things.
However, is focus upon criterion-referencing too simplistic? Is it enough to jump the hurdle once? How about thinking about a different summative assessment.
 |
| From the UK Mirror tabloid newspaper |
The driving test in England, Scotland and Wales is highly structured with a pass rate of just under 50%. In 2014 about 75% of the adult population of Great Britain held a driving licence implying that the skills and knowledge required are attainable by most people, albeit after a few attempts. This makes me think that a driver who needs 33 attempts to pass is an outlier, lacks innate aptitude, and could be assumed to be far less capable than most. Is it then reasonable to do norm-referencing and judge students in relation to their peers? As a patient, would you like to be treated by the worst dentist in their year? Let's look beyond the UK system.
The National Board Dental Examinations are sat US-wide and comprise two parts. Part 1 is a multiple choice test of biomedical sciences and dental anatomy with just under ten thousand students sitting it in 2016. The responses to the examination are collated into a standard score between 49 and 99 with a passing score of 75. It is important to note that the score is not a percentage as the equalising process is non-linear.
 |
| From an ADA presentation |
 |
| From the JCNDE's technical report |
Notably the average score for the reference group is 82 so students scoring between 75 and 82 will pass but might be considered to be 'below average' even though the examination has not been designed to discriminate between passing and excellent students. For this very reason, the scores of passing students are not released!
Returning to the driving test, only around 20% of learners pass the first time they take the driving test. Surprisingly it has been suggested that such first-time successful drivers are more at risk of being involved in an accident than those who pass after 2 or three attempts. This might be explained through applying models such as The Four Stages of Competence or the Dunning-Kruger effect which merely suggest that the learner who passes easily is too naive to recognise their inability, despite being able to exceed the threshold once.
We should also be concerned about testing time. All tests are snapshots of students' ability and it stands to reason that the more a student is observed the more can be learnt about them. The driving test, mentioned above, only lasts 40 minutes, less than the time most people spend commuting to work. Is it really possible to gain a true view of a driver's ability from what amounts to less than a single typical journey? Perhaps random events on test day play a greater effect than the learner driver's skill, irrespective of whether it is the 1st or 33rd attempt? This explains why the NBDE Part 1 contains a massive 400 questions taken over 8.5 hours.
Recent work on longitudinal assessment provides a structure for students to demonstrate their aptitude multiple times. This increases testing time, without needing a single huge test, and also, very importantly generates information to help students learn before their next assignment, this is often described as feed-forward. Repeated, voluntary assessment in now being piloted in some US specialist recertification programmes as a tool that is effective and readily accessible. A form of this approach is in use at Peninsula Dental School to assess student's development and promote learning and although effective it was reported to be labour intensive.
As dental educators strive to encourage lifelong learning and professionalism in students perhaps it is time to move away from milestone examinations and spend time guiding the outlying, underperforming students towards their peers.
Comments
Post a Comment