Taking up the SLAQ

 

Picture from Needpix

Background

Like all British Dental Schools, the University of Sheffield (TUoS) School of Clinical Dentistry was faced with an urgent and novel situation in march 2020 with regard to student assessment due to the Corvid-19 pandemic and subsequent compulsory move to online teaching.

The School normally holds the examination for the 5th BDS in late April and early May and this consists of written invigilated papers and clinical viva examinations during some of which patients are present.  

During the week beginning 13th March it became obvious that all education needed to move to a digital online format and on 20th March the TUoS declared the closure of all buildings. Thus we needed to arrange a modified Statement on Methods of Assessment (SoMoA) and organise a completely new form of assessment which assessed all the learning outcomes and satisfied the requirements of the General Dental Council (the professional regulatory body for dentistry).

On 23rd March the General Dental Council (GDC), Dental Schools council (DSC) and Health Education England (HEE) issued a joint statement providing the conditions under which students could be registered. This provided the parameters for the final assessment of students which needed to be met in order to ensure a registerable degree qualification.

Following TUoS policy on assessment, along with the Joint Statement, the SoMoA was amended and approval sought and obtained from faculty. The External Examiners were also informed of the changes.

What did we do?

There are normally two written papers, one designed to assess the student’s knowledge of oral surgery, oral and maxillofacial surgery, oral medicine, oral pathology, oral radiology and oral microbiology and the other integrated clinical practice, including restorative, paediatric and general primary care dentistry.

As the examinations needed to move to an online version, the emphasis moved from testing knowledge per se to assessing the students’ ability to apply this knowledge. Using educational theory relating to both open book examinations and the long case, alongside our extensive knowledge of how the clinical workplaces actually works we designed our new form of assessment, the Structured Long Answer Question, which quickly became known as the SLAQ.

The SLAQ is based around a clinical case. We looked at the GDC learning outcomes, which had already been mapped to the previous forms of the exams and tried to ensure that the cases covered these. We checked to ensure this by amending the structure of the real case upon which the SLAQ was based with extra details, for example in relation to social history.

We were aware that during the examinations students would have access to many sources of information to help them answer the question, but we felt that trying to stop them using these sources would be both very difficult and counterproductive. After all, in day to day practice we look things up all the time to ensure our patients get the best and safest care.

Giving students free reign to access any resource they wanted would be fair to every student and this would also make sure that students could “cheat all they liked” and thus not “cheat” at all!  The only condition put on this was that they MUST reference where they had used sources.

Our second issue was time. Most literature on open book examinations suggest that there should be a time limit so that students could make full use of any preparation and this helps discriminate between abilities once the assessments are marked.

TUoS had insisted that the examinations were in a form which became known as “take-aways”. The exam was to be held within a 24-hour window. This effectively meant that they would not be time limited, so we decided to use a word count to limit the amount of information the students could provide. It helped with marking and ensured that they would need to demonstrate structure in their thought processes. The scatter approach to examination (I’ll write everything I can think of) was therefore limited, if not eliminated.

A formative question was composed along with a marking rubric. The usual lecture held to prepare the students for finals was held, this time online using Blackboard Collaborate Ultra. The SLAQ was explained to the students and a (long) question and answer session followed. This was a very important part of the process as student buy in and understanding was essential. Good communication with the students appears to be crucial.

The formative question went “live” on 15th April. This was marked and a further meeting using Blackboard Collaborate Ultra was arranged with the students. At this point the senior staff marking the completely new format of online examinations were pleasantly surprised with the outcome. There was an amazing spread of marks ranging from 45% (just fail) to 90% (almost unheard of) Not only was it discriminatory but the students we expected to do well did, whilst those we expected to do less well also met our expectations, so instead of just releasing the already agreed information on pass or fail, it was then agreed we would release the students an indicative mark to aid their further study.

The actual summative examinations to replace both the written papers and the clinical vivas were compiled, approval sought from the External Examiners, a handbook for markers was produced and a training session held for staff, again using Blackboard Collaborate Ultra, on 28th April.

The examination replacing the written papers was held on 29th April and released to markers on 30th April. Each section of the SLAQ was designed so that it could be marked separately and the final result would be the result of the combination of marks from all the sections of both cases. This was calculated separately, just as the answers to the previous finals were calculated from the separate marks of each question. The questions were single marked and then moderated, the moderator using a global mark to see if this aligned with the combination from the examiners, which worked well and in the only case where the result differed it did so by 1%. There were no signs of either plagiarism or collaboration and the variation in answers was marked upon by many of the staff.

The marks were returned and at the analysis meeting on 14th May it became obvious that the system worked well and analysis had shown that the spread and mean results were very comparable with the results from students over the previous 5-year period.

How could an online open examination work so well?

Between the end of the examination and the release of the results we decided to ask both the students and the staff what they thought and did an in depth, on line evaluation. We found that on the whole the students thought it fair, authentic and actually learnt during the exam! The staff too were happy and found they actually preferred online marking.

Unfortunately, that’s all we can share, as TUoS policy prevents us sharing evaluation data further. Suffice it to say you will have to take our word for it, but the case based SLAQ, designed to test understanding, didn’t just fulfil the minimum standard of allowing us to test GDC learning outcomes and thus ensure we qualified students safe to go on the dental register. It exceeded all our expectations, received very positive comments from the external Examiners and met many of the criteria of current best practice assessment as it did indeed examine FOR and encourage further learning.