Quality Assurance in Examining
Paula Goddard, Fellow of The Chartered Institute of Educational Assessors and Senior Examiner, discusses the idea of changing the mindset of examining from quality control (dealing with errors after they have happened) to quality assurance (to getting it right first time) and suggests a system where no error sampling would be required.
The future of exams lies in an increased emphasis on quality assurance rather than quality control. That is there will be more measures to stop mistakes happening and getting the system right rather than checking for errors after they’ve happened.
If you’re an engineer the terms quality assurance and quality control will be familiar to you - it often comes up as an exam question asking you to explain the difference. Applied to examining, the quality control system samples marked exam scripts and looks for errors, then feeds the results back to the exam marker or is used to adjust marks after it’s all over - re-marking we call it.
Switch mindset to saying that no errors should get into the system at all and you get the system of quality assurance introduced by many Japanese manufacturers in the 80s. Applied to examining this means having hired the correct people to write and design the exam questions in the first place, having the correct admin procedures and expertly written exam specifications.
But we have this now! Oh no we don’t. If we did then then the double-checking of exam questions before printing would become unnecessary and no errors would creep into the system and escalate. But how would this impact on examiner recruitment and training? Particularly examiner standardisation? It would reduce the need to recruit several thousand more examiners and redeploy those already in the system to concentrate fully on marking and not sampling or double marking - the traditional role of Team Leaders and Principal Examiners.
Quality Assurance mindset would say that exam markers must be so fully trained and understand the mark scheme that they make no marking errors what so ever. How do we achieve that? By altering the existing 4-5hr standardisation training, where examiners practise marking 10-15 exam scripts to one where they practise marking about 100 exam scripts before they mark ‘live’ scripts.
Yes this would mean the standardisation training would take three days rather than one, and would move all the costs upfront rather than during the process. But it would eliminate the need for checking during marking and so all those currently involved in that would become markers.
So now let’s take this idea to the start of the system - designing of specifications that are so clear and unambiguous that clear exam questions can be written, and the system needs no further explanation. If this became so the main pulling power of teachers into examining - that they get to fully understand the inside ‘secrets’ of answering exams - would disappear.
If the whole system is transparent from the outset then why would teachers become examiners? They wouldn’t. Add this to the increasing time pressures on teachers and you can see that introducing quality assurance mindset may have unintended consequences.
So who would mark the exams then? Well a purist of quality assurance would insist they wouldn’t need marking as the filtration of knowledge through the system would say they would all be correct. But no system is perfect, particularly where human learning is involved. So who would mark the exams?
This article first appeared as a series of posts on LinkedIn during January 2018.