Spent some time last night looking through the Ofqual doc on grading GCSE and A Level exams this summer. Realised I didn't really know how these exams are graded usually... No prizes for guessing that it's not exactly straightforward.

Here's a thread on what I found.

— Sam Mead (@samjamead) January 20, 2021

In January the country plunged back into lockdown (third time lucky…), and GCSE and A Level exams scheduled for the summer were hastily and prematurely cancelled. In the absence of a prepared Plan B, Ofqual opened a consultation period to seek help with figuring out the least-bad way to award grades this year.

Turns out, it’s not the most straightforward process even in a ‘normal’ year…

There are two main ways in which grade boundaries can be determined: Criterion based, or normal distribution based.

Norm-referencing and criterion-referencing — Via @daisychristo

Criterion based grades require students to achieve certain competencies. Grades are awarded based on which competencies are achieved. The more demanding the competencies a student can tick off, the higher their grade. This is how grades are awarded at university (mostly).

Normal distribution based grades basically rank students and award grades on the normal distribution bell curve. The top X% get grade A, the next Y% get grade B, and so on.

GCSEs and A Levels use a combination of both. The grades are mostly based on a normal distribution, but each exam board uses statistics and expert input to adjust their grade boundaries each year. This ensures continuity between years.

@AQA explains: Making the grades – a guide to awarding

So, if in a given year the exam is hard, a student isn’t penalised for sitting a hard exam. The grade boundaries are adjusted to bring that year’s grades into line with the previous year. How are these adjustments made?

Using criteria, of course! Exam boards use criteria to broadly assess what a student needs to know to achieve a grade 6, for example. This isn’t used to determine the grade; it’s just used to move the grade boundaries if necessary.

There is an ethical implication of using a normal distribution to grade students: by default, a certain percentage of students are automatically consigned to ‘bad’ grades, i.e., the grades that society determines to be below par.

Nicky Morgan vs The Bell Curve — Via @teacherhead

By ‘society’, in this case we mostly mean colleges, universities, and employers, each of whom use a student’s grades to determine whether or not a student is of sufficient quality to accept their application.

For example, to study maths at university, you need to know a certain amount of maths before you get there. And the way that a university determines if you’re good enough is by looking at your grades. Which in theory tells them how much maths you know… Except… Hold on…

The grade doesn’t technically tell the university that. It tells them where you ranked that year. You could make the argument that society therefore uses normal distribution based grades to make decisions that would be more accurately informed by criterion based grades.

So, why don’t we use criterion based grades? It seems like the main reason is that they are hard to get right, so normal distribution based grades are the least bad option.

Is everyone OK with fact that we force 30% of children to fail their GCSEs? — Via @RethinkingJames

Which is why we have never used them to grade GCSEs in the UK.

Mythbusting: 3 Common Misconceptions — Via @ofqual

So, in summary, normal distribution based is bad, criterion based is hard to get right. And everything I have linked was written pre-pandemic…

Questions remain: Why do universities use criterion based but schools don’t? Why do we persist with normal distribution based grades when society uses them as a proxy for criterion based grades anyway? Why weren’t things improved when moving from A-G to 9-1?

Still, this was a useful exercise for me before responding to the DFE consultation, and I hope it helps someone else too.

Thanks to Thread Reader for unrolling the original Twitter thread.