When Research Fails

The Cognitive Science Gang's Best Joke

I’m very late to the party on this, but it’s so hilarious it’s had me rolling on the floor, clutching my sides in crucified tears of laughter and mirth just begging it to stop, please, please stop. The merest thought of it substantially brightens my day and causes an involuntary and mildly sardonic smile to turn the corners of my mouth in an upwards direction when I’m otherwise involved in something else. Sometimes, it even makes me want to skip with child-like joy, only skipping tires me these days and I was never much of a skipper in any case.

What is it that has me so tickled that I cannot rouse myself even to do the washing up? What joke is so satisfying that it has surpassed a combination of Bill Hicks and Stewart Lee in full flight and in top form? Which genius has come up with something so wonderfully stupid that even a mirthless bastard like myself made up chiefly of swears, creaking bones and bitterness has seen the point of living again?

The japester in question is that arch prankster, Dylan Wiliam. Dylan, Kate Jones, in an article on the website Evidence Based Education draws our attention to a paper from 1996, ‘Psychological theory and the design of innovative learning environments: On procedures, principles, and systems’ by Brown, A. L., & Campione, J. C. (1996). Sadly, I can’t get access to this paper as I was very much looking forward to more in the way of entertainment.

Broadly, it appears that the paper outlines a quite wonderful construct: where a research-tested idea goes wrong when it’s tried out in the classroom, it’s not because it’s a not a very good idea; it’s not because it doesn’t work in that context; it’s not because you can’t implement laboratory conditions in complex human systems. It’s for another reason. If you’ve found, like I have, that retrieval is pretty effective for facts and not much cop for high level conceptual stuff it’s because of … wait for it … ‘lethal mutation’. When an idea from research doesn’t work in the classroom, it’s not that the research might be flawed, that the technique is too dull for students to engage with, or that its claims have been overstated in the first place, it’s because the teacher didn’t do it properly and caused a lethal mutation of the original research.

Surely, our legs are being pulled here. This cannot be an actual thing. I’d not thought the cognitive science cabal to be such masterful humourists, but it seems they’ve talents in areas we’ve yet to dream of.

It has a science word in it because we are dealing in the empirical world: things that are true and correct (cognitive science) and things that are neither true nor correct (anything else). A mutation is a change in a DNA sequence.

Oh, the beautifully absurd silliness floods over one in waves! What has that got to do with some slightly muddled implementation of tepid idea? Is the DNA sequence of the idea fatally altered? Do ideas have DNA? Nope. They do not. It’s a silly perversion of language designed to make something appear serious when it isn’t. And why lethal? Johnny just failed to recall a piece of information from last week. No one died.

There seems to be so little a teacher could possibly get wrong with the few ideas from this realm that everyone seems to be aware of. How difficult is this stuff? Retrieval is basically asking some questions about the last week’s or last lesson’s learning at the front end of the next lesson. You can play around with it a bit, but it is what it is, and what it is, is deeply uncomplicated, profoundly intellectually untroubling. You’ve got to be some version of an experimental genius or a total dope to get it wrong.

Cold call (which is one person’s idea and not, as far as I know, subject to academic trials) in which you ask a question then say a name – well, that’s not too difficult either. How can you possibly get that wrong? It’s simple as water. Whatever you think of it, it’s just saying a question, then a name. I could have done it perfectly well when I was three. “Can you get me a bread and sugar sandwich …? And that question goes to … Mummy.”

It is dimly reminiscent of Brexit. No one could come up with a reason that it was a good idea and when, predictably, it all went completely tits up, it wasn’t because it was a stupid idea in the first place; it was because it wasn’t the right sort of Brexit – the implementation was a lethal mutation of the original idea. When theoretical ideas don’t work in a classroom, it’s not because they don’t work in practice as well as had been imagined, it’s because it’s not the right sort of implementation. It’s blame-shifting from the science, which is perfect and precise and cannot be questioned, to the teachers, who are none of the above.

The education community should be pouring ridicule on some of this rubbish but, instead, we nod meekly on as we’re deluded by obfuscations that would not stand up to the analytical probing of a gibbering ninny while the sellers of profoundly damaging snake oil totter round with nicely designed Powerpoints and increasingly wordy intellectual justifications for the presentation of a gilded turd.