PISA: the importance of informed conversations

Cameron Malcher

Dr Greg Thompson's book on decoding PISA numbers

Numbers are frequently used as rhetorical devices in a variety of contexts but as the technical information around the production of those numbers becomes more complex we run into the problem of finding it difficult to question the numbers, to understand them and to have conversations about them. The danger is that the use of numbers becomes something that is done to us rather than as something that is done to engage us as educational professionals.

This was how Dr Greg Thompson opened his discussion of PISA and other international standardised tests at Federation’s Friday Forum on 2 June.

Dr Thompson was speaking about the recent publication, The Global Education Race: Taking the Measure of PISA and International Testing, which he co-
authored with Sam Sellar, Reader in Education Studies at Manchester Metropolitan University and member of the Board of Directors for the Laboratory of International Assessment Studies, and David Rutkowski, Professor of Education at the Centre for Educational Measurement (CEMO) at the University of Oslo, Norway.

The book arose out of a series of discussions with the Alberta Teachers Association — which ultimately funded production of the book — about the ways in which testing data was being understood and discussed by government and the media, and the difficulties in trying to create meaningful conversations around data with education professionals and interested people in the community.

The book aims to provide a plain-language explanation of PISA for people in the government, media and people in the community who don’t have a background in statistics so that they can have a better understanding of how PISA works and what the data can be used for.

Dr Thompson said one of the problems with the level of debate about PISA is that it is frequently reduced to rankings and average scores, such as lists of top 10 countries and which countries have increased or fallen in rankings.

These rankings are also often talked about in terms of crisis — that a change in rankings indicates a crisis or an approaching crisis — and all this seems to do is add fuel to existing beliefs that people have about education and education systems. People’s lack of technical knowledge of the processes behind PISA then makes it difficult to have conversations that go beyond those beliefs.

“The problem that we see is a technical one but also a democratic one,” Dr Thompson said. “We call this technical democracy. And in a world where data and assessments and evidence are becoming more and more central to the work that education systems do, we think this is a critical issue for education researchers, education professionals, and the ways that we engage …”.

A question behind the creation of the book is whether it is possible to create “informed, interfering, non-experts”, and to an extent put the onus back on to education professionals to make decisions about how data is used and for what purposes they are prepared to allow data to be used.

Another challenge in discussing PISA is that while governments and system leaders use it as an indicator of curriculum, PISA does not test what is in any national or state curriculum but rather assesses things the OECD thinks that 15-year-olds need to know to be able to participate in a global economy.

The book lists examples of poor interpretations and questionable policy decisions made in response to PISA results, particularly the “crisis” in Alberta, Canada around declining maths achievement, and Australia’s policy goal of “being Top 5 in PISA by 2025”.

Dr Thompson outlined many of the issues with simplistic league tables and international comparisons, and broader elements of PISA that people should consider in such comparisons, including: the structure and statistics of the test itself; translating questions between languages, cultures and contexts; the different experiences of 15-year-olds in school systems in different countries; systems that exclude students from the PISA process and countries that have significantly lower school participation at 15 years of age.

The ultimate goal, according to Dr Thompson, is to help more people have more informed conversations. He concluded by asking audience members to imagine finding themselves in an elevator with the Education Minister who claimed that PISA showed that schools in your country are in crisis. You disagree, but how might you begin that conversation?

A video of Dr Thompson’s presentation will be available in a few days.

Click here to download PDF.