What PISA Really Means

The PISA test results have just been released, and predictable as death and taxes the United States is falling behind the rest of the world in our scores.  Remember the adage, “Never let a good crisis go to waste”?  It looks like we have an educational crisis and we have to fix it.  Common Core to the rescue right?   But what does PISA really mean and should we be worried about our world ranking?

Christopher Tienken has written extensively on PISA and what the scores really mean, but in this piece he uses their own words.

(The  questions at the end of this are great.  I’d really like to see someone important answer them.)

 

What PISA Says About PISA

 

by Dr. Christopher Tienken

Pundits, education bureaucrats, and policy makers rejoice! It’s PISA time once again. Cue the dark music, fear mongering, worn out slogans and dogma about the United States education system failing the country economically. Sprinkle in “global competitiveness” throughout your press release, gush over how well those non-creative, authoritarian Asian countries performed, push your market oriented, anti-local control reforms, and presto, you are ready for prime-time education-reformer status. It seems as if America is suffering from a severe case of PISA envy. But what do the vendors of PISA say about PISA?

Unfortunately, the release of the latest PISA scores tells us nothing about the quality of a country’s education system, nor do the results predict economic doom or success. According to the Organisation for Economic Co-operation and Development (OECD, 2013, p.265), the private group that sells the PISA, the results should not be used to make sweeping indictments of education systems or important policy decisions. In fact, the vendors caution that the results of the PISA tests are a combination of schooling, life experiences, poverty, and access to early childhood programs, just to name a few factors:

“If a country’s scale scores in reading, scientific or mathematical literacy are significantly higher than those in another country, it cannot automatically be inferred that the schools or particular parts of the education system in the first country are more effective than those in the second. However, one can legitimately conclude that the cumulative impact of learning experiences in the first country, starting in early childhood and up to the age of 15, and embracing experiences both in school, home and beyond, have resulted in higher outcomes in the literacy domains that PISA measures.”

Not only are PISA results influenced by experiences “in the home and beyond”, but there is a sizeable relationship between the level of child poverty in a country and PISA results. Poverty explains up to 46% of the PISA scores in OECD countries (OECD, 2013, pp. 35-36). That does not bode well for the U.S. with one of the highest childhood poverty rates of the major industrialized countries.

Schooling does not end when a child turns 15 or 16, the ages of the students tested by PISA. Students continue their education for another 2-3 years and are thus exposed to more content. The vendors of PISA acknowledge that the scores from a 15 year-old child could not possibly predict or account for all that child knows or will grow to learn in the future. According to the PISA technical manual (OECD, 2009 p. 261) curriculum alignment and the selectiveness in countries’ testing populations also contribute to differences in the scores:

“This is not only because different students were assessed but also because the content of the PISA assessment was not expressly designed to match what students had learned in the preceding school year but more broadly to assess the cumulative outcome of learning in school up to age 15. For example, if the curriculum of the grades in which 15-year-olds are enrolled mainly includes material other than that assessed by PISA (which, in turn, may have been included in earlier school years) then the observed performance difference will underestimate student progress.”

Furthermore, the vendors reiterate their cautions that PISA is not aligned to any curriculum (2009, p.48):

“PISA measures knowledge and skills for life and so it does not have a strong curricular focus. This limits the extent to which the study is able to explore relationships between differences in achievement and differences in the implemented curricula.”

But what “skills for life” does PISA measure? A look at the released items suggest that some of the content measured is just rehashed versions of subject matter that has been around for the last 120 years: Hardly 21st century skills. PISA does not measure resilience, persistence, collaboration, cooperation, cultural awareness, strategizing, empathy, compassion, or divergent thinking.

So, if the vendors of PISA repeatedly warn that PISA is not aligned to school curricula, the scores are influenced strongly by poverty and wealth, the skills are left over from the 19th and 20th centuries, and out-of-school factors contribute to the overall education output in a country, then what does PISA really tell us about the quality of a school system or global competitiveness? Not much.

U.S. students have never scored at the top of the ranks on PISA or any other international test given since 1964. Countries like Estonia, Slovenia, Slovak Republic, Poland, and Latvia outscore the U.S. on every PISA. Does that matter? What is their per-capita GDP? How many Nobel Prizes have they won? How many utility patents do they produce each year? Where have high PISA scores gotten them? Are they going to “out-compete” the U.S.? I don’t think so.

Beyond the utterly anti-intellectual statements being made about the latest round of PISA scores, there are some basic questions that policy makers, education bureaucrats, and the latest crop of self-proclaimed savior-reformers should answer before thrusting assertions and untested policies upon 50 million public school children.

What is your definition of global competitiveness?

How can one test predict global competiveness or economic growth?

Was the PISA test designed to predict economic growth (OECD, 2009; 2013)?

What empirical evidence do you have that high PISA scores result in higher levels of innovation, creativity, and entrepreneurship (Zhao, 2012)?

Are you aware, that when you disaggregate the data by percentages of poverty in a school, the U.S. scores at the top of all the PISA tests (Riddle, 2009)?

Do you know what disaggregate means?

If countries like Estonia, Hungary, Slovenia, Vietnam, Latvia, and Poland routinely outscore us on PISA, why isn’t their per capita gross domestic product or other personal economic indicators equal to those in the U.S. (World Bank, 2013)?

What empirical evidence do you have that PISA scores cause economic growth in the G20 countries (Tienken, 2008)?

What jobs are U.S. children competing for in this economy?

What evidence do you have to demonstrate U.S. students are competing for the jobs you cite and with whom are they competing (evidence for that as well…)?

Do you think that lower wages is a reason multinational corporations choose to sell out the American public and set up shops in places like Pakistan,
Indonesia, Cambodia, India, China, Bangladesh, and Haiti?

Are you aware of the strong relationship between our growing trade with China and the loss of our manufacturing jobs (Pierce & Schott, 2012; Traywick, 2013)?

Why are companies like Boeing and GE allowed to give their technology, utility patents, and know-how to the Chinese in return for being able to sell their products in China (Prestowitz, 2012)?

Can higher PISA scores change the policy of allowing U.S. multinationals to give away our technological advantages?

Are you aware that only 10% of Chinese engineering graduates and 25% of Indian engineers are prepared to work in multinational corporations or corporations
outside of China or India (Gereffi, et al., 2006; Kiwana, 2012)?

If you are not aware of that fact, don’t you think you should be?

Are you aware that 81% of U.S. engineers are qualified to work in multinational corporations – the highest percentage in the world (Kiwana, 2012)?

Are you aware that adults in the U.S. rank at the top of the world in creativity, innovation, and entrepreneurship and that those adults were educated during a time of NO state or national standards (Tienken, 2013)?

If you are not aware of that fact, don’t you think you should be?

Are you aware that the U.S. produces the largest numbers of utility patents (innovation patents) per year and has produced over 100,000 a year for at least the last 45 years? No other country comes close (USPTO, 2012).

Did you answer “No” to three or more of these questions? If so, don’t you think it is time that you save the taxpayers money and resources and resign?

 

Here is an interview that Dr. Tienken did on PISA.

 

 

Sources

Gereffi, G., Wadhwa, V. & Rissing, B. (2006). Framing the Engineering Outsourcing Debate: Comparing the Quantity and Quality of Engineering Graduates in the United States, India and China. Available at SSRN: http://ssrn.com/abstract=1015831 or http://dx.doi.org/10.2139/ssrn.1015831

Kiwana, L., Kumar, A., & Randerson, N. (2012).The Skills Threat from China and India – Fact or Fiction. Engineering U.K. Retrieved fromhttp://www.engineeringuk.com/_resources/documents/Engineering_Graduates_in_China_and_India_-_EngineeringUK_-_March_2012.pdf

Organisation for Economic Co-operation and Development. (2010). PISA 2009 results: What students know and can do: Student performance in reading, mathematics and science (Vol. I). Retrieved from http://www.oecd.org/pisa/ pisaproducts/pisa2009/pisa2009resultswhatstudents knowandcandostudentperformanceinreadingmath ematicsandsciencevolumei.htm

Organisation for Economic Co-operation and Development. (2013). PISA 2012 results. What students know and can do: Student performance in reading, mathematics and science (Vol. I). http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-I.pdf

Pierce, J.R. (2012). The Surprisingly Swift Decline of U.S. Manufacturing Employment. Yale School of Management and National Bureau of Economic Research. Retrieved from http://economics.yale.edu/sites/default/files/schott-09-oct-2013.pdf

Prestowitz, C. (2012, Feb. 22). GE’s Competitiveness Charade. Foreign Policy. Retrieved from:http://prestowitz.foreignpolicy.com/posts/2012/02/22/ges_competitiveness_charade 5

Riddle, M. (2010, December 15). PISA: It’s Poverty not Stupid [web post]. The Principal Difference. Retrieved fromhttp://nasspblogs.org/principaldifference/2010/12/pisa_its_poverty_not_stupid_1.html

Tienken, C.H. (2008). Rankings of International Achievement Test Performance and Economic Strength: Correlation or Conjecture? International Journal of Education Policy and Leadership, 3(3), 1-12.

Tienken, C.H. (2013). International Comparisons of Innovation and Creativity. Kappa Delta Pi Record, 49, 153-155.

Traywick, C.A. (2013, Nov. 5). Here’s Proof that Trading with Beijing is Screwing America’s Workers. Foreign Policy. Retrieved from:http://blog.foreignpolicy.com/ posts/2013/11/05/heres_proof_that_trading_with_china_is_screwing_american_workers

U.S. Patent and Trademark Office. (2012): Patents by Country, State, and Year: Utility Patents. Alexandria, VA: Author. Retrieved fromhttp://www.uspto.gov/web/offices/ac/ido/oeip/taf/cst_utl.htm

World Bank. (2013). GDP Per Capita. Retrieved from: http://data.worldbank.org/indicator/NY.GDP.PCAP.CD
– See more at: http://christienken.com/2013/12/05/what-pisa-says-about-pisa/#sthash.iLc3v8ZP.dpuf