Gregory Elacqua ha circulado la siguiente nota sobre un estudio deL. Woessmann que analiza el efecto de pago por merito en los resultados de PISA.
International Merit Pay Study Insufficient To Guide Policy
Contact:
Matthias von Davier, ETS
(609) 734-1717
[email protected]
William Mathis, NEPC
(802) 383-0058
[email protected]
BOULDER, CO (March 31, 2011) – Cross-Country Evidence on Teacher Performance Pay, a report recently published by the Program on Education Policy and Governance (PEPG) at Harvard, claims that students in nations where teachers are paid on “merit” score higher on an international achievement test.
Matthias von Davier, principal research scientist at the Education Testing Service, reviewed the PEPG report for the Think Twice think tank review project. Von Davier found that the analyses and interpretations offered (also published in an abridged form in the journal Education Next) fail to
heed the report’s own cautions, and it has too many crucial caveats to provide any useful policy guidance.
The review is published by the National Education Policy Center, housed at the University of Colorado at Boulder School of Education.
The report, authored by Ludger Woessmann of the University of Munich, contends that in those nations where teachers are paid based on student achievement or another performance-based measure, students have higher reading and math scores on the Programme for International Student
Assessment (PISA) achievement test, an international exam given to 15-year-olds.
Woessmann’s findings rest on a series of regression analyses of PISA scores. The data are analyzed at the country level. Because there are only 28 data points (nations) in the analysis, the small sample size “requires extreme caution in interpretation,” von Davier writes. Simply excluding or including a single country in the analysis “results in large shifts in the size of the reported relationships.” That is, the presence of performance pay may or may not have an effect on PISA scores, but this sort of study can do little to answer that question.
Von Davier notes that, “Although the author lists numerous caveats, his broad conclusions do not heed these cautions.” The caveats are crucial to understanding the usefulness of the study’s findings. For instance, differences in the way various countries provide so-called performance-based
pay are not properly considered. Perhaps one type of approach is beneficial, while another is detrimental. Additionally, variations in the length of timeone or another country has employed such a pay system are not addressed. A system that hasn’t been in place long enough to have a strong effect is
treated equally to one that has been in place much longer.
Von Davier points out that differences among nations, “could be due to any number of factors” – certainly there are enormous differences between countries and between their educational systems. So teasing out differences in PISA scores and then attributing those differences to the presence or
absence of a merit-pay system is problematic.
As a result of the multiple limitations in the report’s data and of issues in the analytic methods used, von Davier concludes, that more careful, in-depth studies of differences between educational systems are needed before looking to international comparisons to substantially inform our own
decisions about performance pay.
Find Matthias von Davier’s review on the NEPC website at:
http://nepc.colorado.edu/thinktank/review-pisa-performance-pay
Find Cross-Country Evidence on Teacher Performance Pay, by Ludger Woessmann
on the web at:
http://www.hks.harvard.edu/pepg/MeritPayPapers/Woessmann_10-11.pdf.
The Think Twice think tank review project (http://thinktankreview.org), a project of the National Education Policy Center, provides the public, policy makers, and the press with timely, academically sound, reviews of selected publications. The project is made possible in part by the support of the
Great Lakes Center for Education Research and Practice.
The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence.
For more information on NEPC, please visit http://nepc.colorado.edu/.
This review is also found on the GLC website at

Home

————————————————–
“Cross-Country Evidence on Teacher Performance” and “Merit Pay International”
by Ludger Woessmann
Harvard University Program on Education Policy and Governance & Education Next
February 8, 2011
Reviewed by Matthias von Davier (Educational Testing Service)
March 31, 2011
The primary claim of this Harvard Program on Education Policy and Governance report and the abridged Education Next version is that nations “that pay teachers on their performance score higher on PISA tests.” After statistically controlling for several variables, the author concludes that nations with some form of merit pay system have, on average, higher reading and math scores on this international test of 15-year-old students. Although the author lists numerous caveats, his broad conclusions do not heed these cautions. The fundamental differences among countries in the types of performance pay system are not properly considered. Nations are simply lumped together as having or not having a performance pay plan. Also, the length of time the program had been in place in each country is not addressed and the unknown intensity of program implementations argue against drawing lessons from this study. The small sample size of 28 observations requires extreme caution in interpretation. For example, the inclusion or exclusion of a single country results in large shifts in the size of the reported relationships. That is, the numbers become unreliable and invalid. With any correlational study, attributing causality is problematic; the differences among nations could be due to any number of factors. Finally, the type of regression-based analyses used to support the performance pay conclusion does not properly consider that the background variables used in these analyses can vary in terms of relationships with student scores and have different definitions across the countries under study. Therefore, drawing policy conclusions about teacher performance pay on the basis of this analysis is not warranted.
von Davier, M. (2011). Review of “Cross-Country Evidence on Teacher Performance Pay.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/thinktank/review-pisa-performance-pay.