Krueger and Zhu challenge Mathematica's voucher study

Professor Alan Krueger, Bendheim Professor of Economics and Public Policy, and graduate student Pei Zhu went to the National Press Club in Washington, D.C. this week to present their findings that Mathematica Policy Research, Inc. incorrectly concluded private school vouchers have a statistically significant impact on the test scores of African-American students.

Krueger and Zhu described the results of their recent research that contradict the original conclusions of an experiment run by Mathematica and the Program on Education Policy and Governance at Harvard University from 1997 to 2000.

The Mathematica experiment involved a random selection of 2,600 students in kindergarten through fourth grade, half of whom received a voucher and half of whom served as a control group. For three years, Mathematica followed the students' standardized test scores. According to a Woodrow Wilson School press release, Mathematica concluded that the combined math and reading test scores of African American students offered the voucher were 5.5 points higher than those of African American students who were not.

After a reanalysis of the data, however, Krueger and Zhu claim they found problems with the original statistical analysis and racial classification, which significantly affected the results of the data. Kreuger and Zhu conclude that vouchers did not in fact have a statistically significant impact on black students.

"I think that [Mathematica] recognized that they haven't done something the best way that they could have," Krueger said.

About a year and a half ago, Krueger and Zhu decided to reexamine the voucher experiment data because of the publicity Mathematica's experiment received. They were curious to replicate the results and understand the process Mathematica used, Zhu said.

Kreuger and Zhu focused especially on certain points raised in the original report, such as why the experiment showed significant effects for only African Americans and not effects on all students given vouchers.

In examining Mathematica's experiment, Krueger and Zhu discovered that the data from 240 students were thrown out from Mathematica's analysis because they did not have a baseline score.

But Zhu said the organization made it possible for Krueger and Zhu to expand the sample and effectively continue the experiment because it provided its data and questions about the data it collected.

The addition of this data into the overall analysis increased the sample size by 44 percent, the Woodrow Wilson press release stated.

Krueger also found problems in the ethnic and racial classification of the students. Mathematica never directly asked for the students' race, but inferred it from the mother, he said. The survey did not question what race Hispanics considered themselves, so Mathematica combined race and ethnicity, Krueger said.

Though he recognized this was an oversight, David Meyer, vice president of Mathematica, stated that the intention of the experiment was not to look at different racial or ethnic subgroups, Krueger said.

Get the best of the ‘Prince’ delivered straight to your inbox. Subscribe now »

Krueger and Zhu made the adjustments to the data and found that for the new sample of Black students, vouchers had a statistically insignificant impact on the combined math and reading test scores. Their results are published in their paper titled, "Another Look at the New York City School Voucher Experiment."

"The safest conclusion is probably that the provision of vouchers did not lower the scores of African American students," Krueger said in the Wilson School press release.

Though unsure about the immediate impact of their new findings, Zhu said he hopes they will help people become more informed about the voucher policy.

"There are a number of feathers that vouchers can have that we don't understand," Krueger said. "We can't measure it yet."

Both Mathematica and Krueger agree that more experiments are needed before making any meaningful conclusions.

Kreuger and Zhu's presentation was a Wilson School in Washington event, through which the school seeks to better inform the public, media and public policy makers about relevant research.